From 9b046cbc0411c49f4973ec00f1fd3cc0d2e5e55c Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 8 Jun 2015 18:21:11 +0100
Subject: [PATCH 001/467] Tiny fix to embedded YAML fragment

---
 specification/25_application_service_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/25_application_service_api.rst b/specification/25_application_service_api.rst
index 87184950..8236b2de 100644
--- a/specification/25_application_service_api.rst
+++ b/specification/25_application_service_api.rst
@@ -72,7 +72,7 @@ An example HS configuration required to pass traffic to the AS is:
 
    users:
      - exclusive: true
-     - regex: @irc.freenode.net/.*
+       regex: @irc.freenode.net/.*
 
 
 Home Server -> Application Service API

From fc1e01f373eb07a016fd3940bff810f8d74d9508 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Thu, 9 Jul 2015 19:52:34 +0100
Subject: [PATCH 002/467] Add a small note about the meaning of the 'target'
 field

---
 specification/30_server_server_api.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/specification/30_server_server_api.rst b/specification/30_server_server_api.rst
index 48434075..2eb80f92 100644
--- a/specification/30_server_server_api.rst
+++ b/specification/30_server_server_api.rst
@@ -553,6 +553,11 @@ them in a JSON object and signing it using the JSON signing algorithm. The
 resulting signatures are added as an Authorization header with an auth scheme
 of X-Matrix.
 
+Note that the target field should include the full path starting with
+``/_matrix/...``, including the ``?`` and any query parameters if present, but
+should not include the leading ``https:``, nor the destination server's
+hostname.
+
 Step 1 sign JSON:
 
 .. code::

From 5f391b5617b022b6ae32f84f384973b446859d6c Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Thu, 9 Jul 2015 20:28:48 +0100
Subject: [PATCH 003/467] Also document the directory server federation API

---
 specification/30_server_server_api.rst | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/specification/30_server_server_api.rst b/specification/30_server_server_api.rst
index 2eb80f92..f0aab39a 100644
--- a/specification/30_server_server_api.rst
+++ b/specification/30_server_server_api.rst
@@ -747,3 +747,22 @@ result field. If such is present, then the result should contain only a field
 of that name, with no others present. If not, the result should contain as much
 of the user's profile as the home server has available and can make public.
 
+Directory
+---------
+
+The server API for directory queries is also based on Federation Queries.
+
+Querying directory information::
+
+  Query type: directory
+
+  Arguments:
+    room_alias: the room alias to query
+
+  Returns: JSON object containing the following keys:
+    room_id: string giving the underlying room ID the alias maps to
+    servers: list of strings giving the join candidates
+
+The list of join candidates is a list of server names that are likely to hold
+the given room; these are servers that the requesting server may wish to try
+joining with. This list may or may not include the server answering the query.

From c6f2f711de9b33d2e4b9660584d59eaf676b9f3d Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 31 Jul 2015 18:05:33 +0100
Subject: [PATCH 004/467] Mention AAAA as well as A records; remark on default
 port 8448 for no-SRV fallback

---
 specification/30_server_server_api.rst | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/specification/30_server_server_api.rst b/specification/30_server_server_api.rst
index 48434075..f3490eeb 100644
--- a/specification/30_server_server_api.rst
+++ b/specification/30_server_server_api.rst
@@ -59,10 +59,13 @@ and an optional TLS port.
 
 .. **
 
-If the port is present then the server is discovered by looking up an A record
-for the DNS name and connecting to the specified TLS port. If the port is
-absent then the server is discovered by looking up a ``_matrix._tcp``
-SRV record for the DNS name.
+If the port is present then the server is discovered by looking up an AAAA or
+A record for the DNS name and connecting to the specified TLS port.
+
+If the port is absent then the server is discovered by looking up a
+``_matrix._tcp`` SRV record for the DNS name. If this record does not exist
+then the server is discovered by looking up an AAAA or A record on the DNS
+name and taking the default fallback port number of 8448.
 
 Home servers may use SRV records to load balance requests between multiple TLS
 endpoints or to failover to another endpoint if an endpoint fails.

From 49c52a96a793b01422722852c14f37acd10cbc07 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Wed, 5 Aug 2015 11:35:48 +0100
Subject: [PATCH 005/467] Rename 'perspective server' to 'notary server' as
 that's what the Perspectives Project calls them too

---
 specification/30_server_server_api.rst | 30 +++++++++++++-------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/specification/30_server_server_api.rst b/specification/30_server_server_api.rst
index 580e7bb0..1c7bf3ed 100644
--- a/specification/30_server_server_api.rst
+++ b/specification/30_server_server_api.rst
@@ -78,17 +78,17 @@ Version 2
 
 Each home server publishes its public keys under ``/_matrix/key/v2/server/``.
 Home servers query for keys by either getting ``/_matrix/key/v2/server/``
-directly or by querying an intermediate perspective server using a
-``/_matrix/key/v2/query`` API. Intermediate perspective servers query the
+directly or by querying an intermediate notary server using a
+``/_matrix/key/v2/query`` API. Intermediate notary servers query the
 ``/_matrix/key/v2/server/`` API on behalf of another server and sign the
-response with their own key. A server may query multiple perspective servers
-to ensure that they all report the same public keys.
+response with their own key. A server may query multiple notary servers to
+ensure that they all report the same public keys.
 
 This approach is borrowed from the Perspectives Project
 (http://perspectives-project.org/), but modified to include the NACL keys and to
 use JSON instead of XML. It has the advantage of avoiding a single trust-root
-since each server is free to pick which perspective servers they trust and can
-corroborate the keys returned by a given perspective server by querying other
+since each server is free to pick which notary servers they trust and can
+corroborate the keys returned by a given notary server by querying other
 servers.
 
 Publishing Keys
@@ -122,15 +122,15 @@ The ``old_verify_keys`` can be used to sign events with an ``origin_server_ts``
 before the ``expired_ts``. The ``expired_ts`` is a millisecond POSIX timestamp
 of when the originating server stopped using that key.
 
-Intermediate perspective servers should cache a response for half of its
-remaining life time to avoid serving a stale response. Originating servers should
-avoid returning responses that expire in less than an hour to avoid repeated
-requests for an about to expire certificate. Requesting servers should limit how
+Intermediate notary servers should cache a response for half of its remaining
+life time to avoid serving a stale response. Originating servers should avoid
+returning responses that expire in less than an hour to avoid repeated requests
+for an about to expire certificate. Requesting servers should limit how
 frequently they query for certificates to avoid flooding a server with requests.
 
-If a server goes offline intermediate perspective servers should continue to
-return the last response they received from that server so that the signatures
-of old events sent by that server can still be checked.
+If a server goes offline intermediate notary servers should continue to return
+the last response they received from that server so that the signatures of old
+events sent by that server can still be checked.
 
 ==================== =================== ======================================
     Key                    Type                         Description
@@ -191,8 +191,8 @@ The ``minimum_valid_until_ts`` is a millisecond POSIX timestamp indicating
 when the returned certificate will need to be valid until to be useful to the 
 requesting server. This can be set using the maximum ``origin_server_ts`` of 
 an batch of events that a requesting server is trying to validate. This allows
-an intermediate perspectives server to give a prompt cached response even if
-the originating server is offline.
+an intermediate notary server to give a prompt cached response even if the
+originating server is offline.
 
 This API can return keys for servers that are offline be using cached responses
 taken from when the server was online. Keys can be queried from multiple

From b24582aead73c81eff57da6325564b63e8a7a18f Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Fri, 14 Aug 2015 11:03:15 +0100
Subject: [PATCH 006/467] Introducing jekyll (jekyllrb.com) for docs

---
 .gitignore                                    |   1 +
 supporting-docs/_config.yml                   |  15 +
 supporting-docs/_includes/footer.html         |   1 +
 supporting-docs/_includes/head.html           |  14 +
 supporting-docs/_includes/nav.html            |  22 +
 supporting-docs/_layouts/default.html         |  19 +
 supporting-docs/_layouts/page.html            |  14 +
 supporting-docs/_layouts/post.html            |   6 +
 .../_plugins/jekyll-rst-master/.gitignore     |   1 +
 .../_plugins/jekyll-rst-master/LICENSE.txt    |  20 +
 .../_plugins/jekyll-rst-master/README.rst     |  97 +++
 .../_plugins/jekyll-rst-master/converter.rb   |  30 +
 .../_plugins/jekyll-rst-master/directives.py  |  97 +++
 .../_plugins/jekyll-rst-master/rst2html.py    |  39 ++
 .../_plugins/jekyll-rst-master/transform.py   |  42 ++
 .../_plugins/project_version_tag.rb           |  51 ++
 .../_posts/2015-08-10-client-server.rst       | 652 ++++++++++++++++++
 .../_posts/2015-08-14-getting_involved.md     | 120 ++++
 supporting-docs/_posts/index.html             |  19 +
 supporting-docs/_sass/_base.scss              | 204 ++++++
 supporting-docs/_sass/_layout.scss            | 236 +++++++
 .../_sass/_syntax-highlighting.scss           |  67 ++
 supporting-docs/about.md                      |  11 +
 supporting-docs/css/basic.css                 | 512 ++++++++++++++
 supporting-docs/css/main.scss                 |  52 ++
 supporting-docs/css/nature.css                | 285 ++++++++
 supporting-docs/css/site.css                  | 102 +++
 supporting-docs/feed.xml                      |  30 +
 supporting-docs/index.html                    |  23 +
 29 files changed, 2782 insertions(+)
 create mode 100644 supporting-docs/_config.yml
 create mode 100644 supporting-docs/_includes/footer.html
 create mode 100644 supporting-docs/_includes/head.html
 create mode 100644 supporting-docs/_includes/nav.html
 create mode 100644 supporting-docs/_layouts/default.html
 create mode 100644 supporting-docs/_layouts/page.html
 create mode 100644 supporting-docs/_layouts/post.html
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/.gitignore
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/README.rst
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/converter.rb
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/directives.py
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/rst2html.py
 create mode 100644 supporting-docs/_plugins/jekyll-rst-master/transform.py
 create mode 100644 supporting-docs/_plugins/project_version_tag.rb
 create mode 100644 supporting-docs/_posts/2015-08-10-client-server.rst
 create mode 100644 supporting-docs/_posts/2015-08-14-getting_involved.md
 create mode 100644 supporting-docs/_posts/index.html
 create mode 100644 supporting-docs/_sass/_base.scss
 create mode 100644 supporting-docs/_sass/_layout.scss
 create mode 100644 supporting-docs/_sass/_syntax-highlighting.scss
 create mode 100644 supporting-docs/about.md
 create mode 100644 supporting-docs/css/basic.css
 create mode 100755 supporting-docs/css/main.scss
 create mode 100644 supporting-docs/css/nature.css
 create mode 100644 supporting-docs/css/site.css
 create mode 100644 supporting-docs/feed.xml
 create mode 100644 supporting-docs/index.html

diff --git a/.gitignore b/.gitignore
index df0135f5..188b74a4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,4 @@
 scripts/gen
 templating/out
 *.pyc
+supporting-docs/_site
diff --git a/supporting-docs/_config.yml b/supporting-docs/_config.yml
new file mode 100644
index 00000000..3da7faed
--- /dev/null
+++ b/supporting-docs/_config.yml
@@ -0,0 +1,15 @@
+# Site settings
+title: Matrix
+email: webmaster@matrix.org
+description: > # this means to ignore newlines until "baseurl:"
+Matrix.org documentation
+baseurl: "/docs/guides" # the subpath of your site, e.g. /blog/
+url: "http://matrix.org" # the base hostname & protocol for your site
+twitter_username: matrixdotorg
+github_username:  matrix-org
+
+# Build settings
+markdown: kramdown
+
+#defaults:
+permalink: /:categories/:title.html
diff --git a/supporting-docs/_includes/footer.html b/supporting-docs/_includes/footer.html
new file mode 100644
index 00000000..14e9c2a5
--- /dev/null
+++ b/supporting-docs/_includes/footer.html
@@ -0,0 +1 @@
+  <div id="footer"><div id="footerContent">&copy 2014-2015 Matrix.org</div>
diff --git a/supporting-docs/_includes/head.html b/supporting-docs/_includes/head.html
new file mode 100644
index 00000000..bb99e6dd
--- /dev/null
+++ b/supporting-docs/_includes/head.html
@@ -0,0 +1,14 @@
+  <meta charset="utf-8">
+  <meta http-equiv="X-UA-Compatible" content="IE=edge">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+
+  <title>{% if page.title %}{{ page.title }}{% else %}{{ site.title }}{% endif %}</title>
+  <meta name="description" content="{% if page.excerpt %}{{ page.excerpt | strip_html | strip_newlines | truncate: 160 }}{% else %}{{ site.description }}{% endif %}">
+
+  <link rel="stylesheet" href="{{ "/css/main.css" | prepend: site.baseurl }}">
+  <link rel="stylesheet" href="{{ "/css/site.css" | prepend: site.baseurl }}">
+  <link rel="stylesheet" href="{{ "/css/basic.css" | prepend: site.baseurl }}">
+  <link rel="stylesheet" href="{{ "/css/nature.css" | prepend: site.baseurl }}">
+
+  <link rel="canonical" href="{{ page.url | replace:'index.html','' | prepend: site.baseurl | prepend: site.url }}">
+  <link rel="alternate" type="application/rss+xml" title="{{ site.title }}" href="{{ "/feed.xml" | prepend: site.baseurl | prepend: site.url }}" />
diff --git a/supporting-docs/_includes/nav.html b/supporting-docs/_includes/nav.html
new file mode 100644
index 00000000..b9f13762
--- /dev/null
+++ b/supporting-docs/_includes/nav.html
@@ -0,0 +1,22 @@
+  <div id="header">
+    <div id="headerContent">
+
+        <a href="/#about2" class="navButton">About</a>
+        <a href="/docs/howtos" class="navButton">HOWTOs</a>
+        <a href="/docs/spec" class="navButton">Spec</a>
+        <a href="/docs/api" class="navButton">APIs</a>
+        <a href="/code" class="navButton">Code</a>
+        <a href="/jira" class="navButton">JIRA</a>
+        <a href="/blog/faq" class="navButton">FAQ</a>
+        <a href="/blog" class="navButton">Blog</a>
+<script>
+  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+  ga('create', 'UA-54779209-1', 'auto');
+  ga('send', 'pageview');
+</script>
+    </div>
+  </div>
diff --git a/supporting-docs/_layouts/default.html b/supporting-docs/_layouts/default.html
new file mode 100644
index 00000000..ae99cc0d
--- /dev/null
+++ b/supporting-docs/_layouts/default.html
@@ -0,0 +1,19 @@
+<!DOCTYPE html>
+<html>
+  <head>
+  {% include head.html %}
+
+  </head>
+  <body>
+  {% include nav.html %}
+    <br />
+    <div class="page-content">
+      <div class="wrapper">
+      {{ content }}
+      </div>
+    </div>
+    {% include footer.html %}
+
+  </body>
+
+</html>
diff --git a/supporting-docs/_layouts/page.html b/supporting-docs/_layouts/page.html
new file mode 100644
index 00000000..74c1a118
--- /dev/null
+++ b/supporting-docs/_layouts/page.html
@@ -0,0 +1,14 @@
+---
+layout: default
+---
+<div class="post">
+
+  <header class="post-header">
+    <h1 class="post-title">{{ page.title }}</h1>
+  </header>
+
+  <article class="post-content">
+    {{ content }}
+  </article>
+
+</div>
diff --git a/supporting-docs/_layouts/post.html b/supporting-docs/_layouts/post.html
new file mode 100644
index 00000000..64fafcbc
--- /dev/null
+++ b/supporting-docs/_layouts/post.html
@@ -0,0 +1,6 @@
+---
+layout: default
+---
+
+    {{ content }}
+
diff --git a/supporting-docs/_plugins/jekyll-rst-master/.gitignore b/supporting-docs/_plugins/jekyll-rst-master/.gitignore
new file mode 100644
index 00000000..7e99e367
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/.gitignore
@@ -0,0 +1 @@
+*.pyc
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt b/supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt
new file mode 100644
index 00000000..988ac9eb
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt
@@ -0,0 +1,20 @@
+The MIT License (MIT)
+Copyright (c) 2011 Greg Thornton, http://xdissent.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/README.rst b/supporting-docs/_plugins/jekyll-rst-master/README.rst
new file mode 100644
index 00000000..48c21f42
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/README.rst
@@ -0,0 +1,97 @@
+Overview
+========
+
+This plugin adds `ReStructuredText`_ support to `Jekyll`_ and `Octopress`_. 
+It renders ReST in posts and pages, and provides a custom directive to
+support Octopress-compatible syntax highlighting.
+
+Requirements
+============
+
+* Jekyll *or* Octopress >= 2.0
+* Docutils
+* Pygments
+* `RbST`_
+
+Installation
+============
+
+1. Install Docutils and Pygments. 
+
+   The most convenient way is to use virtualenv_burrito:
+
+   ::
+
+      $ curl -s https://raw.github.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh | bash
+      $ source /Users/xdissent/.venvburrito/startup.sh
+      $ mkvirtualenv jekyll-rst
+      $ pip install docutils pygments
+
+2. Install RbST.
+
+   If you use `bundler`_ with Octopress, add ``gem 'RbST'`` to 
+   your ``Gemfile`` in the ``development`` group, then run 
+   ``bundle install``. Otherwise, ``gem install RbST``.
+
+3. Install the plugin.
+
+   For Jekyll:
+
+   ::
+
+      $ cd <jekyll-project-path>
+      $ git submodule add https://github.com/xdissent/jekyll-rst.git _plugins/jekyll-rst
+
+   For Octopress:
+
+   ::
+
+      $ cd <octopress-project-path>
+      $ git submodule add https://github.com/xdissent/jekyll-rst.git plugins/jekyll-rst
+
+4. Start blogging in ReStructuredText. Any file with the ``.rst`` extension
+   will be parsed as ReST and rendered into HTML.
+
+   .. note:: Be sure to activate the ``jekyll-rst`` virtualenv before generating
+      the site by issuing a ``workon jekyll-rst``. I suggest you follow `Harry
+      Marr's advice`_ and create a ``.venv`` file that will  automatically 
+      activate the ``jekyll-rst`` virtualenv when you ``cd`` into your project.
+
+Source Code Highlighting
+========================
+
+A ``code-block`` ReST directive is registered and aliased as ``sourcecode``. 
+It adds syntax highlighting to code blocks in your documents::
+
+   .. code-block:: ruby
+      
+      # Output "I love ReST"
+      say = "I love ReST"
+      puts say
+
+Optional arguments exist to supply a caption, link, and link title::
+
+   .. code-block:: console
+      :caption: Read Hacker News on a budget
+      :url: http://news.ycombinator.com
+      :title: Hacker News
+
+      $ curl http://news.ycombinator.com | less
+
+Octopress already includes style sheets for syntax highlighting, but you'll
+need to generate one yourself if using Jekyll::
+
+   $ pygmentize -S default -f html > css/pygments.css
+
+Octopress Tips
+==============
+
+* Use ``.. more`` in your ReST documents to indicate where Octopress's 
+  ``excerpt`` tag should split your content for summary views.
+
+.. _ReStructuredText: http://docutils.sourceforge.net/rst.html
+.. _Jekyll: http://jekyllrb.com/
+.. _Octopress: http://octopress.com/
+.. _RbST: http://rubygems.org/gems/RbST
+.. _bundler: http://gembundler.com/
+.. _Harry Marr's advice: http://hmarr.com/2010/jan/19/making-virtualenv-play-nice-with-git/
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/converter.rb b/supporting-docs/_plugins/jekyll-rst-master/converter.rb
new file mode 100644
index 00000000..64b639cc
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/converter.rb
@@ -0,0 +1,30 @@
+require 'rbst'
+
+module Jekyll
+  class RestConverter < Converter
+    safe true
+
+    priority :low
+
+    def matches(ext)
+      ext =~ /rst/i
+    end 
+
+    def output_ext(ext)
+      ".html"
+    end
+
+    def convert(content)
+      RbST.executables = {:html => "#{File.expand_path(File.dirname(__FILE__))}/rst2html.py"}
+      RbST.new(content).to_html(:part => :fragment, :initial_header_level => 2)
+    end
+  end
+
+  module Filters
+    def restify(input)
+      site = @context.registers[:site]
+      converter = site.getConverterImpl(Jekyll::RestConverter)
+      converter.convert(input)
+    end
+  end
+end  
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/directives.py b/supporting-docs/_plugins/jekyll-rst-master/directives.py
new file mode 100644
index 00000000..bfd2b78b
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/directives.py
@@ -0,0 +1,97 @@
+# Define a new directive `code-block` (aliased as `sourcecode`) that uses the 
+# `pygments` source highlighter to render code in color. 
+#
+# Incorporates code from the `Pygments`_ documentation for `Using Pygments in 
+# ReST documents`_ and `Octopress`_.
+#
+# .. _Pygments: http://pygments.org/
+# .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/
+# .. _Octopress: http://octopress.org/
+
+import re
+import os
+import md5
+import __main__
+
+# Absolute path to pygments cache dir
+PYGMENTS_CACHE_DIR = os.path.abspath(os.path.join(os.path.dirname(__main__.__file__), '../../.pygments-cache'))
+
+# Ensure cache dir exists
+if not os.path.exists(PYGMENTS_CACHE_DIR):
+    os.makedirs(PYGMENTS_CACHE_DIR)
+
+from pygments.formatters import HtmlFormatter
+
+from docutils import nodes
+from docutils.parsers.rst import directives, Directive
+
+from pygments import highlight
+from pygments.lexers import get_lexer_by_name, TextLexer
+
+class Pygments(Directive):
+    """ Source code syntax hightlighting.
+    """
+    required_arguments = 1
+    optional_arguments = 0
+    final_argument_whitespace = True
+    string_opts = ['title', 'url', 'caption']
+    option_spec = dict([(key, directives.unchanged) for key in string_opts])
+    has_content = True
+
+    def run(self):
+        self.assert_has_content()
+        try:
+            lexer_name = self.arguments[0]
+            lexer = get_lexer_by_name(lexer_name)
+        except ValueError:
+            # no lexer found - use the text one instead of an exception
+            lexer_name = 'text'
+            lexer = TextLexer()
+        formatter = HtmlFormatter()
+
+        # Construct cache filename
+        cache_file = None
+        content_text = u'\n'.join(self.content)
+        cache_file_name = '%s-%s.html' % (lexer_name, md5.new(content_text).hexdigest())
+        cached_path = os.path.join(PYGMENTS_CACHE_DIR, cache_file_name)
+
+        # Look for cached version, otherwise parse
+        if os.path.exists(cached_path):
+            cache_file = open(cached_path, 'r')
+            parsed = cache_file.read()
+        else:
+            parsed = highlight(content_text, lexer, formatter)
+
+        # Strip pre tag and everything outside it
+        pres = re.compile("<pre>(.+)<\/pre>", re.S)
+        stripped = pres.search(parsed).group(1)
+
+        # Create tabular code with line numbers
+        table = '<div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers">'
+        lined = ''
+        for idx, line in enumerate(stripped.splitlines(True)):
+            table += '<span class="line-number">%d</span>\n' % (idx + 1)
+            lined  += '<span class="line">%s</span>' % line
+        table += '</pre></td><td class="code"><pre><code class="%s">%s</code></pre></td></tr></table></div>' % (lexer_name, lined)
+
+        # Add wrapper with optional caption and link
+        code = '<figure class="code">'
+        if self.options:
+            caption = ('<span>%s</span>' % self.options['caption']) if 'caption' in self.options else ''
+            title = self.options['title'] if 'title' in self.options else 'link'
+            link = ('<a href="%s">%s</a>' % (self.options['url'], title)) if 'url' in self.options else ''
+
+            if caption or link:
+                code += '<figcaption>%s %s</figcaption>' % (caption, link)
+        code += '%s</figure>' % table
+
+        # Write cache
+        if cache_file is None:
+            cache_file = open(cached_path, 'w')
+            cache_file.write(parsed)
+        cache_file.close()
+
+        return [nodes.raw('', code, format='html')]
+
+directives.register_directive('code-block', Pygments)
+directives.register_directive('sourcecode', Pygments)
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/rst2html.py b/supporting-docs/_plugins/jekyll-rst-master/rst2html.py
new file mode 100644
index 00000000..0190da05
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/rst2html.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env python
+
+# :Author: David Goodger, the Pygments team, Guenter Milde
+# :Date: $Date: $
+# :Copyright: This module has been placed in the public domain.
+
+# This is a merge of the `Docutils`_ `rst2html` front end with an extension
+# suggestion taken from the `Pygments`_ documentation, reworked specifically
+# for `Octopress`_.
+#
+# .. _Pygments: http://pygments.org/
+# .. _Docutils: http://docutils.sourceforge.net/
+# .. _Octopress: http://octopress.org/
+
+"""
+A front end to docutils, producing HTML with syntax colouring using pygments
+"""
+
+try:
+    import locale
+    locale.setlocale(locale.LC_ALL, '')
+except:
+    pass
+
+from transform import transform
+from docutils.writers.html4css1 import Writer
+from docutils.core import default_description
+from directives import Pygments
+
+description = ('Generates (X)HTML documents from standalone reStructuredText '
+               'sources. Uses `pygments` to colorize the content of'
+               '"code-block" directives. Needs an adapted stylesheet' 
+               + default_description)
+
+def main():
+    return transform(writer=Writer(), part='html_body')
+
+if __name__ == '__main__':
+    print(main())
diff --git a/supporting-docs/_plugins/jekyll-rst-master/transform.py b/supporting-docs/_plugins/jekyll-rst-master/transform.py
new file mode 100644
index 00000000..51ad4ef6
--- /dev/null
+++ b/supporting-docs/_plugins/jekyll-rst-master/transform.py
@@ -0,0 +1,42 @@
+import sys
+from docutils.core import publish_parts
+from optparse import OptionParser
+from docutils.frontend import OptionParser as DocutilsOptionParser
+from docutils.parsers.rst import Parser
+
+def transform(writer=None, part=None):
+    p = OptionParser(add_help_option=False)
+    
+    # Collect all the command line options
+    docutils_parser = DocutilsOptionParser(components=(writer, Parser()))
+    for group in docutils_parser.option_groups:
+        p.add_option_group(group.title, None).add_options(group.option_list)
+    
+    p.add_option('--part', default=part)
+    
+    opts, args = p.parse_args()
+    
+    settings = dict({
+        'file_insertion_enabled': False,
+        'raw_enabled': False,
+        'doctitle_xform': False,
+    })
+    
+    if len(args) == 1:
+        try:
+            content = open(args[0], 'r').read()
+        except IOError:
+            content = args[0]
+    else:
+        content = sys.stdin.read()
+    
+    parts = publish_parts(
+        source=content,
+        writer=writer,
+        settings_overrides=settings
+    )
+    if 'html_body' in parts:
+      return parts['html_body']
+    if opts.part in parts:
+        return parts[opts.part]
+    return ''
diff --git a/supporting-docs/_plugins/project_version_tag.rb b/supporting-docs/_plugins/project_version_tag.rb
new file mode 100644
index 00000000..c647eef8
--- /dev/null
+++ b/supporting-docs/_plugins/project_version_tag.rb
@@ -0,0 +1,51 @@
+module Jekyll
+  class ProjectVersionTag < Liquid::Tag
+    NO_GIT_MESSAGE = 'Oops, are you sure this is a git project?'
+    UNABLE_TO_PARSE_MESSAGE = 'Sorry, could not read project version at the moment'
+
+    def render(context)
+      if git_repo?
+        current_version.chomp
+      else
+        NO_GIT_MESSAGE
+      end
+    end
+
+    private
+
+    def current_version
+      @_current_version ||= begin
+        # attempt to find the latest tag, falling back to last commit
+        version = git_describe || parse_head
+
+        version || UNABLE_TO_PARSE_MESSAGE
+      end
+    end
+
+    def git_describe
+      tagged_version = %x{ git describe --tags --always }
+
+      if command_succeeded?
+        tagged_version
+      end
+    end
+
+    def parse_head
+      head_commitish = %x{ git rev-parse --short HEAD }
+
+      if command_succeeded?
+        head_commitish
+      end
+    end
+
+    def command_succeeded?
+      !$?.nil? && $?.success?
+    end
+
+    def git_repo?
+      system('git rev-parse')
+    end
+  end
+end
+
+Liquid::Template.register_tag('project_version', Jekyll::ProjectVersionTag)
diff --git a/supporting-docs/_posts/2015-08-10-client-server.rst b/supporting-docs/_posts/2015-08-10-client-server.rst
new file mode 100644
index 00000000..2800584c
--- /dev/null
+++ b/supporting-docs/_posts/2015-08-10-client-server.rst
@@ -0,0 +1,652 @@
+---
+layout: post
+title: Client-Server API
+categories: guides
+---
+|
+
+.. figure:: http://matrix.org/matrix.png
+   :width: 305px
+   :height: 130px
+   :alt: [matrix]
+   :align: center
+
+
+How to use the client-server API
+================================
+
+.. NOTE::
+  The git version of this document is {% project_version %}
+
+This guide focuses on how the client-server APIs *provided by the reference 
+home server* can be used. Since this is specific to a home server 
+implementation, there may be variations in relation to registering/logging in
+which are not covered in extensive detail in this guide.
+
+If you haven't already, get a home server up and running on 
+``http://localhost:8008``.
+
+
+Accounts
+========
+Before you can send and receive messages, you must **register** for an account. 
+If you already have an account, you must **login** into it.
+
+.. NOTE::
+  `Try out the fiddle`__
+
+  .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/register_login
+
+Registration
+------------
+The aim of registration is to get a user ID and access token which you will need
+when accessing other APIs::
+
+    curl -XPOST -d '{"user":"example", "password":"wordpass", "type":"m.login.password"}' "http://localhost:8008/_matrix/client/api/v1/register"
+
+    {
+        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", 
+        "home_server": "localhost", 
+        "user_id": "@example:localhost"
+    }
+
+NB: If a ``user`` is not specified, one will be randomly generated for you. 
+If you do not specify a ``password``, you will be unable to login to the account
+if you forget the ``access_token``.
+
+Implementation note: The matrix specification does not enforce how users 
+register with a server. It just specifies the URL path and absolute minimum 
+keys. The reference home server uses a username/password to authenticate user,
+but other home servers may use different methods. This is why you need to
+specify the ``type`` of method.
+
+Login
+-----
+The aim when logging in is to get an access token for your existing user ID::
+
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/login"
+
+    {
+        "flows": [
+            {
+                "type": "m.login.password"
+            }
+        ]
+    }
+
+    curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/login"
+
+    {
+        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd", 
+        "home_server": "localhost", 
+        "user_id": "@example:localhost"
+    }
+    
+Implementation note: Different home servers may implement different methods for 
+logging in to an existing account. In order to check that you know how to login 
+to this home server, you must perform a ``GET`` first and make sure you 
+recognise the login type. If you do not know how to login, you can 
+``GET /login/fallback`` which will return a basic webpage which you can use to 
+login. The reference home server implementation support username/password login,
+but other home servers may support different login methods (e.g. OAuth2).
+
+
+Communicating
+=============
+
+In order to communicate with another user, you must **create a room** with that 
+user and **send a message** to that room. 
+
+.. NOTE::
+  `Try out the fiddle`__
+
+  .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/create_room_send_msg
+
+Creating a room
+---------------
+If you want to send a message to someone, you have to be in a room with them. To
+create a room::
+
+    curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token=YOUR_ACCESS_TOKEN"
+
+    {
+        "room_alias": "#tutorial:localhost", 
+        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+    }
+    
+The "room alias" is a human-readable string which can be shared with other users
+so they can join a room, rather than the room ID which is a randomly generated
+string. You can have multiple room aliases per room.
+
+.. TODO(kegan)
+  How to add/remove aliases from an existing room.
+    
+
+Sending messages
+----------------
+You can now send messages to this room::
+
+    curl -XPOST -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/send/m.room.message?access_token=YOUR_ACCESS_TOKEN"
+    
+    {
+        "event_id": "YUwRidLecu"
+    }
+    
+The event ID returned is a unique ID which identifies this message.
+    
+NB: There are no limitations to the types of messages which can be exchanged.
+The only requirement is that ``"msgtype"`` is specified. The Matrix 
+specification outlines the following standard types: ``m.text``, ``m.image``,
+``m.audio``, ``m.video``, ``m.location``, ``m.emote``. See the specification for
+more information on these types.
+
+Users and rooms
+===============
+
+Each room can be configured to allow or disallow certain rules. In particular,
+these rules may specify if you require an **invitation** from someone already in
+the room in order to **join the room**. In addition, you may also be able to 
+join a room **via a room alias** if one was set up.
+
+.. NOTE::
+  `Try out the fiddle`__
+
+  .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/room_memberships
+
+Inviting a user to a room
+-------------------------
+You can directly invite a user to a room like so::
+
+    curl -XPOST -d '{"user_id":"@myfriend:localhost"}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/invite?access_token=YOUR_ACCESS_TOKEN"
+    
+This informs ``@myfriend:localhost`` of the room ID 
+``!CvcvRuDYDzTOzfKKgh:localhost`` and allows them to join the room.
+
+Joining a room via an invite
+----------------------------
+If you receive an invite, you can join the room::
+
+    curl -XPOST -d '{}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/join?access_token=YOUR_ACCESS_TOKEN"
+    
+NB: Only the person invited (``@myfriend:localhost``) can change the membership
+state to ``"join"``. Repeatedly joining a room does nothing.
+
+Joining a room via an alias
+---------------------------
+Alternatively, if you know the room alias for this room and the room config 
+allows it, you can directly join a room via the alias::
+
+    curl -XPOST -d '{}' "http://localhost:8008/_matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=YOUR_ACCESS_TOKEN"
+    
+    {
+        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+    }
+    
+You will need to use the room ID when sending messages, not the room alias.
+
+NB: If the room is configured to be an invite-only room, you will still require
+an invite in order to join the room even though you know the room alias. As a
+result, it is more common to see a room alias in relation to a public room, 
+which do not require invitations.
+
+Getting events
+==============
+An event is some interesting piece of data that a client may be interested in. 
+It can be a message in a room, a room invite, etc. There are many different ways
+of getting events, depending on what the client already knows.
+
+.. NOTE::
+  `Try out the fiddle`__
+
+  .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/event_stream
+
+Getting all state
+-----------------
+If the client doesn't know any information on the rooms the user is 
+invited/joined on, they can get all the user's state for all rooms::
+
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=YOUR_ACCESS_TOKEN"
+    
+    {
+        "end": "s39_18_0", 
+        "presence": [
+            {
+                "content": {
+                    "last_active_ago": 1061436, 
+                    "user_id": "@example:localhost"
+                }, 
+                "type": "m.presence"
+            }
+        ], 
+        "rooms": [
+            {
+                "membership": "join", 
+                "messages": {
+                    "chunk": [
+                        {
+                            "content": {
+                                "@example:localhost": 10, 
+                                "default": 0
+                            }, 
+                            "event_id": "wAumPSTsWF", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.power_levels", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "join_rule": "public"
+                            }, 
+                            "event_id": "jrLVqKHKiI", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.join_rules", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "level": 10
+                            }, 
+                            "event_id": "WpmTgsNWUZ", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.add_state_level", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "level": 0
+                            }, 
+                            "event_id": "qUMBJyKsTQ", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.send_event_level", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "ban_level": 5, 
+                                "kick_level": 5
+                            }, 
+                            "event_id": "YAaDmKvoUW", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.ops_levels", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join"
+                            }, 
+                            "event_id": "RJbPMtCutf", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409665586730, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "body": "hello", 
+                                "hsob_ts": 1409665660439, 
+                                "msgtype": "m.text"
+                            }, 
+                            "event_id": "YUwRidLecu", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "ts": 1409665660439, 
+                            "type": "m.room.message", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "membership": "invite"
+                            }, 
+                            "event_id": "YjNuBKnPsb", 
+                            "membership": "invite", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@myfriend:localhost", 
+                            "ts": 1409666426819, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join", 
+                                "prev": "join"
+                            }, 
+                            "event_id": "KWwdDjNZnm", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409666551582, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join"
+                            }, 
+                            "event_id": "JFLVteSvQc", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409666587265, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }
+                    ], 
+                    "end": "s39_18_0", 
+                    "start": "t1-11_18_0"
+                }, 
+                "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                "state": [
+                    {
+                        "content": {
+                            "creator": "@example:localhost"
+                        }, 
+                        "event_id": "dMUoqVTZca", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.create", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "@example:localhost": 10, 
+                            "default": 0
+                        }, 
+                        "event_id": "wAumPSTsWF", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.power_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "join_rule": "public"
+                        }, 
+                        "event_id": "jrLVqKHKiI", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.join_rules", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 10
+                        }, 
+                        "event_id": "WpmTgsNWUZ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.add_state_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 0
+                        }, 
+                        "event_id": "qUMBJyKsTQ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.send_event_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "ban_level": 5, 
+                            "kick_level": 5
+                        }, 
+                        "event_id": "YAaDmKvoUW", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.ops_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "membership": "invite"
+                        }, 
+                        "event_id": "YjNuBKnPsb", 
+                        "membership": "invite", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@myfriend:localhost", 
+                        "ts": 1409666426819, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "avatar_url": null, 
+                            "displayname": null, 
+                            "membership": "join"
+                        }, 
+                        "event_id": "JFLVteSvQc", 
+                        "membership": "join", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@example:localhost", 
+                        "ts": 1409666587265, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
+                    }
+                ]
+            }
+        ]
+    }
+    
+This returns all the room information the user is invited/joined on, as well as
+all of the presences relevant for these rooms. This can be a LOT of data. You
+may just want the most recent event for each room. This can be achieved by 
+applying query parameters to ``limit`` this request::
+
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?limit=1&access_token=YOUR_ACCESS_TOKEN"
+    
+    {
+        "end": "s39_18_0", 
+        "presence": [
+            {
+                "content": {
+                    "last_active_ago": 1279484, 
+                    "user_id": "@example:localhost"
+                }, 
+                "type": "m.presence"
+            }
+        ], 
+        "rooms": [
+            {
+                "membership": "join", 
+                "messages": {
+                    "chunk": [
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join"
+                            }, 
+                            "event_id": "JFLVteSvQc", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409666587265, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }
+                    ], 
+                    "end": "s39_18_0", 
+                    "start": "t10-30_18_0"
+                }, 
+                "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                "state": [
+                    {
+                        "content": {
+                            "creator": "@example:localhost"
+                        }, 
+                        "event_id": "dMUoqVTZca", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.create", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "@example:localhost": 10, 
+                            "default": 0
+                        }, 
+                        "event_id": "wAumPSTsWF", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.power_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "join_rule": "public"
+                        }, 
+                        "event_id": "jrLVqKHKiI", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.join_rules", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 10
+                        }, 
+                        "event_id": "WpmTgsNWUZ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.add_state_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 0
+                        }, 
+                        "event_id": "qUMBJyKsTQ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.send_event_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "ban_level": 5, 
+                            "kick_level": 5
+                        }, 
+                        "event_id": "YAaDmKvoUW", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.ops_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "membership": "invite"
+                        }, 
+                        "event_id": "YjNuBKnPsb", 
+                        "membership": "invite", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@myfriend:localhost", 
+                        "ts": 1409666426819, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "avatar_url": null, 
+                            "displayname": null, 
+                            "membership": "join"
+                        }, 
+                        "event_id": "JFLVteSvQc", 
+                        "membership": "join", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@example:localhost", 
+                        "ts": 1409666587265, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
+                    }
+                ]
+            }
+        ]
+    }
+
+Getting live state
+------------------
+Once you know which rooms the client has previously interacted with, you need to
+listen for incoming events. This can be done like so::
+
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/events?access_token=YOUR_ACCESS_TOKEN"
+    
+    {
+        "chunk": [], 
+        "end": "s39_18_0", 
+        "start": "s39_18_0"
+    }
+    
+This will block waiting for an incoming event, timing out after several seconds.
+Even if there are no new events (as in the example above), there will be some
+pagination stream response keys. The client should make subsequent requests 
+using the value of the ``"end"`` key (in this case ``s39_18_0``) as the ``from`` 
+query parameter e.g. ``http://localhost:8008/_matrix/client/api/v1/events?access
+_token=YOUR_ACCESS_TOKEN&from=s39_18_0``. This value should be stored so when the 
+client reopens your app after a period of inactivity, you can resume from where 
+you got up to in the event stream. If it has been a long period of inactivity, 
+there may be LOTS of events waiting for the user. In this case, you may wish to 
+get all state instead and then resume getting live state from a newer end token.
+
+NB: The timeout can be changed by adding a ``timeout`` query parameter, which is
+in milliseconds. A timeout of 0 will not block.
+
+
+Example application
+-------------------
+The following example demonstrates registration and login, live event streaming,
+creating and joining rooms, sending messages, getting member lists and getting 
+historical messages for a room. This covers most functionality of a messaging
+application.
+
+.. NOTE::
+  `Try out the fiddle`__
+
+  .. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/matrix-doc/tree/master/supporting-docs/howtos/jsfiddles/example_app
diff --git a/supporting-docs/_posts/2015-08-14-getting_involved.md b/supporting-docs/_posts/2015-08-14-getting_involved.md
new file mode 100644
index 00000000..9d655618
--- /dev/null
+++ b/supporting-docs/_posts/2015-08-14-getting_involved.md
@@ -0,0 +1,120 @@
+---
+layout: post
+title: Getting involved
+categories: guides
+---
+
+# How can I get involved?
+Matrix is an ecosystem consisting of several apps written by lots of people. We at Matrix.org have written one server and a few clients, and people in the community have also written several clients, servers, and Application Services. We are collecting [a list of all known Matrix-apps](https://matrix.org/blog/try-matrix-now/).
+
+|
+
+You have a few options when it comes to getting involved: if you just want to use Matrix, you can [register an account on a public server using a public webclient](#reg). If you have a virtual private server (VPS) or similar, you might want to [run a server and/or client yourself](#run). If you want to look under the hood, you can [checkout the code and modify it - or write your own client or server](#checkout). Or you can write an [Application Service](#as), for example a bridge to an existing ecosystem.
+
+|
+
+<a class="anchor" id="reg"></a>
+
+## Access Matrix via a public webclient/server
+
+The easiest thing to do if you want to just have a play, is to use our reference webclient and create a user on the matrix.org homeserver. You do that by visiting http://matrix.org/beta/, selecting "Create account" and choosing your userID and password on the next page. You can also add your email, but this is optional (adding it will make it easier for your friends to find your Matrix user in the future). 
+
+| 
+
+At the bottom of the account creation page, you can pick the homeserver and identity server you want to use. In this case, we are using matrix.org's homeserver, so just leave it as-is. Your full Matrix-userID will be formed partly from the hostname your server is running as, and partly from an userID you specify when you create the account. For example, if you put bob as your userID, your full Matrix-userID will be @bob:matrix.org ("bob on matrix.org"). 
+
+| 
+
+You can use multiple clients with the same user, so you might want to also look at our [Android](https://github.com/matrix-org/matrix-android-console) or [iOS](https://github.com/matrix-org/matrix-ios-console) clients for your mobile phone!
+
+|
+
+<a class="anchor" id="run"></a>
+
+## Run a server and/or client yourself
+
+### Running your own client:
+
+You can run your own Matrix client; there are [numerous clients available](https://matrix.org/blog/try-matrix-now/). You can take Matrix.org's [reference client](https://github.com/matrix-org/matrix-angular-sdk) and use it as-is - or modify it any way you want! Since it's written in JavaScript, running a client is [really easy](https://github.com/matrix-org/matrix-angular-sdk#running)!
+
+|
+
+### Running your own homeserver:
+
+One of the core features of Matrix is that anyone can run a homeserver and join the federated network on equal terms (there is no hierarchy). If you want to set up your own homeserver, please see the relevant docs of the server you want to run. If you want to run Matrix.org's reference homeserver, please consult the [readme of the Synapse project](https://github.com/matrix-org/synapse/blob/master/README.rst).
+
+|
+
+Note that Synapse comes with a bundled Matrix.org webclient - but you can tell it to use your [development checkout snapshot instead](https://github.com/matrix-org/matrix-angular-sdk#matrix-angular-sdk) (or to not run a webclient at all via the "web_client: false" config option).
+
+|
+
+<a class="anchor" id="checkout"></a>
+
+## Checkout our code - or write your own
+
+As described above, you can clone our code and [run a server and/or client yourself](#run). Infact, all the code that we at Matrix.org write is available from [our github](http://github.com/matrix-org) - and other servers and clients may also be open sourced - see [our list of all known Matrix-apps](https://matrix.org/blog/try-matrix-now/).
+
+|
+
+You can also implement your own client or server - after all, Matrix is at its core "just" a specification of a protocol.
+
+|
+
+### Write your own client:
+
+The [client-server API spec](http://matrix.org/docs/howtos/client-server.html) describes what API calls are available to clients, but a quick step-by-step guide would include:
+
+|
+
+1. Get a user either by registering your user in an existing client or running the [new-user script](https://github.com/matrix-org/synapse/blob/master/scripts/register_new_matrix_user) if you are running your own Synapse homeserver.
+
+2. Assuming the homeserver you are using allows logins by password, log in via the login API: 
+```
+curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/login"
+```
+3. If successful, this returns the following, including an `access_token`:
+
+        {
+            "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd",
+            "home_server": "localhost",
+            "user_id": "@example:localhost"
+        }
+
+4. This ``access_token`` will be used for authentication for the rest of your API calls. Potentially the next step you want is to make a call to the initialSync API and get the last x events from each room your user is in (via the limit parameter):
+```
+curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?limit=1&access_token=YOUR_ACCESS_TOKEN"
+```
+
+5. In addition to the last x events for all the rooms the user is interested in, this returns all the presences relevant for these rooms. Once you know which rooms the client has previously interacted with, you need to listen for incoming events. This can be done like so:
+```
+curl -XGET "http://localhost:8008/_matrix/client/api/v1/events?access_token=YOUR_ACCESS_TOKEN"
+```
+
+6. This request will block waiting for an incoming event, timing out after several seconds if there is no event, returning something like this:
+
+        {
+            "chunk": [],
+            "end": "s39_18_0",
+            "start": "s39_18_0"
+        }
+7. Even if there are no new events (as in the example above), there will be some pagination stream response keys. The client should make subsequent requests using the value of the "end" key (in this case s39_18_0) as the from query parameter e.g.
+```
+http://localhost:8008/_matrix/client/api/v1/events?access _token=YOUR_ACCESS_TOKEN&from=s39_18_0
+```
+
+8. This ensures that you only get new events. Now you have initial rooms and presence, and a stream of events - a good client should be able to process all these events and present them to the user. And potentially you might want to add functionality to generate events as well (such as messages from the user, for example) - again please consult the [client-server API spec](http://matrix.org/docs/howtos/client-server.html)!
+
+### Write your own server:
+
+We are still working on the server-server spec, so the best thing to do if you are interested in writing a server, is to come talk to us in [#matrix:matrix.org](https://matrix.org/beta/#/room/%23matrix:matrix.org).
+ 
+If you are interested in how federation works, please see the [federation API chapter in the spec](http://matrix.org/docs/spec/#federation-api).
+
+|
+
+<a class="anchor" id="as"></a>
+
+## Write an Application Service:
+
+One of our core developers, Kegan, wrote an excellent [blogpost about the Application Services API](https://matrix.org/blog/2015/03/02/introduction-to-application-services/), as well as a [blogpost about the Matrix<->IRC Application Service](https://matrix.org/blog/2015/04/22/matrix-irc-application-service/) - these are the best resources to consult if you want to write an Application Service yourself.
diff --git a/supporting-docs/_posts/index.html b/supporting-docs/_posts/index.html
new file mode 100644
index 00000000..c442d6e1
--- /dev/null
+++ b/supporting-docs/_posts/index.html
@@ -0,0 +1,19 @@
+---
+layout: default
+---
+  {% include site.css %}
+  {% include basic.css %}
+  {% include nature.css %}
+
+<div class="post">
+
+  <header class="post-header">
+    <div class="post-title">{{ page.title }}</div>
+    <p class="post-meta">{{ page.date | date: "%b %-d, %Y" }}{% if page.author %} • {{ page.author }}{% endif %}{% if page.meta %} • {{ page.meta }}{% endif %}</p>
+  </header>
+
+  <article class="post-content">
+    {{ content }}
+  </article>
+
+</div>
diff --git a/supporting-docs/_sass/_base.scss b/supporting-docs/_sass/_base.scss
new file mode 100644
index 00000000..e5fd0fd8
--- /dev/null
+++ b/supporting-docs/_sass/_base.scss
@@ -0,0 +1,204 @@
+/**
+ * Reset some basic elements
+ */
+body, h1, h2, h3, h4, h5, h6,
+p, blockquote, pre, hr,
+dl, dd, ol, ul, figure {
+    margin: 0;
+    padding: 0;
+}
+
+
+
+/**
+ * Basic styling
+ */
+body {
+    font-family: $base-font-family;
+    font-size: $base-font-size;
+    line-height: $base-line-height;
+    font-weight: 300;
+    color: $text-color;
+    background-color: $background-color;
+    -webkit-text-size-adjust: 100%;
+}
+
+
+
+/**
+ * Set `margin-bottom` to maintain vertical rhythm
+ */
+h1, h2, h3, h4, h5, h6,
+p, blockquote, pre,
+ul, ol, dl, figure,
+%vertical-rhythm {
+    margin-bottom: $spacing-unit / 2;
+}
+
+
+
+/**
+ * Images
+ */
+img {
+    max-width: 100%;
+    vertical-align: middle;
+}
+
+
+
+/**
+ * Figures
+ */
+figure > img {
+    display: block;
+}
+
+figcaption {
+    font-size: $small-font-size;
+}
+
+
+
+/**
+ * Lists
+ */
+ul, ol {
+    margin-left: $spacing-unit;
+}
+
+li {
+    > ul,
+    > ol {
+         margin-bottom: 0;
+    }
+}
+
+
+
+/**
+ * Headings
+ */
+h1, h2, h3, h4, h5, h6 {
+    font-weight: 300;
+}
+
+
+
+/**
+ * Links
+ */
+a {
+    color: $brand-color;
+    text-decoration: none;
+
+    &:visited {
+        color: darken($brand-color, 15%);
+    }
+
+    &:hover {
+        color: $text-color;
+        text-decoration: underline;
+    }
+}
+
+
+
+/**
+ * Blockquotes
+ */
+blockquote {
+    color: $grey-color;
+    border-left: 4px solid $grey-color-light;
+    padding-left: $spacing-unit / 2;
+    font-size: 18px;
+    letter-spacing: -1px;
+    font-style: italic;
+
+    > :last-child {
+        margin-bottom: 0;
+    }
+}
+
+
+
+/**
+ * Code formatting
+ */
+pre,
+code {
+    font-size: 15px;
+    border: 1px solid $grey-color-light;
+    border-radius: 3px;
+    background-color: #eef;
+}
+
+code {
+    padding: 1px 5px;
+}
+
+pre {
+    padding: 8px 12px;
+    overflow-x: scroll;
+
+    > code {
+        border: 0;
+        padding-right: 0;
+        padding-left: 0;
+    }
+}
+
+
+
+/**
+ * Wrapper
+ */
+.wrapper {
+    max-width: -webkit-calc(#{$content-width} - (#{$spacing-unit} * 2));
+    max-width:         calc(#{$content-width} - (#{$spacing-unit} * 2));
+    margin-right: auto;
+    margin-left: auto;
+    padding-right: $spacing-unit;
+    padding-left: $spacing-unit;
+    @extend %clearfix;
+
+    @include media-query($on-laptop) {
+        max-width: -webkit-calc(#{$content-width} - (#{$spacing-unit}));
+        max-width:         calc(#{$content-width} - (#{$spacing-unit}));
+        padding-right: $spacing-unit / 2;
+        padding-left: $spacing-unit / 2;
+    }
+}
+
+
+
+/**
+ * Clearfix
+ */
+%clearfix {
+
+    &:after {
+        content: "";
+        display: table;
+        clear: both;
+    }
+}
+
+
+
+/**
+ * Icons
+ */
+.icon {
+
+    > svg {
+        display: inline-block;
+        width: 16px;
+        height: 16px;
+        vertical-align: middle;
+
+        path {
+            fill: $grey-color;
+        }
+    }
+}
diff --git a/supporting-docs/_sass/_layout.scss b/supporting-docs/_sass/_layout.scss
new file mode 100644
index 00000000..def56f89
--- /dev/null
+++ b/supporting-docs/_sass/_layout.scss
@@ -0,0 +1,236 @@
+/**
+ * Site header
+ */
+.site-header {
+    border-top: 5px solid $grey-color-dark;
+    border-bottom: 1px solid $grey-color-light;
+    min-height: 56px;
+
+    // Positioning context for the mobile navigation icon
+    position: relative;
+}
+
+.site-title {
+    font-size: 26px;
+    line-height: 56px;
+    letter-spacing: -1px;
+    margin-bottom: 0;
+    float: left;
+
+    &,
+    &:visited {
+        color: $grey-color-dark;
+    }
+}
+
+.site-nav {
+    float: right;
+    line-height: 56px;
+
+    .menu-icon {
+        display: none;
+    }
+
+    .page-link {
+        color: $text-color;
+        line-height: $base-line-height;
+
+        // Gaps between nav items, but not on the first one
+        &:not(:first-child) {
+            margin-left: 20px;
+        }
+    }
+
+    @include media-query($on-palm) {
+        position: absolute;
+        top: 9px;
+        right: 30px;
+        background-color: $background-color;
+        border: 1px solid $grey-color-light;
+        border-radius: 5px;
+        text-align: right;
+
+        .menu-icon {
+            display: block;
+            float: right;
+            width: 36px;
+            height: 26px;
+            line-height: 0;
+            padding-top: 10px;
+            text-align: center;
+
+            > svg {
+                width: 18px;
+                height: 15px;
+
+                path {
+                    fill: $grey-color-dark;
+                }
+            }
+        }
+
+        .trigger {
+            clear: both;
+            display: none;
+        }
+
+        &:hover .trigger {
+            display: block;
+            padding-bottom: 5px;
+        }
+
+        .page-link {
+            display: block;
+            padding: 5px 10px;
+        }
+    }
+}
+
+
+
+/**
+ * Site footer
+ */
+.site-footer {
+    border-top: 1px solid $grey-color-light;
+    padding: $spacing-unit 0;
+}
+
+.footer-heading {
+    font-size: 18px;
+    margin-bottom: $spacing-unit / 2;
+}
+
+.contact-list,
+.social-media-list {
+    list-style: none;
+    margin-left: 0;
+}
+
+.footer-col-wrapper {
+    font-size: 15px;
+    color: $grey-color;
+    margin-left: -$spacing-unit / 2;
+    @extend %clearfix;
+}
+
+.footer-col {
+    float: left;
+    margin-bottom: $spacing-unit / 2;
+    padding-left: $spacing-unit / 2;
+}
+
+.footer-col-1 {
+    width: -webkit-calc(35% - (#{$spacing-unit} / 2));
+    width:         calc(35% - (#{$spacing-unit} / 2));
+}
+
+.footer-col-2 {
+    width: -webkit-calc(20% - (#{$spacing-unit} / 2));
+    width:         calc(20% - (#{$spacing-unit} / 2));
+}
+
+.footer-col-3 {
+    width: -webkit-calc(45% - (#{$spacing-unit} / 2));
+    width:         calc(45% - (#{$spacing-unit} / 2));
+}
+
+@include media-query($on-laptop) {
+    .footer-col-1,
+    .footer-col-2 {
+        width: -webkit-calc(50% - (#{$spacing-unit} / 2));
+        width:         calc(50% - (#{$spacing-unit} / 2));
+    }
+
+    .footer-col-3 {
+        width: -webkit-calc(100% - (#{$spacing-unit} / 2));
+        width:         calc(100% - (#{$spacing-unit} / 2));
+    }
+}
+
+@include media-query($on-palm) {
+    .footer-col {
+        float: none;
+        width: -webkit-calc(100% - (#{$spacing-unit} / 2));
+        width:         calc(100% - (#{$spacing-unit} / 2));
+    }
+}
+
+
+
+/**
+ * Page content
+ */
+.page-content {
+    padding: $spacing-unit 0;
+}
+
+.page-heading {
+    font-size: 20px;
+}
+
+.post-list {
+    margin-left: 0;
+    list-style: none;
+
+    > li {
+        margin-bottom: $spacing-unit;
+    }
+}
+
+.post-meta {
+    font-size: $small-font-size;
+    color: $grey-color;
+}
+
+.post-link {
+    display: block;
+    font-size: 24px;
+}
+
+
+
+/**
+ * Posts
+ */
+.post-header {
+    margin-bottom: $spacing-unit;
+}
+
+.post-title {
+    font-size: 42px;
+    letter-spacing: -1px;
+    line-height: 1;
+
+    @include media-query($on-laptop) {
+        font-size: 36px;
+    }
+}
+
+.post-content {
+    margin-bottom: $spacing-unit;
+
+    h2 {
+        font-size: 32px;
+
+        @include media-query($on-laptop) {
+            font-size: 28px;
+        }
+    }
+
+    h3 {
+        font-size: 26px;
+
+        @include media-query($on-laptop) {
+            font-size: 22px;
+        }
+    }
+
+    h4 {
+        font-size: 20px;
+
+        @include media-query($on-laptop) {
+            font-size: 18px;
+        }
+    }
+}
diff --git a/supporting-docs/_sass/_syntax-highlighting.scss b/supporting-docs/_sass/_syntax-highlighting.scss
new file mode 100644
index 00000000..e36627da
--- /dev/null
+++ b/supporting-docs/_sass/_syntax-highlighting.scss
@@ -0,0 +1,67 @@
+/**
+ * Syntax highlighting styles
+ */
+.highlight {
+    background: #fff;
+    @extend %vertical-rhythm;
+
+    .c     { color: #998; font-style: italic } // Comment
+    .err   { color: #a61717; background-color: #e3d2d2 } // Error
+    .k     { font-weight: bold } // Keyword
+    .o     { font-weight: bold } // Operator
+    .cm    { color: #998; font-style: italic } // Comment.Multiline
+    .cp    { color: #999; font-weight: bold } // Comment.Preproc
+    .c1    { color: #998; font-style: italic } // Comment.Single
+    .cs    { color: #999; font-weight: bold; font-style: italic } // Comment.Special
+    .gd    { color: #000; background-color: #fdd } // Generic.Deleted
+    .gd .x { color: #000; background-color: #faa } // Generic.Deleted.Specific
+    .ge    { font-style: italic } // Generic.Emph
+    .gr    { color: #a00 } // Generic.Error
+    .gh    { color: #999 } // Generic.Heading
+    .gi    { color: #000; background-color: #dfd } // Generic.Inserted
+    .gi .x { color: #000; background-color: #afa } // Generic.Inserted.Specific
+    .go    { color: #888 } // Generic.Output
+    .gp    { color: #555 } // Generic.Prompt
+    .gs    { font-weight: bold } // Generic.Strong
+    .gu    { color: #aaa } // Generic.Subheading
+    .gt    { color: #a00 } // Generic.Traceback
+    .kc    { font-weight: bold } // Keyword.Constant
+    .kd    { font-weight: bold } // Keyword.Declaration
+    .kp    { font-weight: bold } // Keyword.Pseudo
+    .kr    { font-weight: bold } // Keyword.Reserved
+    .kt    { color: #458; font-weight: bold } // Keyword.Type
+    .m     { color: #099 } // Literal.Number
+    .s     { color: #d14 } // Literal.String
+    .na    { color: #008080 } // Name.Attribute
+    .nb    { color: #0086B3 } // Name.Builtin
+    .nc    { color: #458; font-weight: bold } // Name.Class
+    .no    { color: #008080 } // Name.Constant
+    .ni    { color: #800080 } // Name.Entity
+    .ne    { color: #900; font-weight: bold } // Name.Exception
+    .nf    { color: #900; font-weight: bold } // Name.Function
+    .nn    { color: #555 } // Name.Namespace
+    .nt    { color: #000080 } // Name.Tag
+    .nv    { color: #008080 } // Name.Variable
+    .ow    { font-weight: bold } // Operator.Word
+    .w     { color: #bbb } // Text.Whitespace
+    .mf    { color: #099 } // Literal.Number.Float
+    .mh    { color: #099 } // Literal.Number.Hex
+    .mi    { color: #099 } // Literal.Number.Integer
+    .mo    { color: #099 } // Literal.Number.Oct
+    .sb    { color: #d14 } // Literal.String.Backtick
+    .sc    { color: #d14 } // Literal.String.Char
+    .sd    { color: #d14 } // Literal.String.Doc
+    .s2    { color: #d14 } // Literal.String.Double
+    .se    { color: #d14 } // Literal.String.Escape
+    .sh    { color: #d14 } // Literal.String.Heredoc
+    .si    { color: #d14 } // Literal.String.Interpol
+    .sx    { color: #d14 } // Literal.String.Other
+    .sr    { color: #009926 } // Literal.String.Regex
+    .s1    { color: #d14 } // Literal.String.Single
+    .ss    { color: #990073 } // Literal.String.Symbol
+    .bp    { color: #999 } // Name.Builtin.Pseudo
+    .vc    { color: #008080 } // Name.Variable.Class
+    .vg    { color: #008080 } // Name.Variable.Global
+    .vi    { color: #008080 } // Name.Variable.Instance
+    .il    { color: #099 } // Literal.Number.Integer.Long
+}
diff --git a/supporting-docs/about.md b/supporting-docs/about.md
new file mode 100644
index 00000000..3ed64bb6
--- /dev/null
+++ b/supporting-docs/about.md
@@ -0,0 +1,11 @@
+---
+layout: page
+title: About
+permalink: /about/
+---
+
+This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at [jekyllrb.com](http://jekyllrb.com/)
+
+You can find the source code for the Jekyll new theme at: [github.com/jglovier/jekyll-new](https://github.com/jglovier/jekyll-new)
+
+You can find the source code for Jekyll at [github.com/jekyll/jekyll](https://github.com/jekyll/jekyll)
diff --git a/supporting-docs/css/basic.css b/supporting-docs/css/basic.css
new file mode 100644
index 00000000..bf8f4712
--- /dev/null
+++ b/supporting-docs/css/basic.css
@@ -0,0 +1,512 @@
+/*
+ * basic.css
+ * ~~~~~~~~~
+ *
+ * Sphinx stylesheet -- basic theme.
+ *
+ * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/* -- main layout ----------------------------------------------------------- */
+
+div.clearer {
+    clear: both;
+}
+
+/* -- relbar ---------------------------------------------------------------- */
+
+div.related {
+    width: 100%;
+    font-size: 90%;
+}
+
+div.related h3 {
+    display: none;
+}
+
+div.related ul {
+    margin: 0;
+    padding: 0 0 0 10px;
+    list-style: none;
+}
+
+div.related li {
+    display: inline;
+}
+
+div.related li.right {
+    float: right;
+    margin-right: 5px;
+}
+
+/* -- sidebar --------------------------------------------------------------- */
+
+div.sphinxsidebarwrapper {
+    padding: 10px 5px 0 10px;
+}
+
+div.sphinxsidebar {
+    float: left;
+    width: 230px;
+    margin-left: -100%;
+    font-size: 90%;
+}
+
+div.sphinxsidebar ul {
+    list-style: none;
+}
+
+div.sphinxsidebar ul ul,
+div.sphinxsidebar ul.want-points {
+    margin-left: 20px;
+    list-style: square;
+}
+
+div.sphinxsidebar ul ul {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+div.sphinxsidebar form {
+    margin-top: 10px;
+}
+
+div.sphinxsidebar input {
+    border: 1px solid #98dbcc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+img {
+    border: 0;
+}
+
+/* -- search page ----------------------------------------------------------- */
+
+ul.search {
+    margin: 10px 0 0 20px;
+    padding: 0;
+}
+
+ul.search li {
+    padding: 5px 0 5px 20px;
+    background-image: url(file.png);
+    background-repeat: no-repeat;
+    background-position: 0 7px;
+}
+
+ul.search li a {
+    font-weight: bold;
+}
+
+ul.search li div.context {
+    color: #888;
+    margin: 2px 0 0 30px;
+    text-align: left;
+}
+
+ul.keywordmatches li.goodmatch a {
+    font-weight: bold;
+}
+
+/* -- index page ------------------------------------------------------------ */
+
+table.contentstable {
+    width: 90%;
+}
+
+table.contentstable p.biglink {
+    line-height: 150%;
+}
+
+a.biglink {
+    font-size: 1.3em;
+}
+
+span.linkdescr {
+    font-style: italic;
+    padding-top: 5px;
+    font-size: 90%;
+}
+
+/* -- general index --------------------------------------------------------- */
+
+table.indextable {
+    width: 100%;
+}
+
+table.indextable td {
+    text-align: left;
+    vertical-align: top;
+}
+
+table.indextable dl, table.indextable dd {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+table.indextable tr.pcap {
+    height: 10px;
+}
+
+table.indextable tr.cap {
+    margin-top: 10px;
+    background-color: #f2f2f2;
+}
+
+img.toggler {
+    margin-right: 3px;
+    margin-top: 3px;
+    cursor: pointer;
+}
+
+div.modindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+div.genindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+/* -- general body styles --------------------------------------------------- */
+
+a.headerlink {
+    visibility: hidden;
+}
+
+h1:hover > a.headerlink,
+h2:hover > a.headerlink,
+h3:hover > a.headerlink,
+h4:hover > a.headerlink,
+h5:hover > a.headerlink,
+h6:hover > a.headerlink,
+dt:hover > a.headerlink {
+    visibility: visible;
+}
+
+div.document p.caption {
+    text-align: inherit;
+}
+
+div.document td {
+    text-align: left;
+}
+
+.field-list ul {
+    padding-left: 1em;
+}
+
+.first {
+    margin-top: 0 !important;
+}
+
+p.rubric {
+    margin-top: 30px;
+    font-weight: bold;
+}
+
+.align-left {
+    text-align: left;
+}
+
+.align-center {
+    clear: both;
+    text-align: center;
+}
+
+.align-right {
+    text-align: right;
+}
+
+/* -- sidebars -------------------------------------------------------------- */
+
+div.sidebar {
+    margin: 0 0 0.5em 1em;
+    border: 1px solid #ddb;
+    padding: 7px 7px 0 7px;
+    background-color: #ffe;
+    width: 40%;
+    float: right;
+}
+
+p.sidebar-title {
+    font-weight: bold;
+}
+
+/* -- topics ---------------------------------------------------------------- */
+
+div.topic {
+    border: 1px solid #ccc;
+    padding: 7px 7px 0 7px;
+    margin: 10px 0 10px 0;
+}
+
+p.topic-title {
+    font-size: 1.1em;
+    font-weight: bold;
+    margin-top: 10px;
+}
+
+/* -- admonitions ----------------------------------------------------------- */
+
+div.admonition {
+    margin-top: 10px;
+    margin-bottom: 10px;
+    padding: 7px;
+}
+
+div.admonition dt {
+    font-weight: bold;
+}
+
+div.admonition dl {
+    margin-bottom: 0;
+}
+
+p.admonition-title {
+    margin: 0px 10px 5px 0px;
+    font-weight: bold;
+}
+
+div.document p.centered {
+    text-align: center;
+    margin-top: 25px;
+}
+
+/* -- tables ---------------------------------------------------------------- */
+
+table.docutils {
+    border: 0;
+    border-collapse: collapse;
+}
+
+table.docutils td, table.docutils th {
+    padding: 1px 8px 1px 5px;
+    border-top: 0;
+    border-left: 0;
+    border-right: 0;
+    border-bottom: 1px solid #aaa;
+}
+
+table.field-list td, table.field-list th {
+    border: 0 !important;
+}
+
+table.footnote td, table.footnote th {
+    border: 0 !important;
+}
+
+th {
+    text-align: left;
+    padding-right: 5px;
+}
+
+table.citation {
+    border-left: solid 1px gray;
+    margin-left: 1px;
+}
+
+table.citation td {
+    border-bottom: none;
+}
+
+/* -- other body styles ----------------------------------------------------- */
+
+ol.arabic {
+    list-style: decimal;
+}
+
+ol.loweralpha {
+    list-style: lower-alpha;
+}
+
+ol.upperalpha {
+    list-style: upper-alpha;
+}
+
+ol.lowerroman {
+    list-style: lower-roman;
+}
+
+ol.upperroman {
+    list-style: upper-roman;
+}
+
+dl {
+    margin-bottom: 15px;
+}
+
+dd p {
+    margin-top: 0px;
+}
+
+dd ul, dd table {
+    margin-bottom: 10px;
+}
+
+dd {
+    margin-top: 3px;
+    margin-bottom: 10px;
+    margin-left: 30px;
+}
+
+dt:target, .highlighted {
+    background-color: #fbe54e;
+}
+
+dl.glossary dt {
+    font-weight: bold;
+    font-size: 1.1em;
+}
+
+.field-list ul {
+    margin: 0;
+    padding-left: 1em;
+}
+
+.field-list p {
+    margin: 0;
+}
+
+.refcount {
+    color: #060;
+}
+
+.optional {
+    font-size: 1.3em;
+}
+
+.versionmodified {
+    font-style: italic;
+}
+
+.system-message {
+    background-color: #fda;
+    padding: 5px;
+    border: 3px solid red;
+}
+
+.footnote:target  {
+    background-color: #ffa
+}
+
+.line-block {
+    display: block;
+    margin-top: 1em;
+    margin-bottom: 1em;
+}
+
+.line-block .line-block {
+    margin-top: 0;
+    margin-bottom: 0;
+    margin-left: 1.5em;
+}
+
+.guilabel, .menuselection {
+    font-family: sans-serif;
+}
+
+.accelerator {
+    text-decoration: underline;
+}
+
+.classifier {
+    font-style: oblique;
+}
+
+/* -- code displays --------------------------------------------------------- */
+
+pre {
+    overflow: auto;
+    font-size: 1.1em;
+
+}
+
+td.linenos pre {
+    padding: 5px 0px;
+    border: 0;
+    background-color: transparent;
+    color: #aaa;
+}
+
+table.highlighttable {
+    margin-left: 0.5em;
+}
+
+table.highlighttable td {
+    padding: 0 0.5em 0 0.5em;
+}
+
+tt.descname {
+    background-color: transparent;
+    font-weight: bold;
+    font-size: 1.2em;
+}
+
+tt.descclassname {
+    background-color: transparent;
+}
+
+tt.xref, a tt {
+    background-color: transparent;
+    font-weight: bold;
+}
+
+h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt {
+    background-color: transparent;
+}
+
+.viewcode-link {
+    float: right;
+}
+
+.viewcode-back {
+    float: right;
+    font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+    margin: -1px -10px;
+    padding: 0 10px;
+}
+
+/* -- math display ---------------------------------------------------------- */
+
+img.math {
+    vertical-align: middle;
+}
+
+div.document div.math p {
+    text-align: center;
+}
+
+span.eqno {
+    float: right;
+}
+
+/* -- printout stylesheet --------------------------------------------------- */
+
+@media print {
+    div.document,
+    div.documentwrapper,
+    div.bodywrapper {
+        margin: 0 !important;
+        width: 100%;
+    }
+
+    div.sphinxsidebar,
+    div.related,
+    div.footer,
+    #top-link {
+        display: none;
+    }
+}
+
diff --git a/supporting-docs/css/main.scss b/supporting-docs/css/main.scss
new file mode 100755
index 00000000..b8a3e2be
--- /dev/null
+++ b/supporting-docs/css/main.scss
@@ -0,0 +1,52 @@
+---
+# Only the main Sass file needs front matter (the dashes are enough)
+---
+@charset "utf-8";
+
+
+
+// Our variables
+$base-font-family: Helvetica, Arial, sans-serif;
+$base-font-size:   16px;
+$small-font-size:  $base-font-size * 0.875;
+$base-line-height: 1.5;
+
+$spacing-unit:     30px;
+
+$text-color:       #111;
+$background-color: #fdfdfd;
+$brand-color:      #2a7ae2;
+
+$grey-color:       #828282;
+$grey-color-light: lighten($grey-color, 40%);
+$grey-color-dark:  darken($grey-color, 25%);
+
+// Width of the content area
+$content-width:    1340px;
+
+$on-palm:          600px;
+$on-laptop:        800px;
+
+
+
+// Using media queries with like this:
+// @include media-query($on-palm) {
+//     .wrapper {
+//         padding-right: $spacing-unit / 2;
+//         padding-left: $spacing-unit / 2;
+//     }
+// }
+@mixin media-query($device) {
+    @media screen and (max-width: $device) {
+        @content;
+    }
+}
+
+
+
+// Import partials from `sass_dir` (defaults to `_sass`)
+@import
+        "base",
+        "layout",
+        "syntax-highlighting"
+;
diff --git a/supporting-docs/css/nature.css b/supporting-docs/css/nature.css
new file mode 100644
index 00000000..9bf583be
--- /dev/null
+++ b/supporting-docs/css/nature.css
@@ -0,0 +1,285 @@
+/*
+ * nature.css_t
+ * ~~~~~~~~~~~~
+ *
+ * Sphinx stylesheet -- nature theme.
+ *
+ * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+ 
+/* -- page layout ----------------------------------------------------------- */
+ 
+body {
+    font-family: Arial, sans-serif;
+    font-size: 100%;
+    /*background-color: #111;*/
+    color: #555;
+    margin: 0;
+    padding: 0;
+}
+
+div.documentwrapper {
+    float: left;
+    width: 100%;
+}
+
+div.bodywrapper {
+    margin: 0 0 0 230px;
+}
+
+hr {
+    border: 1px solid #B1B4B6;
+}
+ 
+/*
+div.document {
+    background-color: #eee;
+}
+*/
+ 
+div.document {
+    background-color: #ffffff;
+    color: #3E4349;
+    padding: 0 30px 30px 30px;
+    font-size: 0.9em;
+}
+ 
+div.footer {
+    color: #555;
+    width: 100%;
+    padding: 13px 0;
+    text-align: center;
+    font-size: 75%;
+}
+ 
+div.footer a {
+    color: #444;
+    text-decoration: underline;
+}
+ 
+div.related {
+    background-color: #6BA81E;
+    line-height: 32px;
+    color: #fff;
+    text-shadow: 0px 1px 0 #444;
+    font-size: 0.9em;
+}
+ 
+div.related a {
+    color: #E2F3CC;
+}
+ 
+div.sphinxsidebar {
+    font-size: 0.75em;
+    line-height: 1.5em;
+}
+
+div.sphinxsidebarwrapper{
+    padding: 20px 0;
+}
+ 
+div.sphinxsidebar h3,
+div.sphinxsidebar h4 {
+    font-family: Arial, sans-serif;
+    color: #222;
+    font-size: 1.2em;
+    font-weight: normal;
+    margin: 0;
+    padding: 5px 10px;
+    background-color: #ddd;
+    text-shadow: 1px 1px 0 white
+}
+
+div.sphinxsidebar h4{
+    font-size: 1.1em;
+}
+ 
+div.sphinxsidebar h3 a {
+    color: #444;
+}
+ 
+ 
+div.sphinxsidebar p {
+    color: #888;
+    padding: 5px 20px;
+}
+ 
+div.sphinxsidebar p.topless {
+}
+ 
+div.sphinxsidebar ul {
+    margin: 10px 20px;
+    padding: 0;
+    color: #000;
+}
+ 
+div.sphinxsidebar a {
+    color: #444;
+}
+ 
+div.sphinxsidebar input {
+    border: 1px solid #ccc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+div.sphinxsidebar input[type=text]{
+    margin-left: 20px;
+}
+ 
+/* -- body styles ----------------------------------------------------------- */
+ 
+a {
+    color: #005B81;
+    text-decoration: none;
+}
+ 
+a:hover {
+    color: #E32E00;
+    text-decoration: underline;
+}
+ 
+div.document h1,
+div.document h2,
+div.document h3,
+div.document h4,
+div.document h5,
+div.document h6 {
+    font-family: Arial, sans-serif;
+    background-color: #BED4EB;
+    font-weight: normal;
+    color: #212224;
+    margin: 30px 0px 10px 0px;
+    padding: 5px 0 5px 10px;
+    text-shadow: 0px 1px 0 white
+}
+ 
+div.document h1 { border-top: 20px solid white; margin-top: 0; font-size: 200%; }
+div.document h2 { font-size: 150%; background-color: #C8D5E3; }
+div.document h3 { font-size: 120%; background-color: #D8DEE3; }
+div.document h4 { font-size: 110%; background-color: #D8DEE3; }
+div.document h5 { font-size: 100%; background-color: #D8DEE3; }
+div.document h6 { font-size: 100%; background-color: #D8DEE3; }
+ 
+a.headerlink {
+    color: #c60f0f;
+    font-size: 0.8em;
+    padding: 0 4px 0 4px;
+    text-decoration: none;
+}
+ 
+a.headerlink:hover {
+    background-color: #c60f0f;
+    color: white;
+}
+ 
+div.document p, div.document dd, div.document li {
+    line-height: 1.5em;
+}
+ 
+div.admonition p.admonition-title + p {
+    display: inline;
+}
+
+div.highlight{
+    background-color: white;
+}
+
+div.note {
+    background-color: #eee;
+    border: 1px solid #ccc;
+}
+ 
+div.seealso {
+    background-color: #ffc;
+    border: 1px solid #ff6;
+}
+ 
+div.topic {
+    background-color: #eee;
+}
+ 
+div.warning {
+    background-color: #ffe4e4;
+    border: 1px solid #f66;
+}
+ 
+p.admonition-title {
+    display: inline;
+}
+ 
+p.admonition-title:after {
+    content: ":";
+}
+ 
+pre {
+    padding: 10px;
+    background-color: White;
+    color: #222;
+    line-height: 1.2em;
+    border: 1px solid #C6C9CB;
+    font-size: 1.1em;
+    margin: 1.5em 0 1.5em 0;
+    -webkit-box-shadow: 1px 1px 1px #d8d8d8;
+    -moz-box-shadow: 1px 1px 1px #d8d8d8;
+}
+ 
+tt {
+    background-color: #ecf0f3;
+    color: #222;
+    /* padding: 1px 2px; */
+    font-size: 1.1em;
+    font-family: monospace;
+}
+
+.viewcode-back {
+    font-family: Arial, sans-serif;
+}
+
+div.viewcode-block:target {
+    background-color: #f4debf;
+    border-top: 1px solid #ac9;
+    border-bottom: 1px solid #ac9;
+}
+
+p {
+	margin: 0;
+}
+
+ul li dd {
+	margin-top: 0;
+}
+
+ul li dl {
+	margin-bottom: 0;
+}
+
+li dl dd {
+	margin-bottom: 0;
+}
+
+dd ul {
+	padding-left: 0;
+}
+
+li dd ul {
+	margin-bottom: 0;
+}
+
+table {
+    margin-top: 10px;
+    margin-bottom: 10px;
+    border: 0;
+    border-collapse: collapse;
+}
+
+td[colspan]:not([colspan="1"]) {
+    background: #eeeeee;
+}
+
+thead {
+    background: #eeeeee;
+}
+
diff --git a/supporting-docs/css/site.css b/supporting-docs/css/site.css
new file mode 100644
index 00000000..5baf9b28
--- /dev/null
+++ b/supporting-docs/css/site.css
@@ -0,0 +1,102 @@
+/** Common layout **/
+
+html {
+    height: 100%;
+}
+
+body {
+    height: 100%;
+    font-family: "Open Sans", Helvetica, Arial, sans-serif ! important;
+    font-size: 12pt;
+    margin: 0px;
+}
+
+h1 {
+    font-size: 20pt;
+}
+
+a:link {    color: #666; 
+            text-decoration: underline;
+}
+a:visited { color: #666; }
+a:hover   { color: #000; }
+a:active  { color: #000; }
+
+a.anchor {  
+    display: block; 
+    position: relative; 
+    top: -50px; 
+    visibility: hidden;
+}
+
+#page {
+    min-height: 100%;
+    margin-bottom: -32px; /* to make room for the footer */
+}
+
+#wrapper {
+    margin: auto;
+    max-width: 1280px;
+    padding-top: 40px;
+    padding-bottom: 40px;
+    padding-left: 20px;
+    padding-right: 20px;
+}
+
+#header
+{
+    position: fixed;
+    top: 0px;
+    width: 100%;
+    background-color: #000;
+    min-height: 40px;
+}
+
+.navButton {
+    font-size: 20px;
+    font-weight: bold;        
+    padding: 10px 20px 10px 20px;
+    text-decoration: none ! important; 
+}
+
+#header .navButton {
+    color: #fff ! important;
+}
+
+#headerContent {
+    /*
+    color: #ccc;
+    text-align: right;
+    */
+    text-align: center;
+    max-width: 1280px;
+    margin: auto;
+    min-height: 40px;
+    line-height: 40px;
+}
+
+#headerContent a:link,
+#headerContent a:visited,
+#headerContent a:hover,
+#headerContent a:active {
+    color: #fff;
+}
+
+#footer
+{
+    width: 100%;
+    border-top: #666 1px solid;
+    background-color: #aaa;
+    height: 32px;
+}
+
+#footerContent
+{
+    font-size: 8pt;
+    color: #fff;
+    max-width: 1280px;
+    margin: auto;
+    text-align: center;
+    height: 32px;
+    line-height: 32px;
+}
diff --git a/supporting-docs/feed.xml b/supporting-docs/feed.xml
new file mode 100644
index 00000000..a6628bd8
--- /dev/null
+++ b/supporting-docs/feed.xml
@@ -0,0 +1,30 @@
+---
+layout: null
+---
+<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+  <channel>
+    <title>{{ site.title | xml_escape }}</title>
+    <description>{{ site.description | xml_escape }}</description>
+    <link>{{ site.url }}{{ site.baseurl }}/</link>
+    <atom:link href="{{ "/feed.xml" | prepend: site.baseurl | prepend: site.url }}" rel="self" type="application/rss+xml"/>
+    <pubDate>{{ site.time | date_to_rfc822 }}</pubDate>
+    <lastBuildDate>{{ site.time | date_to_rfc822 }}</lastBuildDate>
+    <generator>Jekyll v{{ jekyll.version }}</generator>
+    {% for post in site.posts limit:10 %}
+      <item>
+        <title>{{ post.title | xml_escape }}</title>
+        <description>{{ post.content | xml_escape }}</description>
+        <pubDate>{{ post.date | date_to_rfc822 }}</pubDate>
+        <link>{{ post.url | prepend: site.baseurl | prepend: site.url }}</link>
+        <guid isPermaLink="true">{{ post.url | prepend: site.baseurl | prepend: site.url }}</guid>
+        {% for tag in post.tags %}
+        <category>{{ tag | xml_escape }}</category>
+        {% endfor %}
+        {% for cat in post.categories %}
+        <category>{{ cat | xml_escape }}</category>
+        {% endfor %}
+      </item>
+    {% endfor %}
+  </channel>
+</rss>
diff --git a/supporting-docs/index.html b/supporting-docs/index.html
new file mode 100644
index 00000000..83d93985
--- /dev/null
+++ b/supporting-docs/index.html
@@ -0,0 +1,23 @@
+---
+layout: default
+---
+
+<div class="home">
+
+  <h1 class="page-heading">Posts</h1>
+
+  <ul class="post-list">
+    {% for post in site.posts %}
+      <li>
+        <span class="post-meta">{{ post.date | date: "%b %-d, %Y" }}</span>
+
+        <h2>
+          <a class="post-link" href="{{ post.url | prepend: site.baseurl }}">{{ post.title }}</a>
+        </h2>
+      </li>
+    {% endfor %}
+  </ul>
+
+  <p class="rss-subscribe">subscribe <a href="{{ "/feed.xml" | prepend: site.baseurl }}">via RSS</a></p>
+
+</div>

From 902c7d3ea69c56b9c1576756123d98721135af12 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 19 Aug 2015 15:00:22 +0100
Subject: [PATCH 007/467] Add draft macaroon caveat specification

---
 drafts/macaroons_caveats.rst | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 drafts/macaroons_caveats.rst

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
new file mode 100644
index 00000000..b6920d6c
--- /dev/null
+++ b/drafts/macaroons_caveats.rst
@@ -0,0 +1,30 @@
+Macaroon Caveats
+================
+
+Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted, by adding caveats to them.
+
+Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
+
+Some caveats are specified in this specification, and must be understood by all servers. The use of non-standard caveats is allowed.
+
+All caveats must take the form:
+
+`key` `operator` `value`
+where `key` is a non-empty string drawn from the character set [A-Za-z0-9_]
+`operator` is a non-empty string which does not contain whitespace
+`value` is a non-empty string
+And these are joined by single space characters.
+
+Specified caveats:
+
++-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
+| Caveat name | Description                                      | Legal Values                                                                               |
++-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
+| gen         | Generation of the macaroon caveat spec.          | 1                                                                                          |
+| user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                   |
+| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                 |
+|                                                                   refresh - only used to authorize a token refresh                                          |
+| time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                |
+|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server. |
+|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.  |
++-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+

From 803b3e9146c7497e6ddf835658c37cc655847d4f Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Wed, 19 Aug 2015 17:00:26 +0100
Subject: [PATCH 008/467] FAQ converted from WP

---
 supporting-docs/_posts/2015-08-19-faq.md | 413 +++++++++++++++++++++++
 1 file changed, 413 insertions(+)
 create mode 100644 supporting-docs/_posts/2015-08-19-faq.md

diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
new file mode 100644
index 00000000..9a5a355c
--- /dev/null
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -0,0 +1,413 @@
+---
+layout: post
+title: FAQ
+date: 2015-08-15 16:58:07
+categories: guides
+---
+<link href="/docs/guides/css/faq.css" type="text/css" rel="stylesheet" />
+
+# FAQ
+{:.no_toc}
+
+Categories
+----------
+{:.no_toc}
+
+[General](#general)
+
+[Quick Start](#quick-start)
+
+[Standard](#standard)
+
+[APIs](#apis)
+
+[Reference Implementations](#reference-implementations)
+
+|
+
+FAQ Content
+-----------
+{:.no_toc}
+
+
+* TOC
+{:toc}
+
+### General
+
+##### What is Matrix?
+
+Matrix is an ambitious new open standard for open, distributed,
+real-time communication over IP. It defines interoperable Instant
+Messaging and VoIP, providing pragmatic HTTP APIs and open source
+reference implementations for creating and running your own real-time
+communication infrastructure.
+
+##### What is Matrix's Mission?
+
+Matrix.org's initial inspiration and goal has been to fix the problem of
+fragmented IP communications. But Matrix's real potential and ultimate
+mission is to be a generic messaging and data synchronisation system for
+the web - allowing people, services and devices to easily communicate
+with each other with full history.
+
+##### What does Matrix provide?
+
+Today Matrix provides a new [open standard](/docs/spec),
+[APIs](/docs/api) to integrate a service to the Matrix ecosystem and
+reference [open source
+implementations](http://github.com/matrix-org/synapse) of the standard.
+
+##### What does this mean for users?
+
+The aim is to provide an analogous ecosystem to email - one where you
+can communicate with pretty much anyone, without caring what app or
+server they are using, using whichever app & server you chose to use,
+and a nice neutral identity system like an e-mail address or phone
+number to discover people to talk to.
+
+##### What kind of company is Matrix.org?
+
+Matrix is an open initiative which acts as a neutral custodian of the
+Matrix standard. It's not actually incorporated anywhere at the moment
+but we are looking at the best legal structure for the future. We are
+committed to keeping the Matrix project open.
+
+##### Who is funding Matrix.org?
+
+We have been given permission by our employers, Amdocs, to work on
+Matrix as an independent non-profit initiative.
+
+##### Who is building Matrix?
+
+We're a team of ~10 people with decades of experience building custom
+VoIP and Messaging apps for mobile network operators. Most of us have
+day jobs at Amdocs or OpenMarket, but we are supported by a mix of
+freelancers and volunteers.
+
+##### Why are you called Matrix?
+
+We are called Matrix because we provide a structure in which all
+communication can be matrixed together.
+
+##### Why have you released this as open source?
+
+We believe that any open standard defining interoperable communication
+needs to be justified, demonstrated and validated with transparent open
+source implementations. For Matrix to achieve its mission of making all
+communications services interoperable we believe it needs to be truly
+open, giving people access to take all the code we produce and to use
+and build on top of it.
+
+##### What do you mean by open?
+
+Matrix is an open standard, meaning that we have freely published the
+details for how to interface with Matrix compliant servers and clients,
+and encourage anyone and everyone to interface with them.  We also
+ensure the standard is not encumbered by any known patent licensing
+requirements.
+
+| 
+
+Matrix is also open source, meaning that we have released the source
+code of the reference servers and clients to the public domain under the
+[Apache Licence v2](http://www.apache.org/licenses/LICENSE-2.0.html), to
+encourage anyone and everyone to run their own servers and clients, and
+enhance them and contribute their enhancements as they see fit.
+
+##### What does federated mean?
+
+Federation allows separate deployments of a communication service to
+communicate with each other - for instance a mail server run by Google
+federates with a mail server run by Microsoft when you send email from
+@gmail.com to @outlook.com.
+
+|
+
+Federation is different to interoperability, as interoperable clients
+may simply be running on the same deployment - whereas in federation the
+deployments themselves are exchanging data in a compatible manner.
+
+|
+
+Matrix provides open federation - meaning that anyone on the internet
+can join into the Matrix ecosystem by deploying their own server.
+
+##### How is this like e-mail?
+
+When email was first set up in the early ‘80s, companies like Compuserve
+and AT&T and Sprint set up isolated email communities which only allowed
+you to exchange mail with users on the same system.  If you got your
+email from one service and your friend from another, then you couldn't
+message each other.  This is basically the situation we're in today with
+VoIP and IM.
+
+##### Why has no-one done this before?
+
+There have been several attempts before including SIP, XMPP and RCS.
+ All of these have had some level of success, but
+technological/usability/economic factors have ended up limiting their
+success in providing true open federation.
+
+##### What is the difference between Matrix and IRC?
+
+We love IRC.  In fact, as of today the core Matrix team still uses it as
+our primary communication tool. Between us we've written IRCds, IRC bots
+and admined dreamforge, UnrealIRCd, epona, ircservices and several
+others.  That said, it has some limitations that Matrix seeks to improve
+on:
+
+-   Text only
+-   No history
+-   No multiple-device support
+-   No presence support
+-   Fragmented identity model
+-   No open federation
+-   No standard APIs, just an archaic TCP line protocol
+-   Non-standardised federation protocol
+-   No built-in end-to-end encryption
+-   Disruptive net-splits
+-   Non-extensible
+
+##### What is the difference between Matrix and XMPP?
+
+The Matrix team used XMPP (Openfire, ejabberd, spectrum, asmack,
+XMPPFramework) for IM before starting to experiment with open HTTP APIs
+as an alternative.   The main issues with XMPP that drove us in this
+direction were:
+
+-   Not particularly web-friendly - you can't easily speak XMPP from a
+    web browser. (N.B. Nowadays you have options like XMPP-FTW and
+    Stanza.io that help loads with letting browsers talk XMPP).
+-   Single logical server per MUC is a single point of control and
+    availability. (MUCs can be distributed over multiple physical
+    servers, but they still sit behind a single logical JID and domain.
+    FMUC improves this with a similar approach to Matrix, but at time of
+    writing there are no open implementations.)
+-   History synchronisation is very much a second class citizen feature
+-   Stanzas aren't framed or reliably delivered without extensions. (See
+    [wiki.xmpp.org](http://wiki.xmpp.org/web/Myths#Myth_Four:_XMPP_is_unreliable_without_a_bunch_of_extensions.)
+    for an XMPP take on this)
+-   Multiple device support is limited. (Apparently Carbons and MAM help
+    with this)
+-   Baseline feature set is so minimal that fragmentation of features
+    between clients and servers is common
+-   No strong identity system (i.e. no standard E2E PKI, unless you
+    count X.509 certs, which [are
+    questionable](http://www.thoughtcrime.org/blog/ssl-and-the-future-of-authenticity/))
+-   Not particularly well designed for mobile use cases: push;
+    bandwidth-efficient transports. (Since the time of writing a [Push
+    XEP has appeared](http://xmpp.org/extensions/xep-0357.html), and
+    [wiki.xmpp.org](http://wiki.xmpp.org/web/Myths#Myth_Three:_It.27s_too_bandwidth-inefficient_for_mobile.)
+    claims that XMPP runs fine over a 9600bps + 30s latency link.)
+
+The whole subject of XMPP vs Matrix seems to bring out the worst in
+people. We think of the standards as being quite different; at its core
+Matrix can be thought of as an eventually consistent global JSON db with
+an HTTP API and pubsub semantics - whilst XMPP can be thought of as a
+message passing protocol. You can use them both to build chat systems;
+you can use them both to build pubsub systems; each comes with different
+tradeoffs. Matrix has a 'kitchen sink' baseline of functionality; XMPP
+has a deliberately minimal baseline set of functionality. If XMPP does
+what you need it to do, then we're genuinely happy for you :) Meanwhile,
+rather than competing, an XMPP Bridge like [Skaverat's xmpptrix
+beta](https://github.com/SkaveRat/xmpptrix) has potential to let both
+environments coexist and make the most of each other's benefits.
+
+##### What is the difference between Matrix and PSYC?
+
+PSYC is a open federated messaging protocol loosely inspired by IRC.  In
+version 1 it was a standalone protocol, and in version 2 it is being
+reutilised as the messaging layer on top of GNUnet.  We honestly don't
+know that much about it, beyond trying to use psycd as an XMPP\<-\>IRC
+bridge in 2010. Matrix differentiates primarily by providing simple HTTP
+APIs rather than the more exotic compact line protocol in PSYC v1 or the
+complicated GNUnet stack in v2.  Meanwhile, Matrix doesn't provide of
+the metadata protection guarantees that GNUnet/PSYC aims for.
+
+| 
+
+See [http://about.psyc.eu/Matrix](http://about.psyc.eu/Matrix) for
+PSYC's views on Matrix.
+
+##### What is the difference between Matrix and Tox?
+
+Tox.im looks to be a very cool clone of Skype - a fully decentralised
+peer-to-peer network.  Matrix is deliberately not peer-to-peer; instead
+each user has a well-defined homeserver which stores his data and that
+he can depend upon.  Matrix provides HTTP APIs; Tox.im provides C APIs.
+ We haven't actually played with Tox at all yet.
+
+##### How does Matrix compare with something like Trillian or Pidgin?
+
+Trillian and Pidgin and similar aggregating IM clients merge all your IM
+activity into a single user experience.  However, your history and
+identity is still fragmented across the networks.  People can't find you
+easily, and your history is fragmented (other than on the device
+where the client runs).   And rather than being able to chose the right
+app for the job when communicating with people, you are pushed towards
+relying on a specific aggregation app.
+
+##### What Matrix compliant apps are there?
+
+None yet, other than our examples.  It's early days :)
+
+##### Why do you think existing apps will ever join this?
+
+We firmly believe it is what is right for the consumer. As people begin
+to use interoperable communications tools service providers will see the
+benefit and compete on quality of service, security and features rather
+than relying on locking people into their walled garden. We believe as
+soon as users see the availability and benefits of interoperable
+services they will demand it.
+
+##### Why aren't you doing this through the IETF? or W3C? or 3GPP?
+
+We do recognise the advantages of working with existing standards
+bodies. We have been focused on writing code and getting it out. As
+Matrix matures it may well be appropriate to work with an official
+standard body.
+
+|
+
+### Quick Start
+
+##### How do I get an account and get started?
+
+The quickest way is to just jump to the demo webclient at
+[http://matrix.org/beta](http://matrix.org/beta) and sign up.  Please note that you can point the
+webclient to access any homeserver - you don't have to use matrix.org,
+although as of day 1, matrix.org is the only communal homeserver
+available.
+
+##### What can I actually do with this?
+
+The demo webclient provides a simple chatroom interface to Matrix -
+letting the user interact with users and rooms anywhere within the
+Matrix federation.  Text and image messages are supported, and basic
+voice-only VoIP calling via WebRTC is supported in one-to-one rooms.
+
+##### How do I connect my homeserver to the public Matrix network?
+
+See
+[http://github.com/matrix-org/synapse](http://github.com/matrix-org/synapse)
+for details
+
+##### How do I Matrix-enable my existing app?
+
+See the [Client-Server API
+HOWTO](http://matrix.org/docs/howtos/client-server.html) for an example
+of how to use Matrix's client-server API to let your app communicate
+with users via Matrix.  We're currently working out the best way to
+integrate your application's existing identity system with Matrix.
+
+##### How can I write a client on Matrix?
+
+See the [Client-Server API
+HOWTO](http://matrix.org/docs/howtos/client-server.html) and the [API
+docs](/docs/api) and [the Spec](/docs/spec) for all the details you need
+to write a client.
+
+##### *How can I help out with this?*
+
+Install synapse and tell us how you get on. Critique the spec.  Write
+clients.  Just come say hi on [\#matrix:matrix.org](/alpha) or the
+[mailing lists](/mailman/listinfo/matrix-users)!
+
+##### Where can I get support?
+
+[\#matrix:matrix.org](/alpha), \#matrix on irc.freenode.net or
+the [mailing lists](/mailman/listinfo/matrix-users) are your best bets.
+
+##### How do I register custom matrix event types?
+
+We're not yet managing a registry of custom matrix event types.  If you
+have any particularly good ones you want to tell the world about, please
+use the [mailing list](/mailman/listinfo/matrix-users) for now.
+
+##### How mature is this?
+
+We started working on Matrix in July 2014, and have opened it to the
+public in September 2014.  It's early days, and under no circumstances
+should you use Matrix or Synapse for anything other than experimentation
+and learning at this point.  Obviously the spec and apps are maturing
+rapidly, but as of the time of writing APIs are not frozen and the apps
+are very much a work in progress.
+
+|
+
+*Sorry, the FAQ is still work in progress, the rest of it will up
+soon!* *In the mean time, don't hesitate to get in touch on
+[\#matrix:matrix.org](/alpha) or the [mailing
+lists](/mailman/listinfo/matrix-users)!*
+
+|
+
+### Standard
+
+##### What is a home server?
+
+##### What is an identity sever?
+
+##### Where do my conversations get stored?
+
+##### What is a 3PID?
+
+##### How do you do VoIP calls on Matrix?
+
+##### Can I log into other homeservers with my username and password?
+
+##### Why Apache Licence?
+
+##### Can I write a Matrix homeserver?
+
+##### How secure is this?
+
+##### Why aren't you using an ORM layer like SqlAlchemy?  
+
+|
+
+### APIs
+
+##### How do I join the global Matrix federation?
+
+##### What ports do I have to open up to join the global Matrix federation?
+
+|
+
+### Reference Implementations
+
+##### What is Matrix built on - and why?
+
+##### How do I run my own home server?
+
+##### Can I run my own identity server?
+
+Yes - the reference implementation is
+[sydent](https://github.com/matrix-org/sydent) and you can run your own
+ID server cluster that tracks 3rd party to Matrix ID mappings. If you
+want your server to participate in the global replicated Matrix ID
+service then please get in touch with us. Meanwhile, we are looking at
+ways of decentralising the 'official' Matrix identity service so that
+identity servers are 100% decentralised and can openly federate with
+each other. **N.B. that you can use Matrix without ever using the
+identity service - it exists only to map 3rd party IDs (e.g. email
+addresses) to matrix IDs to aid user discovery**.
+
+##### What is Synapse?
+
+##### Why is Synapse in Python/Twisted?
+
+##### What are Synapse's platform requirements?
+
+##### What are the Synapse webclient's requirements?
+
+##### Where is the mobile app?
+
+##### What decides the room member order on the webclient?
+
+|
+
+Any other question? Please contact us on
+[\#matrix:matrix.org](/alpha) or the [mailing
+lists](/mailman/listinfo/matrix-users)!

From 65194befe8fa20ac01fb420c51489898d60e7e26 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Wed, 19 Aug 2015 17:01:40 +0100
Subject: [PATCH 009/467] ...and associated css

---
 supporting-docs/css/faq.css | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 supporting-docs/css/faq.css

diff --git a/supporting-docs/css/faq.css b/supporting-docs/css/faq.css
new file mode 100644
index 00000000..d8dec230
--- /dev/null
+++ b/supporting-docs/css/faq.css
@@ -0,0 +1,24 @@
+ul
+{
+    padding-left: 30px;
+}
+
+p
+{
+    padding-left: 30px;
+
+}
+
+h4,h5 
+{
+    border-top: 40px solid transparent;
+    margin-top: -40px;
+    -webkit-background-clip: padding-box;
+    -moz-background-clip: padding;
+    background-clip: padding-box;
+    font-style: italic;
+    font-size: 16px;
+    padding-top: 16px;
+}
+
+

From ade068f7ec70737f6e2a8c82bd7ad4d0993ed235 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Wed, 19 Aug 2015 17:12:56 +0100
Subject: [PATCH 010/467] date and URL change

---
 supporting-docs/_config.yml              | 3 ++-
 supporting-docs/_posts/2015-08-19-faq.md | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/supporting-docs/_config.yml b/supporting-docs/_config.yml
index 3da7faed..bab34b5b 100644
--- a/supporting-docs/_config.yml
+++ b/supporting-docs/_config.yml
@@ -12,4 +12,5 @@ github_username:  matrix-org
 markdown: kramdown
 
 #defaults:
-permalink: /:categories/:title.html
+#permalink: /:categories/:title.html # can use this when/if we use jekyll for all docs
+permalink: :title.html
diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index 9a5a355c..517b2ccf 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -1,7 +1,7 @@
 ---
 layout: post
 title: FAQ
-date: 2015-08-15 16:58:07
+date: 2015-08-19 16:58:07
 categories: guides
 ---
 <link href="/docs/guides/css/faq.css" type="text/css" rel="stylesheet" />

From 4a2c2d015f94f165ca067e9700529829e2ef6766 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Wed, 19 Aug 2015 17:18:58 +0100
Subject: [PATCH 011/467] indent or die

---
 supporting-docs/_config.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/supporting-docs/_config.yml b/supporting-docs/_config.yml
index bab34b5b..1b3ea192 100644
--- a/supporting-docs/_config.yml
+++ b/supporting-docs/_config.yml
@@ -2,7 +2,7 @@
 title: Matrix
 email: webmaster@matrix.org
 description: > # this means to ignore newlines until "baseurl:"
-Matrix.org documentation
+  Matrix.org documentation
 baseurl: "/docs/guides" # the subpath of your site, e.g. /blog/
 url: "http://matrix.org" # the base hostname & protocol for your site
 twitter_username: matrixdotorg

From 16ff77adf9d8b27b19840eea40a85caad42cba25 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Mon, 24 Aug 2015 10:39:59 +0100
Subject: [PATCH 012/467] AS guide and index page added

---
 .gitignore                                    |   1 +
 .../_posts/2015-08-14-getting_involved.md     |   2 +-
 .../_posts/2015-08-21-application_services.md | 143 ++++++++++++++++++
 supporting-docs/css/default.css               |   5 +
 supporting-docs/index.html                    |  21 +--
 5 files changed, 158 insertions(+), 14 deletions(-)
 create mode 100644 supporting-docs/_posts/2015-08-21-application_services.md
 create mode 100644 supporting-docs/css/default.css

diff --git a/.gitignore b/.gitignore
index 188b74a4..63ef5bb3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,4 @@ scripts/gen
 templating/out
 *.pyc
 supporting-docs/_site
+supporting-docs/.sass-cache
diff --git a/supporting-docs/_posts/2015-08-14-getting_involved.md b/supporting-docs/_posts/2015-08-14-getting_involved.md
index 9d655618..9a37a0ea 100644
--- a/supporting-docs/_posts/2015-08-14-getting_involved.md
+++ b/supporting-docs/_posts/2015-08-14-getting_involved.md
@@ -117,4 +117,4 @@ If you are interested in how federation works, please see the [federation API ch
 
 ## Write an Application Service:
 
-One of our core developers, Kegan, wrote an excellent [blogpost about the Application Services API](https://matrix.org/blog/2015/03/02/introduction-to-application-services/), as well as a [blogpost about the Matrix<->IRC Application Service](https://matrix.org/blog/2015/04/22/matrix-irc-application-service/) - these are the best resources to consult if you want to write an Application Service yourself.
+Information about Application services and how they can be used can be found in the [Application services](./application_services.html) document! (This is based on Kegan's excellent blog posts, but now lives here so it can be kept up-to-date!)
diff --git a/supporting-docs/_posts/2015-08-21-application_services.md b/supporting-docs/_posts/2015-08-21-application_services.md
new file mode 100644
index 00000000..bd0453e8
--- /dev/null
+++ b/supporting-docs/_posts/2015-08-21-application_services.md
@@ -0,0 +1,143 @@
+---
+layout: post
+title: Application services
+categories: guides
+---
+
+# Application services
+
+Application services are distinct modules which which sit alongside a home server providing arbitrary extensible functionality decoupled from the home server implementation. Just like the rest of Matrix, they communicate via HTTP using JSON. Application services function in a very similar way to traditional clients, but they are given much more power than a normal client. They can reserve entire namespaces of room aliases and user IDs for their own purposes. They can silently monitor events in rooms, or any events directed at any user ID. This power allows application services to have extremely useful abilities which overall enhance the end user experience.
+
+|
+
+One of the main use cases for application services is protocol bridges. Our Matrix server on Matrix.org links in to various IRC channels and networks. This functionality was initially implemented as a simple bot which resided as a user on the Matrix rooms we wanted to link to freenode channels (#matrix, #matrix-dev, #openwebrtc and #vuc etc). There was nothing special about this bot; it is just treated as a client. However, as we started to rely on it more and more though, we realised that there were things that were impossible for simple client-side bots to do by themselves - for example, the bot could not reserve the virtual user IDs it wanted to create, and could not lazily bridge arbitrary IRC rooms on-the-fly - and this spurred the development of Application Services. 
+
+|
+
+### Some of the features of the IRC application service we have since implemented include:
+
+- Specific channel-to-matrix room bridging : This is what the original IRC bot did. You can specify specific channels and specific room IDs, and messages will be bridged.
+- Dynamic channel-to-matrix room bridging> : This allows Matrix users to join any channel on an IRC network, rather than being forced to use one of the specific channels configured.
+- Two-way PM support : IRC users can PM the virtual "M-" users and private Matrix rooms will be created. Likewise, Matrix users can invite the virtual "@irc_Nick:domain" user IDs to a room and a PM to the IRC nick will be made.
+- IRC nick changing support: Matrix users are no longer forced to use "M-" nicks and can change them by sending "!nick" messages directly to the bridge.
+- Ident support: This allows usernames to be authenticated for virtual IRC clients, which means IRC bans can be targeted at the Matrix user rather than the entire application service.
+
+|
+
+### The use of the Application Services API means:
+
+- The bot can reserve user IDs. This prevents humans from registering for @irc_... user IDs which would then clash with the operation of the bot.
+- The bot can reserve room aliases. This prevents humans from register for #irc_... aliases which would then clash with the operation of the bot.
+- The bot can trivially manage hundreds of users. Events are pushed to the application service directly. If you tried to do this as a client-side bot, you would need one event stream connection per virtual user.
+- The bot can lazily create rooms on demand. This means Matrix users can join non-existent room aliases and have the application service quickly track an IRC channel and create a room with that alias, allowing the join request to succeed.
+
+|
+
+### Implementation details:
+
+- Written in Node.js, designed to be run using <code>forever</code>.
+- Built on the generic <a href="http://github.com/matrix-org/matrix-appservice-node">matrix-appservice-node</a> framework.
+- Supports sending metrics in statsd format.
+- Uses matrix-appservice-node to provide a standardised interface when writing application services, rather than an explicit web framework (though under the hood matrix-appservice-node is using Express).
+
+|
+
+At present, the IRC application service is in beta, and is being run on #matrix and #matrix-dev. If you want to give it a go, <a title="Matrix-IRC Application Service" href="https://github.com/matrix-org/matrix-appservice-irc">check it out on Github</a>!
+
+|
+
+# What Application services can do for you
+Application services have enormous potential for creating new and exciting ways to transform and enhance the core Matrix protocol. For example, you could aggregate information from multiple rooms into a summary room, or create throwaway virtual user accounts to proxy messages for a fixed user ID on-the-fly. As you may expect, all of this power assumes a high degree of trust between application services and home servers. Only home server admins can allow an application service to link up with their home server, and the application service is in no way federated to other home servers. You can think of application services as additional logic on the home server itself, without messing around with the book-keeping that home servers have to do. This makes adding useful functionality very easy.
+
+|
+
+### Example
+
+The application service (AS) API itself uses webhooks to communicate from the home server to the AS:
+
+- Room Alias Query API : The home server hits a URL on your application server to see if a room alias exists.
+- User Query API : The home server hits a URL on your application server to see if a user ID exists.
+- Push API : The home server hits a URL on your application server to notify you of new events for your users and rooms.
+
+A very basic application service may want to log all messages in rooms which have an alias starting with "#logged_" (side note: logging won't work if these rooms are using end-to-end encryption).
+
+Here's an example of a very basic application service using Python (with Flask and Requests) which logs room activity:
+
+        # app_service.py:
+
+        import json, requests  # we will use this later
+        from flask import Flask, jsonify, request
+        app = Flask(__name__)
+
+        @app.route("/transactions/&lt;transaction&gt;", methods=["PUT"])
+        def on_receive_events(transaction):
+            events = request.get_json()["events"]
+            for event in events:
+                print "User: %s Room: %s" % (event["user_id"], event["room_id"])
+                print "Event Type: %s" % event["type"]
+                print "Content: %s" % event["content"]
+            return jsonify({})
+
+        if __name__ == "__main__":
+            app.run()
+
+Set your new application service running on port 5000 with:
+
+        python app_service.py
+
+The home server needs to know that the application service exists before it will send requests to it. This is done via a registration YAML file which is specified in Synapse's main config file e.g. <code>homeserver.yaml</code>. The server admin needs to add the application service registration configuration file as an entry to this file.
+
+        # homeserver.yaml
+        app_service_config_files:
+          - "/path/to/appservice/registration.yaml"
+
+NB: Note the "-" at the start; this indicates a list element. The registration file <code>registration.yaml</code> should look like:
+
+        # registration.yaml
+        
+        # this is the base URL of the application service
+        url: "http://localhost:5000"
+        
+        # This is the token that the AS should use as its access_token when using the Client-Server API
+        # This can be anything you want.
+        as_token: wfghWEGh3wgWHEf3478sHFWE
+        
+        # This is the token that the HS will use when sending requests to the AS.
+        # This can be anything you want.
+        hs_token: ugw8243igya57aaABGFfgeyu
+        
+        # this is the local part of the desired user ID for this AS (in this case @logging:localhost)
+        sender_localpart: logging
+        namespaces:
+          users: []
+          rooms: []
+          aliases:
+            - exclusive: false
+              regex: "#logged_.*"
+
+**You will need to restart the home server after editing the config file before it will take effect.**
+
+|
+
+To test everything is working correctly, go ahead and explicitly create a room with the alias "#logged_test:localhost" and send a message into the room: the HS will relay the message to the AS by PUTing to /transactions/&lt;tid&gt; and you should see your AS print the event on the terminal. This will monitor any room which has an alias prefix of "#logged_", but it won't lazily create room aliases if they don't already exist. This means it will only log messages in the room you created before: #logged_test:localhost. Try joining the room "#logged_test2:localhost" without creating it, and it will fail. Let's fix that and add in lazy room creation:
+
+        @app.route("/rooms/&lt;alias&gt;")
+        def query_alias(alias):
+            alias_localpart = alias.split(":")[0][1:]
+            requests.post(
+                # NB: "TOKEN" is the as_token referred to in registration.yaml
+                "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token=TOKEN",
+                json.dumps({
+                    "room_alias_name": alias_localpart
+                }),
+                headers={"Content-Type":"application/json"}
+            )
+            return jsonify({})
+
+This makes the application service lazily create a room with the requested alias whenever the HS queries the AS for the existence of that alias (when users try to join that room), allowing any room with the alias prefix #logged_ to be sent to the AS. Now try joining the room "#logged_test2:localhost" and it will work as you'd expect.  You can see that if this were a real bridge, the AS would have checked for the existence of #logged_test2 in the remote network, and then lazily-created it in Matrix as required.
+
+|
+
+Application services are powerful components which extend the functionality of home servers, but they are limited. They can only ever function in a "passive" way. For example, you cannot implement an application service which censors swear words in rooms, because there is no way to prevent the event from being sent. Aside from the fact that censoring will not work when using end-to-end encryption, all federated home servers would also need to reject the event in order to stop developing an inconsistent event graph. To "actively" monitor events, another component called a "Policy Server" is required, which is beyond the scope of this post.  Also, Application Services can result in a performance bottleneck, as all events on the homeserver must be ordered and sent to the registered application services.  If you are bridging huge amounts of traffic, you may be better off having your bridge directly talk the Server-Server federation API rather than the simpler Application Service API.
+
+I hope this demonstrates how easy it is to create an application service, along with a few ideas of the kinds of things you can do with them. Obvious uses include build protocol bridges, search engines, invisible bots, etc. For more information on the AS HTTP API, check out the new <a href="http://matrix.org/docs/spec/#application-service-api">Application Service API</a> section in the spec, or the raw drafts and spec in <a href="https://github.com/matrix-org/matrix-doc/" target="_blank">https://github.com/matrix-org/matrix-doc/</a>.
diff --git a/supporting-docs/css/default.css b/supporting-docs/css/default.css
new file mode 100644
index 00000000..7910683d
--- /dev/null
+++ b/supporting-docs/css/default.css
@@ -0,0 +1,5 @@
+p {
+    line-height: 1.5;
+}
+
+
diff --git a/supporting-docs/index.html b/supporting-docs/index.html
index 83d93985..2bc0177f 100644
--- a/supporting-docs/index.html
+++ b/supporting-docs/index.html
@@ -1,23 +1,18 @@
 ---
 layout: default
 ---
+<link href="/docs/guides/css/default.css" type="text/css" rel="stylesheet" />
 
-<div class="home">
 
-  <h1 class="page-heading">Posts</h1>
+<div class="home">
 
-  <ul class="post-list">
-    {% for post in site.posts %}
-      <li>
-        <span class="post-meta">{{ post.date | date: "%b %-d, %Y" }}</span>
+  <h1>Guides</h1>
 
-        <h2>
-          <a class="post-link" href="{{ post.url | prepend: site.baseurl }}">{{ post.title }}</a>
-        </h2>
-      </li>
-    {% endfor %}
-  </ul>
+  <p>Here is a collection of guides that might help you get involved with Matrix.</p>
+  <p>First, there is the <a href="./getting_involved.html" title="Getting Involved">Getting Involved</a> guide, which explains various ways of getting started with Matrix.</p>
+  <p>The <a href="./client-server.html" title="Client-Server API">Client-Server API</a> guide explains in detail how to use the CS API, which is useful if you want to write a client (or modify an existing one) - or if you're just interested in how it works "under the hood".</p>
+  <p>The <a href="./application_services.html" title="Application services">Application services</a> guide introduces and explains Application services, and what they can be used for.
+  <p>Finally there is the <a href="./faq.html" title="FAQ">FAQ</a>, which tries to answer all your questions relating to Matrix!</p>
 
-  <p class="rss-subscribe">subscribe <a href="{{ "/feed.xml" | prepend: site.baseurl }}">via RSS</a></p>
 
 </div>

From 0440983c4ab74f30a38f27d70c51f47ea1a8ff30 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 28 Aug 2015 10:31:45 +0100
Subject: [PATCH 013/467] Spec exchanging refresh tokens for new access tokens

---
 specification/10_client_server_api.rst | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 83c8b7a8..6b8ffbcc 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1087,10 +1087,33 @@ On success, this returns a JSON object with keys:
 user_id
   The fully-qualified Matrix ID that has been registered.
 access_token
-  An access token for the new account.
+  An access token for the account. This token may expire at some point, and if
+  so, it MAY come with a refersh_token, described below.
+refresh_token (optional)
+  A refresh token may be exchanged for a new access_token as described in
+  `Refreshing access tokens`.
 home_server
   The hostname of the Home Server on which the account has been registered.
 
+Refreshing access tokens
+~~~~~~~~~~~~~~~~~~~~~~~~
+Exchanging a refresh token for an access token is done using the request::
+
+  POST $PREFIX/tokenrefresh
+
+The body of the POST request is a JSON object containing:
+
+refresh_token
+  The refresh token.
+
+On success, this invalidates the refresh token, so that it cannot be used again,
+and returns a JSON object with keys:
+
+access_token
+  An access token for the account, as is returned from login.
+refresh_token (optional)
+  A refresh token, as is returned from login.
+
 Changing Password
 ~~~~~~~~~~~~~~~~~
 This section refers to API Version 2. These API calls currently use the prefix

From 9515640d5e0ef51e075a0635ebd8261888e04501 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 28 Aug 2015 10:41:49 +0100
Subject: [PATCH 014/467] Add note about no specific 'token expired' error

---
 specification/10_client_server_api.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 6b8ffbcc..5dd215d4 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1114,6 +1114,11 @@ access_token
 refresh_token (optional)
   A refresh token, as is returned from login.
 
+There is no specific error message to indicate that a request has failed because
+an access token has expired; instead, if a client has reason to believe its
+access token is valid, and it receives an auth error, they should attempt to
+refresh for a new token on failure, and re-try the request with the new token.
+
 Changing Password
 ~~~~~~~~~~~~~~~~~
 This section refers to API Version 2. These API calls currently use the prefix

From 0dc22e721755fa31ca4af105459275eda2762707 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 28 Aug 2015 14:50:31 +0100
Subject: [PATCH 015/467] Document receipts

---
 specification/46_receipts.rst | 64 +++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 specification/46_receipts.rst

diff --git a/specification/46_receipts.rst b/specification/46_receipts.rst
new file mode 100644
index 00000000..43b84947
--- /dev/null
+++ b/specification/46_receipts.rst
@@ -0,0 +1,64 @@
+Receipts
+========
+
+Receipts are used to publish which events in a room the user or their devices
+have interacted with. For example, which events the user has read.
+
+For efficiency this is done as "up to" markers, i.e. marking a particular event
+as, e.g., read indicates the user has read all events *up to* that event.
+
+Client-Server API
+-----------------
+
+Clients will receive receipts in the following format::
+
+    {
+        "type": "m.receipt",
+        "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org"
+        "content": {
+            "$1435641916114394fHBLK:matrix.org": {
+                "read": {
+                    "@erikj:jki.re": { "ts": 1436451550453 },
+                    ...
+                }
+            },
+            ...
+        }
+    }
+
+For efficiency, receipts are batched into one event per room. In the initialSync
+and v2 sync APIs the receipts are listed in a seperate top level receipts key.
+
+Each ``user_id``, ``receipt_type`` pair must be associated with only a single
+``event_id``.
+
+New receipts that come down the event streams are deltas. Deltas update
+existing mappings, clobbering based on ``user_id``, ``receipt_type`` pairs.
+
+
+A client can update the markers for its user by issuing a request::
+
+    POST /_matrix/client/v2_alpha/rooms/<room_id>/receipt/read/<event_id>
+
+Where the contents will of the POST will be included in the content sent to
+other users. The server will automatically set the ``ts`` field.
+
+
+Server-Server API
+-----------------
+
+Receipts are sent across federation as EDUs with type ``m.receipt``. The
+format of the EDUs are::
+
+    {
+        <room_id>: {
+            <receipt_type>: {
+                <user_id>: { <content> }
+            },
+            ....
+        },
+        ...
+    }
+
+These are always sent as deltas to previously sent reciepts.
+

From 83dfc2bf61ef776ad32f7a897c82a67f95e0caa3 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 28 Aug 2015 15:01:02 +0100
Subject: [PATCH 016/467] Fix some mistakes/typos

---
 specification/46_receipts.rst | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

diff --git a/specification/46_receipts.rst b/specification/46_receipts.rst
index 43b84947..6428d6b5 100644
--- a/specification/46_receipts.rst
+++ b/specification/46_receipts.rst
@@ -5,16 +5,32 @@ Receipts are used to publish which events in a room the user or their devices
 have interacted with. For example, which events the user has read.
 
 For efficiency this is done as "up to" markers, i.e. marking a particular event
-as, e.g., read indicates the user has read all events *up to* that event.
+as, say, ``read`` indicates the user has read all events *up to* that event.
 
 Client-Server API
 -----------------
 
 Clients will receive receipts in the following format::
 
+     {
+        "type": "m.receipt",
+        "room_id": <room_id>,
+        "content": {
+            <event_id>: {
+                <receipt_type>: {
+                    <user_id>: { "ts": <ts>, ... },
+                    ...
+                }
+            },
+            ...
+        }
+    }
+
+For example::
+
     {
         "type": "m.receipt",
-        "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org"
+        "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org",
         "content": {
             "$1435641916114394fHBLK:matrix.org": {
                 "read": {
@@ -27,7 +43,8 @@ Clients will receive receipts in the following format::
     }
 
 For efficiency, receipts are batched into one event per room. In the initialSync
-and v2 sync APIs the receipts are listed in a seperate top level receipts key.
+and v2 sync APIs the receipts are listed in a seperate top level ``receipts``
+key.
 
 Each ``user_id``, ``receipt_type`` pair must be associated with only a single
 ``event_id``.
@@ -40,7 +57,7 @@ A client can update the markers for its user by issuing a request::
 
     POST /_matrix/client/v2_alpha/rooms/<room_id>/receipt/read/<event_id>
 
-Where the contents will of the POST will be included in the content sent to
+Where the contents of the ``POST`` will be included in the content sent to
 other users. The server will automatically set the ``ts`` field.
 
 
@@ -55,7 +72,7 @@ format of the EDUs are::
             <receipt_type>: {
                 <user_id>: { <content> }
             },
-            ....
+            ...
         },
         ...
     }

From 4b24b887c7004739ef0e7cc7e29c00d46fadc283 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 1 Sep 2015 11:53:30 +0100
Subject: [PATCH 017/467] Document m.room.history_visibility

---
 specification/47_history_visibility.rst | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 specification/47_history_visibility.rst

diff --git a/specification/47_history_visibility.rst b/specification/47_history_visibility.rst
new file mode 100644
index 00000000..d1c1f0d3
--- /dev/null
+++ b/specification/47_history_visibility.rst
@@ -0,0 +1,25 @@
+Room History Visibility
+=======================
+
+The visibility of events to users before they joined can be controlled by the
+``m.room.history_visibility`` state event.
+
+More specifically, the string value of the content key ``history_visibility``
+controls the visibility of events. The valid values are:
+
+- ``shared``
+- ``invited``
+- ``joined``
+
+By default, ``history_visibility`` is set to ``shared``.
+
+The rules governing whether a user is allowed to see an event depend solely on
+the state at that event:
+
+1. If the user was joined, allow.
+2. If the user was invited and the ``history_visibility`` was set to
+   ``invited`` or ``shared``, allow.
+3. If the user was neither invited nor joined but the ``history_visibility``
+   was set to ``shared``, allow.
+4. Otherwise, deny.
+

From 80bb51d9bbf752550ff5f930fbf0787ebddad4d9 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 11:21:14 +0100
Subject: [PATCH 018/467] Make it clearer

---
 specification/47_history_visibility.rst | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/specification/47_history_visibility.rst b/specification/47_history_visibility.rst
index d1c1f0d3..b1630f1e 100644
--- a/specification/47_history_visibility.rst
+++ b/specification/47_history_visibility.rst
@@ -1,20 +1,19 @@
 Room History Visibility
 =======================
 
-The visibility of events to users before they joined can be controlled by the
-``m.room.history_visibility`` state event.
-
-More specifically, the string value of the content key ``history_visibility``
-controls the visibility of events. The valid values are:
+Whether a member of a room can see the events that happened in a room from
+before they joined the room is controlled by the ``history_visibility`` key
+of the ``m.room.history_visibility`` state event. The valid values for
+``history_visibility`` are:
 
 - ``shared``
 - ``invited``
 - ``joined``
 
-By default, ``history_visibility`` is set to ``shared``.
+By default if no ``history_visibility`` is set it is assumed to be ``shared``.
 
 The rules governing whether a user is allowed to see an event depend solely on
-the state at that event:
+the state of the room at that event:
 
 1. If the user was joined, allow.
 2. If the user was invited and the ``history_visibility`` was set to

From 03b8c7a12b0ce16bb6b720ba556b6ca78c3fc5a5 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 11:43:11 +0100
Subject: [PATCH 019/467] Document the canonical alias event

---
 .../examples/v1/m.room.canonical_alias        | 12 ++++++++
 .../schema/v1/m.room.canonical_alias          | 30 +++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 event-schemas/examples/v1/m.room.canonical_alias
 create mode 100644 event-schemas/schema/v1/m.room.canonical_alias

diff --git a/event-schemas/examples/v1/m.room.canonical_alias b/event-schemas/examples/v1/m.room.canonical_alias
new file mode 100644
index 00000000..9fd170d9
--- /dev/null
+++ b/event-schemas/examples/v1/m.room.canonical_alias
@@ -0,0 +1,12 @@
+{
+  "age": 242352,
+  "content": {
+    "alias": "#somewhere:localhost"
+  },
+  "state_key": "",
+  "origin_server_ts": 1431961217939,
+  "event_id": "$WLGTSEFSEF:localhost",
+  "type": "m.room.aliases",
+  "room_id": "!Cuyf34gef24t:localhost",
+  "user_id": "@example:localhost"
+}
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
new file mode 100644
index 00000000..bca91aa7
--- /dev/null
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -0,0 +1,30 @@
+{
+    "$schema": "http://json-schema.org/draft-04/schema#",
+    "type": "object",
+    "title": "Informs the room as to which alias is the canonical one.",
+    "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
+    "allOf": [{
+        "$ref": "core#/definitions/state_event"
+    }],
+    "properties": {
+        "content": {
+            "type": "object",
+            "properties": {
+                "alias": {
+                    "type": "String",
+                    "description": "The canonical alias.",
+                }
+            },
+            "required": ["alias"]
+        },
+        "state_key": {
+            "type": "string",
+            "description": "A zero-length string.",
+            "pattern": "^$"
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.room.canonical_alias"]
+        }
+    }
+}

From 61f1424124a2643fcbb0d55f6e8df93a06b162df Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 11:51:13 +0100
Subject: [PATCH 020/467] No trailing commas

---
 event-schemas/schema/v1/m.room.canonical_alias | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index bca91aa7..2f8c0c36 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -12,7 +12,7 @@
             "properties": {
                 "alias": {
                     "type": "String",
-                    "description": "The canonical alias.",
+                    "description": "The canonical alias."
                 }
             },
             "required": ["alias"]

From 3aabb77cb4679fee642bd7c0a915a60f6eac1309 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 11:53:07 +0100
Subject: [PATCH 021/467] Capitilasation matters

---
 event-schemas/schema/v1/m.room.canonical_alias | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 2f8c0c36..9f044e58 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -11,7 +11,7 @@
             "type": "object",
             "properties": {
                 "alias": {
-                    "type": "String",
+                    "type": "string",
                     "description": "The canonical alias."
                 }
             },

From 40f037da55e01b03af466ab9072973e9603954a8 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 11:54:53 +0100
Subject: [PATCH 022/467] Get the right type

---
 event-schemas/examples/v1/m.room.canonical_alias | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/examples/v1/m.room.canonical_alias b/event-schemas/examples/v1/m.room.canonical_alias
index 9fd170d9..0203a851 100644
--- a/event-schemas/examples/v1/m.room.canonical_alias
+++ b/event-schemas/examples/v1/m.room.canonical_alias
@@ -6,7 +6,7 @@
   "state_key": "",
   "origin_server_ts": 1431961217939,
   "event_id": "$WLGTSEFSEF:localhost",
-  "type": "m.room.aliases",
+  "type": "m.room.canonical_alias",
   "room_id": "!Cuyf34gef24t:localhost",
   "user_id": "@example:localhost"
 }

From c6d143447b5112265741613e9af3571a6b840ca1 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 13:27:01 +0100
Subject: [PATCH 023/467] Add schemas

---
 .../examples/v1/m.room.history_visibility     | 12 +++++++
 .../schema/v1/m.room.history_visibility       | 31 +++++++++++++++++++
 2 files changed, 43 insertions(+)
 create mode 100644 event-schemas/examples/v1/m.room.history_visibility
 create mode 100644 event-schemas/schema/v1/m.room.history_visibility

diff --git a/event-schemas/examples/v1/m.room.history_visibility b/event-schemas/examples/v1/m.room.history_visibility
new file mode 100644
index 00000000..ad1ea9ae
--- /dev/null
+++ b/event-schemas/examples/v1/m.room.history_visibility
@@ -0,0 +1,12 @@
+{
+  "age": 242352,
+  "content": {
+    "history_visibility": "shared"
+  },
+  "state_key": "localhost",
+  "origin_server_ts": 1431961217939,
+  "event_id": "$WLGTSEFSEF:localhost",
+  "type": "m.room.history_visibility",
+  "room_id": "!Cuyf34gef24t:localhost",
+  "user_id": "@example:localhost"
+}
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
new file mode 100644
index 00000000..58cd7e7a
--- /dev/null
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -0,0 +1,31 @@
+
+    "$schema": "http://json-schema.org/draft-04/schema#",
+    "type": "object",
+    "title": "Controls visibility of history.",
+    "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
+    "allOf": [{
+        "$ref": "core#/definitions/state_event"
+    }],
+    "properties": {
+        "content": {
+            "type": "object",
+            "properties": {
+                "history_visibility": {
+                    "type": "string",
+                    "description": "Who can see the room history.",
+                    "enum": ["invited","joined","shared"]
+                }
+            },
+            "required": ["history_visibility"]
+        },
+        "state_key": {
+            "type": "string",
+            "description": "A zero-length string.",
+            "pattern": "^$"
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.room.history_visibility"]
+        }
+    }
+}

From 1e3d9a6a503f576ecbd01d5dda29eefd4030bc2a Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 13:30:11 +0100
Subject: [PATCH 024/467] Typo

---
 event-schemas/schema/v1/m.room.history_visibility | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 58cd7e7a..07bd39ff 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -1,4 +1,4 @@
-
+{
     "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Controls visibility of history.",

From b4c2cba4062003c2e915e8249571d5ac1f4e69b2 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 4 Sep 2015 13:31:51 +0100
Subject: [PATCH 025/467] State key should be empty

---
 event-schemas/examples/v1/m.room.history_visibility | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/examples/v1/m.room.history_visibility b/event-schemas/examples/v1/m.room.history_visibility
index ad1ea9ae..fcc3f881 100644
--- a/event-schemas/examples/v1/m.room.history_visibility
+++ b/event-schemas/examples/v1/m.room.history_visibility
@@ -3,7 +3,7 @@
   "content": {
     "history_visibility": "shared"
   },
-  "state_key": "localhost",
+  "state_key": "",
   "origin_server_ts": 1431961217939,
   "event_id": "$WLGTSEFSEF:localhost",
   "type": "m.room.history_visibility",

From 362a081d68bad05af8c7167d6bcb1549ea96ae1d Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Fri, 4 Sep 2015 16:32:20 +0100
Subject: [PATCH 026/467] css wrangling

---
 supporting-docs/_includes/head.html    |   4 +-
 supporting-docs/_layouts/default.html  |   5 +-
 supporting-docs/_posts/index.html      |   1 -
 supporting-docs/css/default.css        |   5 --
 supporting-docs/css/faq.css            |  28 +++++--
 supporting-docs/css/site.css           | 102 -------------------------
 supporting-docs/css/site_overrides.css |  36 +++++++++
 supporting-docs/index.html             |   3 -
 8 files changed, 64 insertions(+), 120 deletions(-)
 delete mode 100644 supporting-docs/css/default.css
 delete mode 100644 supporting-docs/css/site.css
 create mode 100644 supporting-docs/css/site_overrides.css

diff --git a/supporting-docs/_includes/head.html b/supporting-docs/_includes/head.html
index bb99e6dd..ddb4d988 100644
--- a/supporting-docs/_includes/head.html
+++ b/supporting-docs/_includes/head.html
@@ -5,8 +5,10 @@
   <title>{% if page.title %}{{ page.title }}{% else %}{{ site.title }}{% endif %}</title>
   <meta name="description" content="{% if page.excerpt %}{{ page.excerpt | strip_html | strip_newlines | truncate: 160 }}{% else %}{{ site.description }}{% endif %}">
 
+  <link rel="stylesheet" href="http://matrix.org/site.css">
   <link rel="stylesheet" href="{{ "/css/main.css" | prepend: site.baseurl }}">
-  <link rel="stylesheet" href="{{ "/css/site.css" | prepend: site.baseurl }}">
+
+  <link rel="stylesheet" href="{{ "/css/site_overrides.css" | prepend: site.baseurl }}">
   <link rel="stylesheet" href="{{ "/css/basic.css" | prepend: site.baseurl }}">
   <link rel="stylesheet" href="{{ "/css/nature.css" | prepend: site.baseurl }}">
 
diff --git a/supporting-docs/_layouts/default.html b/supporting-docs/_layouts/default.html
index ae99cc0d..ff7f69f3 100644
--- a/supporting-docs/_layouts/default.html
+++ b/supporting-docs/_layouts/default.html
@@ -7,8 +7,9 @@
   <body>
   {% include nav.html %}
     <br />
-    <div class="page-content">
-      <div class="wrapper">
+    <div class="page-content" id="page">
+      <div class="wrapper" id="wrapper">
+        <div class="document_foo" id="foo">
       {{ content }}
       </div>
     </div>
diff --git a/supporting-docs/_posts/index.html b/supporting-docs/_posts/index.html
index c442d6e1..15c7d5c2 100644
--- a/supporting-docs/_posts/index.html
+++ b/supporting-docs/_posts/index.html
@@ -1,7 +1,6 @@
 ---
 layout: default
 ---
-  {% include site.css %}
   {% include basic.css %}
   {% include nature.css %}
 
diff --git a/supporting-docs/css/default.css b/supporting-docs/css/default.css
deleted file mode 100644
index 7910683d..00000000
--- a/supporting-docs/css/default.css
+++ /dev/null
@@ -1,5 +0,0 @@
-p {
-    line-height: 1.5;
-}
-
-
diff --git a/supporting-docs/css/faq.css b/supporting-docs/css/faq.css
index d8dec230..9730efa3 100644
--- a/supporting-docs/css/faq.css
+++ b/supporting-docs/css/faq.css
@@ -1,16 +1,30 @@
-ul
-{
+.toc {
+    background: #f9f9f9;
+    border: 1px solid #aaaaaa;
+    display: table;
+    padding-top: 16px;
+    padding-right: 16px;
+    padding-bottom: 16px;
+    list-style: none;
+    color: #2EA3F2;
+}
+
+ul {
     padding-left: 30px;
 }
 
-p
-{
+p {
     padding-left: 30px;
+    color: #666;
 
 }
 
-h4,h5 
-{
+h3 {
+    padding-top: 48px;
+    color: #333;
+}
+
+h4, h5 {
     border-top: 40px solid transparent;
     margin-top: -40px;
     -webkit-background-clip: padding-box;
@@ -19,6 +33,8 @@ h4,h5
     font-style: italic;
     font-size: 16px;
     padding-top: 16px;
+    color: #333;
+
 }
 
 
diff --git a/supporting-docs/css/site.css b/supporting-docs/css/site.css
deleted file mode 100644
index 5baf9b28..00000000
--- a/supporting-docs/css/site.css
+++ /dev/null
@@ -1,102 +0,0 @@
-/** Common layout **/
-
-html {
-    height: 100%;
-}
-
-body {
-    height: 100%;
-    font-family: "Open Sans", Helvetica, Arial, sans-serif ! important;
-    font-size: 12pt;
-    margin: 0px;
-}
-
-h1 {
-    font-size: 20pt;
-}
-
-a:link {    color: #666; 
-            text-decoration: underline;
-}
-a:visited { color: #666; }
-a:hover   { color: #000; }
-a:active  { color: #000; }
-
-a.anchor {  
-    display: block; 
-    position: relative; 
-    top: -50px; 
-    visibility: hidden;
-}
-
-#page {
-    min-height: 100%;
-    margin-bottom: -32px; /* to make room for the footer */
-}
-
-#wrapper {
-    margin: auto;
-    max-width: 1280px;
-    padding-top: 40px;
-    padding-bottom: 40px;
-    padding-left: 20px;
-    padding-right: 20px;
-}
-
-#header
-{
-    position: fixed;
-    top: 0px;
-    width: 100%;
-    background-color: #000;
-    min-height: 40px;
-}
-
-.navButton {
-    font-size: 20px;
-    font-weight: bold;        
-    padding: 10px 20px 10px 20px;
-    text-decoration: none ! important; 
-}
-
-#header .navButton {
-    color: #fff ! important;
-}
-
-#headerContent {
-    /*
-    color: #ccc;
-    text-align: right;
-    */
-    text-align: center;
-    max-width: 1280px;
-    margin: auto;
-    min-height: 40px;
-    line-height: 40px;
-}
-
-#headerContent a:link,
-#headerContent a:visited,
-#headerContent a:hover,
-#headerContent a:active {
-    color: #fff;
-}
-
-#footer
-{
-    width: 100%;
-    border-top: #666 1px solid;
-    background-color: #aaa;
-    height: 32px;
-}
-
-#footerContent
-{
-    font-size: 8pt;
-    color: #fff;
-    max-width: 1280px;
-    margin: auto;
-    text-align: center;
-    height: 32px;
-    line-height: 32px;
-}
diff --git a/supporting-docs/css/site_overrides.css b/supporting-docs/css/site_overrides.css
new file mode 100644
index 00000000..700a09e1
--- /dev/null
+++ b/supporting-docs/css/site_overrides.css
@@ -0,0 +1,36 @@
+p {
+    line-height: 1.5;
+    color: #666;
+}
+
+a.anchor {  
+    display: block; 
+    position: relative; 
+    top: -50px; 
+    visibility: hidden;
+}
+
+a:link {
+    color: #2EA3F2;
+    text-decoration: none;
+}
+
+a:visited {
+    color: #2EA3F2;
+    text-decoration: none;
+}
+
+a:active {
+    color: #2EA3F2;
+    text-decoration: none;
+}
+
+a:hover {
+    color: #2EA3F2;
+    text-decoration: underline;
+}
+
+h1,h2,h3,h4,h5 {
+    color: #333;
+}
+
diff --git a/supporting-docs/index.html b/supporting-docs/index.html
index 2bc0177f..2a77562d 100644
--- a/supporting-docs/index.html
+++ b/supporting-docs/index.html
@@ -1,9 +1,6 @@
 ---
 layout: default
 ---
-<link href="/docs/guides/css/default.css" type="text/css" rel="stylesheet" />
-
-
 <div class="home">
 
   <h1>Guides</h1>

From 1284f3f0924d9da391ab6fa4efb4e89cd1c561b0 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Fri, 4 Sep 2015 17:53:39 +0100
Subject: [PATCH 027/467] glue footer to the bottom

---
 supporting-docs/_includes/footer.html  |  2 +-
 supporting-docs/_layouts/default.html  |  8 ++++----
 supporting-docs/css/site_overrides.css | 21 +++++++++++++++++++++
 supporting-docs/index.html             |  1 -
 4 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/supporting-docs/_includes/footer.html b/supporting-docs/_includes/footer.html
index 14e9c2a5..fc43ec97 100644
--- a/supporting-docs/_includes/footer.html
+++ b/supporting-docs/_includes/footer.html
@@ -1 +1 @@
-  <div id="footer"><div id="footerContent">&copy 2014-2015 Matrix.org</div>
+  <div id="footer"><div id="footerContent">&copy 2014-2015 Matrix.org</div></div>
diff --git a/supporting-docs/_layouts/default.html b/supporting-docs/_layouts/default.html
index ff7f69f3..00ea25ce 100644
--- a/supporting-docs/_layouts/default.html
+++ b/supporting-docs/_layouts/default.html
@@ -5,16 +5,16 @@
 
   </head>
   <body>
-  {% include nav.html %}
-    <br />
     <div class="page-content" id="page">
+      {% include nav.html %}
       <div class="wrapper" id="wrapper">
-        <div class="document_foo" id="foo">
+        <div class="document_foo" id="document">
       {{ content }}
+        </div>
       </div>
+      <div class="push"></div>
     </div>
     {% include footer.html %}
-
   </body>
 
 </html>
diff --git a/supporting-docs/css/site_overrides.css b/supporting-docs/css/site_overrides.css
index 700a09e1..0f9a2c80 100644
--- a/supporting-docs/css/site_overrides.css
+++ b/supporting-docs/css/site_overrides.css
@@ -34,3 +34,24 @@ h1,h2,h3,h4,h5 {
     color: #333;
 }
 
+.page-content {
+    min-height: 100%;
+    height: auto !important; /* This line and the next line are not necessary unless you need IE6 support */
+    height: 100%;
+    margin: 0 auto -33px; /* the bottom margin is the negative value of the footer's height */
+    padding-bottom: 0;
+    padding-top: 0;
+}
+
+.push {
+    height: 33px; 
+}
+
+#page {
+    margin-bottom: -33px;
+}
+
+#document {
+    margin-top: 10px; /* We want a little whitespace before the page content starts */
+}
+
diff --git a/supporting-docs/index.html b/supporting-docs/index.html
index 2a77562d..5a874f06 100644
--- a/supporting-docs/index.html
+++ b/supporting-docs/index.html
@@ -11,5 +11,4 @@ layout: default
   <p>The <a href="./application_services.html" title="Application services">Application services</a> guide introduces and explains Application services, and what they can be used for.
   <p>Finally there is the <a href="./faq.html" title="FAQ">FAQ</a>, which tries to answer all your questions relating to Matrix!</p>
 
-
 </div>

From 0414ff4cc346eb171d1a0e793af36c3fb4dac33b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 7 Sep 2015 16:27:18 +0100
Subject: [PATCH 028/467] Output error if command fails

---
 scripts/gendoc.py | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index f8ec6316..e3e8d3d4 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -32,17 +32,23 @@ def rst2html(i, o):
             )
 
 def run_through_template(input):
-    null = open(os.devnull, 'w')
-    subprocess.check_output(
-        [
-            'python', 'build.py', 
-            "-i", "matrix_templates", 
-            "-o", "../scripts/tmp", 
-            "../scripts/"+input
-        ],
-        stderr=null,
-        cwd="../templating",
-    )
+    tmpfile = './tmp/output'
+    try:
+        with open(tmpfile, 'w') as out:
+            subprocess.check_output(
+                [
+                    'python', 'build.py',
+                    "-i", "matrix_templates",
+                    "-o", "../scripts/tmp",
+                    "../scripts/"+input
+                ],
+                stderr=out,
+                cwd="../templating",
+            )
+    except subprocess.CalledProcessError as e:
+        with open(tmpfile, 'r') as f:
+            print f.read()
+        raise
 
 def prepare_env():
     try:

From c74c1ab098876e26efe561a5ecd6a1cc88d76284 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 7 Sep 2015 17:09:41 +0100
Subject: [PATCH 029/467] SPEC-216: Add rooms that have been left to initial
 sync

---
 api/client-server/v1/sync.yaml | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index a08146b4..777e27db 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -285,8 +285,11 @@ paths:
                         chunk:
                           type: array
                           description: |-
-                            A list of the most recent messages for this room. This
-                            array will consist of at most ``limit`` elements.
+                            If the user is a member of the room this will be a
+                            list of the most recent messages for this room. If
+                            the user has left the room this will be the
+                            messages that preceeded them leaving. This array
+                            will consist of at most ``limit`` elements.
                           items:
                             type: object
                             title: RoomEvent
@@ -296,8 +299,10 @@ paths:
                     state:
                       type: array
                       description: |-
-                        A list of state events representing the current state
-                        of the room.
+                        If the user is a member of the room this will be the
+                        current state of the room as a list of events. If the
+                        user has left the room this will be the state of the
+                        room when they left it.
                       items:
                         title: StateEvent
                         type: object
@@ -347,4 +352,4 @@ paths:
             allOf:
               - "$ref": "definitions/event.yaml"
         404:
-          description: The event was not found or you do not have permission to read this event.
\ No newline at end of file
+          description: The event was not found or you do not have permission to read this event.

From 94b13c01218864fac8f9a8e9ce3544194c589719 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 7 Sep 2015 17:14:36 +0100
Subject: [PATCH 030/467] Show all responses, not just the successful one

This still filters out responses lacking either a description or an
example
---
 .../matrix_templates/templates/http-api.tmpl    | 17 +++++++++++++++--
 templating/matrix_templates/units.py            | 16 +++++++++++-----
 2 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 5e80f168..1384daab 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -46,6 +46,19 @@ Example request::
 
   {{endpoint.example.req | indent_block(2)}}
 
-Example response::
+{% if endpoint.example.responses|length > 0 -%}
+Response{{"s" if endpoint.example.responses|length > 1 else "" }}:
 
-  {{endpoint.example.res | indent_block(2)}}
\ No newline at end of file
+{% endif -%}
+
+{% for res in endpoint.example.responses -%}
+
+**Status code {{res["code"]}}:**
+
+{{res["description"]}}
+
+Example::
+
+  {{res["example"] | indent_block(2)}}
+
+{% endfor %}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index d607160f..42ce3f22 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -177,11 +177,17 @@ class MatrixUnits(Units):
                         endpoint["req_param_by_loc"][p["loc"]] = []
                     endpoint["req_param_by_loc"][p["loc"]].append(p)
 
-                # add example response if it has one
-                res = single_api["responses"][200]  # get the 200 OK response
-                endpoint["example"]["res"] = res.get("examples", {}).get(
-                    "application/json", ""
-                )
+                endpoint["example"]["responses"] = []  # Ordered list of maps
+                for code, res in single_api["responses"].items():
+                    description = res.get("description", "")
+                    example = res.get("examples", {}).get("application/json", "")
+                    if description and example:
+                        endpoint["example"]["responses"].append({
+                            "code": code,
+                            "description": description,
+                            "example": example,
+                        })
+
                 # form example request if it has one. It "has one" if all params
                 # have either "x-example" or a "schema" with an "example".
                 params_missing_examples = [

From 0a9f61029a82ad67f9b02e054fd5238e976905d3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 11:16:48 +0100
Subject: [PATCH 031/467] Allow for missing responses key

---
 templating/matrix_templates/units.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 42ce3f22..5852b438 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -178,7 +178,7 @@ class MatrixUnits(Units):
                     endpoint["req_param_by_loc"][p["loc"]].append(p)
 
                 endpoint["example"]["responses"] = []  # Ordered list of maps
-                for code, res in single_api["responses"].items():
+                for code, res in single_api.get("responses", {}).items():
                     description = res.get("description", "")
                     example = res.get("examples", {}).get("application/json", "")
                     if description and example:

From 5031c26f7b4b39ece5bb04df4be38ff0b3000292 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 11:21:19 +0100
Subject: [PATCH 032/467] Make res optional, and rename to good_response

---
 templating/matrix_templates/units.py | 42 ++++++++++++++++------------
 1 file changed, 24 insertions(+), 18 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 5852b438..c4ed8358 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -116,7 +116,8 @@ class MatrixUnits(Units):
                     "res_tables": [],
                     "example": {
                         "req": "",
-                        "res": ""
+                        "responses": [],
+                        "good_response": ""
                     }
                 }
                 self.log(".o.O.o. Endpoint: %s %s" % (method, path))
@@ -177,8 +178,10 @@ class MatrixUnits(Units):
                         endpoint["req_param_by_loc"][p["loc"]] = []
                     endpoint["req_param_by_loc"][p["loc"]].append(p)
 
-                endpoint["example"]["responses"] = []  # Ordered list of maps
+                good_response = None
                 for code, res in single_api.get("responses", {}).items():
+                    if not good_response and code == 200:
+                        good_response = res
                     description = res.get("description", "")
                     example = res.get("examples", {}).get("application/json", "")
                     if description and example:
@@ -222,22 +225,25 @@ class MatrixUnits(Units):
                     )
 
                 # add response params if this API has any.
-                res_type = Units.prop(res, "schema/type")
-                if res_type and res_type not in ["object", "array"]:
-                    # response is a raw string or something like that
-                    endpoint["res_tables"].append({
-                        "title": None,
-                        "rows": [{
-                            "key": res["schema"].get("name", ""),
-                            "type": res_type,
-                            "desc": res.get("description", "")
-                        }]
-                    })
-                elif res_type and Units.prop(res, "schema/properties"):  # object
-                    res_tables = get_json_schema_object_fields(res["schema"])
-                    for table in res_tables:
-                        if "no-table" not in table:
-                            endpoint["res_tables"].append(table)
+                if good_response:
+                    res_type = Units.prop(good_response, "schema/type")
+                    if res_type and res_type not in ["object", "array"]:
+                        # response is a raw string or something like that
+                        endpoint["res_tables"].append({
+                            "title": None,
+                            "rows": [{
+                                "key": good_response["schema"].get("name", ""),
+                                "type": res_type,
+                                "desc": res.get("description", "")
+                            }]
+                        })
+                    elif res_type and Units.prop(good_response, "schema/properties"):
+                        # response is an object:
+                        schema = good_response["schema"]
+                        res_tables = get_json_schema_object_fields(schema)
+                        for table in res_tables:
+                            if "no-table" not in table:
+                                endpoint["res_tables"].append(table)
 
                 endpoints.append(endpoint)
 

From 26ebe3e68b413c3151b12885da713f9be556e80b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 16:27:20 +0100
Subject: [PATCH 033/467] Add ability to refer to aliases of endpoints

---
 .../matrix_templates/templates/http-api.tmpl      |  6 ++++++
 templating/matrix_templates/units.py              | 15 ++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 1384daab..a0b25924 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -1,5 +1,10 @@
 ``{{endpoint.method}} {{endpoint.path}}``
 {{(5 + (endpoint.path | length) + (endpoint.method | length)) * title_kind}}
+{% if "alias_for_path" in endpoint -%}
+``{{endpoint.path}}`` is an alias for `{{endpoint.alias_for_path}}`_.
+
+.. _`{{endpoint.alias_for_path}}`: #{{endpoint.alias_link}}
+{% else -%}
 
 {{endpoint.desc | wrap(80)}}
 
@@ -62,3 +67,4 @@ Example::
   {{res["example"] | indent_block(2)}}
 
 {% endfor %}
+{% endif -%}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index c4ed8358..89b34bb1 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -105,11 +105,24 @@ class MatrixUnits(Units):
         for path in api["paths"]:
             for method in api["paths"][path]:
                 single_api = api["paths"][path][method]
+
+                full_path = api.get("basePath", "") + path
+                alias_for_path = single_api.get("x-alias-for-path", "")
+                alias_link = single_api.get("x-alias-link", "")
+                if alias_for_path:
+                    endpoints.append({
+                        "method": method.upper(),
+                        "path": full_path,
+                        "alias_for_path": alias_for_path,
+                        "alias_link": alias_link
+                    })
+                    continue
+
                 endpoint = {
                     "title": single_api.get("summary", ""),
                     "desc": single_api.get("description", single_api.get("summary", "")),
                     "method": method.upper(),
-                    "path": api.get("basePath", "") + path,
+                    "path": full_path,
                     "requires_auth": "security" in single_api,
                     "rate_limited": 429 in single_api.get("responses", {}),
                     "req_params": [],

From d53943c8c2aa266d894d7fa0b5a2a3a8545f5c25 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 16:35:08 +0100
Subject: [PATCH 034/467] Spec /room/{roomId}/join

This is just replacing the existing spec with a swagger version.

Subsequent pull requests will add 3pid join to this, as well as specing
the invite, leave, ban, and kick endpoints.
---
 api/client-server/v1/membership.yaml    | 68 +++++++++++++++++++++++++
 specification/10_client_server_api.rst  | 38 +-------------
 templating/matrix_templates/sections.py |  6 +++
 3 files changed, 75 insertions(+), 37 deletions(-)
 create mode 100644 api/client-server/v1/membership.yaml

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
new file mode 100644
index 00000000..24f3835d
--- /dev/null
+++ b/api/client-server/v1/membership.yaml
@@ -0,0 +1,68 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Room Membership API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/room/{roomId}/join":
+    post:
+      summary: Start the requesting user participating in a particular room.
+      description: |-
+        This API starts a user participating in a particular room, if that user
+        is allowed to participate in that room. After this call, the client is
+        allowed to see all current state events in the room, and all subsequent
+        events associated with the room until the user leaves the room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier or room alias to join.
+          required: true
+          x-example: "#monkeys:matrix.org"
+      responses:
+        200:
+          description: |-
+            The room has been joined.
+
+            If the room was joined with an room alias, rather than a room ID,
+            the room ID must be returned in the room_id field.
+
+            If the room was joined with a room ID, the room_id field must not be
+            present.
+          examples:
+            application/json: |-
+              {"room_id": "!primates:matrix.org"}
+          schema:
+            type: object  # empty json object
+        403:
+          description: |-
+            You do not have permission to join the room. A meaningful errcode and description error text will be returned. Example reasons for rejection are:
+              - The room is invite-only and the user was not invited.
+              - The user has been banned from the room.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN", "error": "You are not invited to this room."}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+  "/join/{roomId}":
+    post:
+      x-alias-for-path: "/room/{roomId}/join"
+      x-alias-link: "post-matrix-client-api-v1-room-roomid-join"
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 83c8b7a8..d0942ad6 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -880,43 +880,7 @@ certain operations such as kicking, banning and sending state events. See
 `m.room.power_levels`_ for more information.
 
 
-Joining rooms
-~~~~~~~~~~~~~
-.. TODO-doc What does the home server have to do to join a user to a room?
-   -  See SPEC-30.
-
-Users need to join a room in order to send and receive events in that room. A
-user can join a room by making a request to |/join/<room_alias_or_id>|_ with::
-
-  {}
-
-Alternatively, a user can make a request to |/rooms/<room_id>/join|_ with the
-same request content.  This is only provided for symmetry with the other
-membership APIs: ``/rooms/<room id>/invite`` and ``/rooms/<room id>/leave``. If
-a room alias was specified, it will be automatically resolved to a room ID,
-which will then be joined. The room ID that was joined will be returned in
-response::
-
-  {
-    "room_id": "!roomid:domain"
-  }
-
-The membership state for the joining user can also be modified directly to be
-``join`` by sending the following request to
-``/rooms/<room id>/state/m.room.member/<url encoded user id>``::
-
-  {
-    "membership": "join"
-  }
-
-See the `Room events`_ section for more information on ``m.room.member``.
-
-After the user has joined a room, they will receive subsequent events in that
-room. This room will now appear as an entry in the |initialSync|_ API.
-
-Some rooms enforce that a user is *invited* to a room before they can join that
-room. Other rooms will allow anyone to join the room even if they have not
-received an invite.
+{{membership_http_api}}
 
 Inviting users
 ~~~~~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 7bc25095..7e21434f 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -90,6 +90,12 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
+    def render_membership_http_api(self):
+        return self._render_http_api_group(
+            "membership",
+            title_kind="~"
+        )
+
     def render_room_events(self):
         def filterFn(eventType):
             return (

From e58f816ad39c26461527c2ba47d2db8d544f7cb1 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 8 Sep 2015 16:49:14 +0100
Subject: [PATCH 035/467] Add m.receipt schema

---
 event-schemas/examples/v1/m.receipt | 13 +++++++++
 event-schemas/schema/v1/m.receipt   | 43 +++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 event-schemas/examples/v1/m.receipt
 create mode 100644 event-schemas/schema/v1/m.receipt

diff --git a/event-schemas/examples/v1/m.receipt b/event-schemas/examples/v1/m.receipt
new file mode 100644
index 00000000..83515317
--- /dev/null
+++ b/event-schemas/examples/v1/m.receipt
@@ -0,0 +1,13 @@
+{
+    "type": "m.receipt",
+    "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org",
+    "content": {
+        "$1435641916114394fHBLK:matrix.org": {
+            "read": {
+                "@rikj:jki.re": {
+                    "ts": 1436451550453
+                }
+            }
+        }
+    }
+}
diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
new file mode 100644
index 00000000..b9d29c30
--- /dev/null
+++ b/event-schemas/schema/v1/m.receipt
@@ -0,0 +1,43 @@
+{
+    "$schema": "http://json-schema.org/draft-04/schema#",
+    "type": "object",
+    "title": "Receipt Event",
+    "description": "Informs the client of new receipts.",
+    "properties": {
+        "content": {
+            "type": "object",
+            "description": "The event ids which the receipts relate to.",
+            "patternProperties": {
+                "^\$": {
+                    "type": "object",
+                    "description": "The types of the receipts."
+                    "additionalProperties": {
+                        "type": "object",
+                        "description": "User ids of the receipts",
+                        "patternProperties": {
+                            "^@": {
+                                "type": "object",
+                                "properties": {
+                                    "ts": {
+                                        "type": "number",
+                                        "description": "The timestamp the receipt was sent at"
+                                    }
+                                }
+                            }
+                        },
+                        "additionalProperties": false
+                    }
+                }
+            },
+            "additionalProperties": false
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.presence"]
+        },
+        "room_id": {
+            "type": "string"
+        }
+    },
+    "required": ["room_id", "type", "content"]
+}

From 5157c2f52ed6dfc3019b097387d6c3678413a87e Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 8 Sep 2015 16:51:08 +0100
Subject: [PATCH 036/467] More escapes

---
 event-schemas/schema/v1/m.receipt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index b9d29c30..9848f6c6 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -8,7 +8,7 @@
             "type": "object",
             "description": "The event ids which the receipts relate to.",
             "patternProperties": {
-                "^\$": {
+                "^\\$": {
                     "type": "object",
                     "description": "The types of the receipts."
                     "additionalProperties": {

From 53756cfd73ed18d74839b16fe4b697f0dd25cc70 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 8 Sep 2015 16:52:43 +0100
Subject: [PATCH 037/467] More commas

---
 event-schemas/schema/v1/m.receipt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index 9848f6c6..3f242eb0 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -10,7 +10,7 @@
             "patternProperties": {
                 "^\\$": {
                     "type": "object",
-                    "description": "The types of the receipts."
+                    "description": "The types of the receipts.",
                     "additionalProperties": {
                         "type": "object",
                         "description": "User ids of the receipts",

From 500b4eb32de4054281b5bbeda69df1dc5f4fdf96 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 8 Sep 2015 16:54:14 +0100
Subject: [PATCH 038/467] Typo

---
 event-schemas/schema/v1/m.receipt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index 3f242eb0..4bd9f17b 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -33,7 +33,7 @@
         },
         "type": {
             "type": "string",
-            "enum": ["m.presence"]
+            "enum": ["m.receipt"]
         },
         "room_id": {
             "type": "string"

From 90f5dc370b8e00b2f57ba98f0846b8bbbd7c1e12 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 16:49:28 +0100
Subject: [PATCH 039/467] Populate aliases from canonical endpoint

Swagger validates badly if you have endpoints without the full
specification of things, so instead let's generate them the other way
around.
---
 templating/matrix_templates/units.py | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 89b34bb1..8e991fa8 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -105,19 +105,7 @@ class MatrixUnits(Units):
         for path in api["paths"]:
             for method in api["paths"][path]:
                 single_api = api["paths"][path][method]
-
                 full_path = api.get("basePath", "") + path
-                alias_for_path = single_api.get("x-alias-for-path", "")
-                alias_link = single_api.get("x-alias-link", "")
-                if alias_for_path:
-                    endpoints.append({
-                        "method": method.upper(),
-                        "path": full_path,
-                        "alias_for_path": alias_for_path,
-                        "alias_link": alias_link
-                    })
-                    continue
-
                 endpoint = {
                     "title": single_api.get("summary", ""),
                     "desc": single_api.get("description", single_api.get("summary", "")),
@@ -260,6 +248,16 @@ class MatrixUnits(Units):
 
                 endpoints.append(endpoint)
 
+                for alias in single_api.get("x-aliases", []):
+                    canonical_alias_link = single_api.get("x-alias-link")
+                    alias_link = single_api.get("x-alias-link", "")
+                    endpoints.append({
+                        "method": method.upper(),
+                        "path": alias,
+                        "alias_for_path": full_path,
+                        "alias_link": alias_link
+                    })
+
         return {
             "base": api.get("basePath"),
             "group": group_name,

From 1feb9565e4727f9c7c10619b84bb5f77976dd513 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 16:51:39 +0100
Subject: [PATCH 040/467] Use other-way-around alias format

---
 api/client-server/v1/membership.yaml | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 24f3835d..1845ce09 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -62,7 +62,6 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
-  "/join/{roomId}":
-    post:
-      x-alias-for-path: "/room/{roomId}/join"
+      x-aliases:
+        - /join/{roomId}
       x-alias-link: "post-matrix-client-api-v1-room-roomid-join"

From 64cfc00f5055959f4a67d51d29711c7f4b746e97 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 8 Sep 2015 17:07:52 +0100
Subject: [PATCH 041/467] Remove unused fluff

---
 templating/matrix_templates/units.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 8e991fa8..24fb3767 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -249,8 +249,7 @@ class MatrixUnits(Units):
                 endpoints.append(endpoint)
 
                 for alias in single_api.get("x-aliases", []):
-                    canonical_alias_link = single_api.get("x-alias-link")
-                    alias_link = single_api.get("x-alias-link", "")
+                    alias_link = single_api.get("x-alias-link")
                     endpoints.append({
                         "method": method.upper(),
                         "path": alias,

From a92fa6392d1b7eb9dd9c75f6d209506a2898ca30 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 8 Sep 2015 17:14:31 +0100
Subject: [PATCH 042/467] Include patternProperties as normal properties (they
 basically are just patterns instead of keys; we could probably annotate this
 more nicely in the future)

---
 templating/matrix_templates/units.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index d607160f..2ae6cd0b 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -30,7 +30,7 @@ def get_json_schema_object_fields(obj, enforce_title=False):
     }
     tables = [fields]
 
-    props = obj.get("properties")
+    props = obj.get("properties", obj.get("patternProperties"))
     parents = obj.get("allOf")
     if not props and not parents:
         raise Exception(

From d399e5b93bac92bcbe82a79e6aecb90a92f38fe7 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 9 Sep 2015 13:16:46 +0100
Subject: [PATCH 043/467] Use an alias object rather than flat fields

---
 templating/matrix_templates/units.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 24fb3767..d8cc213d 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -248,14 +248,16 @@ class MatrixUnits(Units):
 
                 endpoints.append(endpoint)
 
-                for alias in single_api.get("x-aliases", []):
-                    alias_link = single_api.get("x-alias-link")
-                    endpoints.append({
-                        "method": method.upper(),
-                        "path": alias,
-                        "alias_for_path": full_path,
-                        "alias_link": alias_link
-                    })
+                aliases = single_api.get("x-alias", None)
+                if aliases:
+                    alias_link = aliases["canonical-link"]
+                    for alias in aliases["aliases"]:
+                        endpoints.append({
+                            "method": method.upper(),
+                            "path": alias,
+                            "alias_for_path": full_path,
+                            "alias_link": alias_link
+                        })
 
         return {
             "base": api.get("basePath"),

From 04b2b2588f1453fb62d9e86277b0f2cefc5a6207 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 9 Sep 2015 13:18:23 +0100
Subject: [PATCH 044/467] Fix up formatting and typos

---
 api/client-server/v1/membership.yaml   | 22 +++++++++++-----------
 specification/10_client_server_api.rst | 10 ++++++++++
 2 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 1845ce09..b86e8b03 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -18,7 +18,7 @@ securityDefinitions:
     name: access_token
     in: query
 paths:
-  "/room/{roomId}/join":
+  "/rooms/{roomId}/join":
     post:
       summary: Start the requesting user participating in a particular room.
       description: |-
@@ -26,6 +26,9 @@ paths:
         is allowed to participate in that room. After this call, the client is
         allowed to see all current state events in the room, and all subsequent
         events associated with the room until the user leaves the room.
+
+        After a user has joined a room, the room will appear as an entry in the
+        response of the |initialSync| API.
       security:
         - accessToken: []
       parameters:
@@ -40,19 +43,15 @@ paths:
           description: |-
             The room has been joined.
 
-            If the room was joined with an room alias, rather than a room ID,
-            the room ID must be returned in the room_id field.
-
-            If the room was joined with a room ID, the room_id field must not be
-            present.
+            The joined room ID must be returned in the room_id field.
           examples:
             application/json: |-
-              {"room_id": "!primates:matrix.org"}
+              {"room_id": "!d41d8cd:matrix.org"}
           schema:
             type: object  # empty json object
         403:
           description: |-
-            You do not have permission to join the room. A meaningful errcode and description error text will be returned. Example reasons for rejection are:
+            You do not have permission to join the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejection are:
               - The room is invite-only and the user was not invited.
               - The user has been banned from the room.
           examples:
@@ -62,6 +61,7 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
-      x-aliases:
-        - /join/{roomId}
-      x-alias-link: "post-matrix-client-api-v1-room-roomid-join"
+      x-alias:
+        canonical-link: "post-matrix-client-api-v1-rooms-roomid-join"
+        aliases:
+          - /join/{roomId}
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index d0942ad6..1fdb9ae6 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -879,6 +879,16 @@ The keys contained in ``m.room.power_levels`` determine the levels required for
 certain operations such as kicking, banning and sending state events. See
 `m.room.power_levels`_ for more information.
 
+Joining rooms
+-------------
+Users need to be a member of a room in order to send and receive events in that
+room. There are several states in which a user may be, in relation to a room:
+
+ - Unrelated (the user cannot send or receive events in the room)
+ - Invited (the user has been invited to participate in the room, but is not
+   yet participating)
+ - Joined (the user can send and receive events in the room)
+ - Banned (the user is not allowed to join the room)
 
 {{membership_http_api}}
 

From 267d2bd3decd69002044b417784c30389fef3856 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 9 Sep 2015 13:40:32 +0100
Subject: [PATCH 045/467] Add api/node_modules to .gitignore

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 63ef5bb3..f4177c79 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,3 +3,4 @@ templating/out
 *.pyc
 supporting-docs/_site
 supporting-docs/.sass-cache
+api/node_modules

From 1b591a023ecc5a167d630a6e1dad88981edcfefc Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 9 Sep 2015 14:04:48 +0100
Subject: [PATCH 046/467] Monospace room_id

---
 api/client-server/v1/membership.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index b86e8b03..d847a78f 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -43,7 +43,7 @@ paths:
           description: |-
             The room has been joined.
 
-            The joined room ID must be returned in the room_id field.
+            The joined room ID must be returned in the ``room_id`` field.
           examples:
             application/json: |-
               {"room_id": "!d41d8cd:matrix.org"}

From 2797fac3aa2a45049d3ee1932a7470fbb10b9c3e Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 9 Sep 2015 17:55:40 +0100
Subject: [PATCH 047/467] Swagger /room/{roomId}/invite

This is a transcription of the current spec, with nothing new or
controversial in it.
---
 api/client-server/v1/membership.yaml   | 64 +++++++++++++++++++++++++-
 specification/10_client_server_api.rst | 33 ++-----------
 2 files changed, 66 insertions(+), 31 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index d847a78f..86e84111 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -48,7 +48,7 @@ paths:
             application/json: |-
               {"room_id": "!d41d8cd:matrix.org"}
           schema:
-            type: object  # empty json object
+            type: object
         403:
           description: |-
             You do not have permission to join the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejection are:
@@ -65,3 +65,65 @@ paths:
         canonical-link: "post-matrix-client-api-v1-rooms-roomid-join"
         aliases:
           - /join/{roomId}
+
+  "/rooms/{roomId}/invite":
+    post:
+      summary: Invite a user to participate in a particular room.
+      # It's a crying shame that I don't know how to force line breaks.
+      description: |-
+        This API invites a user to participate in a particular room.
+        They do not start participating in the room until they actually join the
+        room.
+
+        This serves two purposes; firstly, to notify the user that the room
+        exists (and that their presence is requested). Secondly, some rooms can
+        only be joined if a user is invited to join it; sending the invite gives
+        that user permission to join the room.
+
+        Only users currently in a particular room can invite other users to
+        join that room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier or room alias to which to invite the user.
+          required: true
+          x-example: "#monkeys:matrix.org"
+        - in: body
+          name: user_id
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "user_id": "@cheeky_monkey:matrix.org"
+              }
+            properties:
+              user_id:
+                type: string
+                description: The fully qualified user ID of the invitee.
+            required: ["user_id"]
+      responses:
+        200:
+          description: The user has been invited to join the room.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
+        403:
+          description: |-
+            You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
+              - The invitee has been banned from the room.
+              - The invitee is already a member of the room.
+              - The inviter is not currently in the room.
+              - The inviter's power level is insufficient to invite users to the room.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN", "error": "@cheeky_monkey:matrix.org is banned from the room"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 1fdb9ae6..f0d6be42 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -890,17 +890,9 @@ room. There are several states in which a user may be, in relation to a room:
  - Joined (the user can send and receive events in the room)
  - Banned (the user is not allowed to join the room)
 
-{{membership_http_api}}
-
-Inviting users
-~~~~~~~~~~~~~~
-.. TODO-doc Invite-join dance
-  - Outline invite join dance. What is it? Why is it required? How does it work?
-  - What does the home server have to do?
+Some rooms require that users be invited to it before they can join; others
+allow anyone to join.
 
-The purpose of inviting users to a room is to notify them that the room exists
-so they can choose to become a member of that room. Some rooms require that all
-users who join a room are previously invited to it (an "invite-only" room).
 Whether a given room is an "invite-only" room is determined by the room config
 key ``m.room.join_rules``. It can have one of the following values:
 
@@ -910,26 +902,7 @@ key ``m.room.join_rules``. It can have one of the following values:
 ``invite``
   This room can only be joined if you were invited.
 
-Only users who have a membership state of ``join`` in a room can invite new
-users to said room. The person being invited must not be in the ``join`` state
-in the room. The fully-qualified user ID must be specified when inviting a
-user, as the user may reside on a different home server. To invite a user, send
-the following request to |/rooms/<room_id>/invite|_, which will manage the
-entire invitation process::
-
-  {
-    "user_id": "<user id to invite>"
-  }
-
-Alternatively, the membership state for this user in this room can be modified
-directly by sending the following request to
-``/rooms/<room id>/state/m.room.member/<url encoded user id>``::
-
-  {
-    "membership": "invite"
-  }
-
-See the `Room events`_ section for more information on ``m.room.member``.
+{{membership_http_api}}
 
 Leaving rooms
 ~~~~~~~~~~~~~

From c4acee3bcbf8fc99d93a912f09c565729f4aad39 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 11:16:31 +0100
Subject: [PATCH 048/467] continuserv: Live serves updates to the spec

---
 .gitignore                  |   1 +
 scripts/continuserv/main.go | 129 ++++++++++++++++++++++++++++++++++++
 2 files changed, 130 insertions(+)
 create mode 100644 scripts/continuserv/main.go

diff --git a/.gitignore b/.gitignore
index f4177c79..e2250131 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
 scripts/gen
+scripts/continuserv/continuserv
 templating/out
 *.pyc
 supporting-docs/_site
diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
new file mode 100644
index 00000000..26dfc4a1
--- /dev/null
+++ b/scripts/continuserv/main.go
@@ -0,0 +1,129 @@
+// continuserv proactively re-generates the spec on filesystem changes, and serves it over HTTP.
+// It will always serve the most recent version of the spec, and may block an HTTP request until regeneration is finished.
+// It does not currently pre-empt stale generations, but will block until they are complete.
+package main
+
+import (
+	"bytes"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"net/http"
+	"os"
+	"os/exec"
+	"path"
+	"path/filepath"
+	"strings"
+	"sync"
+	"sync/atomic"
+
+	fsnotify "gopkg.in/fsnotify.v1"
+)
+
+var toServe atomic.Value // Always contains valid []byte to serve. May be stale unless wg is zero.
+var wg sync.WaitGroup    // Indicates how many updates are pending.
+var mu sync.Mutex        // Prevent multiple updates in parallel.
+
+func main() {
+	w, err := fsnotify.NewWatcher()
+	if err != nil {
+		log.Fatalf("Error making watcher: %v", err)
+	}
+
+	dir, err := os.Getwd()
+	if err != nil {
+		log.Fatalf("Error getting wd: %v", err)
+	}
+	for ; !exists(path.Join(dir, ".git")); dir = path.Dir(dir) {
+		if dir == "/" {
+			log.Fatalf("Could not find git root")
+		}
+	}
+
+	filepath.Walk(dir, makeWalker(w))
+
+	wg.Add(1)
+	populateOnce(dir)
+
+	ch := make(chan struct{}, 100) // Buffered to ensure we can multiple-increment wg for pending writes
+	go doPopulate(ch, dir)
+
+	http.HandleFunc("/", serve)
+	go http.ListenAndServe(":8000", nil)
+
+	for {
+		select {
+		case e := <-w.Events:
+			if filter(e) {
+				wg.Add(1)
+				ch <- struct{}{}
+			}
+		}
+	}
+}
+
+func makeWalker(w *fsnotify.Watcher) filepath.WalkFunc {
+	return func(path string, _ os.FileInfo, err error) error {
+		if err != nil {
+			log.Fatalf("Error walking: %v", err)
+		}
+		w.Add(path)
+		return nil
+	}
+}
+
+// Return true if event should trigger re-population
+func filter(e fsnotify.Event) bool {
+	// vim is *really* noisy about how it writes files
+	if e.Op != fsnotify.Write {
+		return false
+	}
+	// Avoid some temp files that vim writes
+	if strings.HasSuffix(e.Name, "~") || strings.HasSuffix(e.Name, ".swp") || strings.HasPrefix(e.Name, ".") {
+		return false
+	}
+
+	// Avoid infinite cycles being caused by writing actual output
+	if strings.Contains(e.Name, "/tmp/") || strings.Contains(e.Name, "/gen/") {
+		return false
+	}
+	return true
+}
+
+func serve(w http.ResponseWriter, req *http.Request) {
+	wg.Wait()
+	b := toServe.Load().([]byte)
+	w.Write(b)
+}
+
+func populateOnce(dir string) {
+	defer wg.Done()
+	mu.Lock()
+	defer mu.Unlock()
+	cmd := exec.Command("python", "gendoc.py")
+	cmd.Dir = path.Join(dir, "scripts")
+	var b bytes.Buffer
+	cmd.Stderr = &b
+	err := cmd.Run()
+	if err != nil {
+		toServe.Store([]byte(fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String()).Error()))
+		return
+	}
+	specBytes, err := ioutil.ReadFile(path.Join(dir, "scripts", "gen", "specification.html"))
+	if err != nil {
+		toServe.Store([]byte(fmt.Errorf("error reading spec: %v", err).Error()))
+		return
+	}
+	toServe.Store(specBytes)
+}
+
+func doPopulate(ch chan struct{}, dir string) {
+	for _ = range ch {
+		populateOnce(dir)
+	}
+}
+
+func exists(path string) bool {
+	_, err := os.Stat(path)
+	return !os.IsNotExist(err)
+}

From a54814856357db32f2e430561a9e8fd5aa381024 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 11:21:18 +0100
Subject: [PATCH 049/467] Invite doesn't accept room aliases

---
 api/client-server/v1/membership.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 86e84111..04922656 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -88,9 +88,9 @@ paths:
         - in: path
           type: string
           name: roomId
-          description: The room identifier or room alias to which to invite the user.
+          description: The room identifier (not alias) to which to invite the user.
           required: true
-          x-example: "#monkeys:matrix.org"
+          x-example: "!d41d8cd:matrix.org"
         - in: body
           name: user_id
           required: true

From e30272733b48a3afca3c2e5e6fd87ef1a0ff9727 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 11:29:16 +0100
Subject: [PATCH 050/467] continuserv: Make port flag-specified

---
 scripts/continuserv/main.go | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index 26dfc4a1..c5d623a4 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -5,6 +5,7 @@ package main
 
 import (
 	"bytes"
+	"flag"
 	"fmt"
 	"io/ioutil"
 	"log"
@@ -20,11 +21,17 @@ import (
 	fsnotify "gopkg.in/fsnotify.v1"
 )
 
-var toServe atomic.Value // Always contains valid []byte to serve. May be stale unless wg is zero.
-var wg sync.WaitGroup    // Indicates how many updates are pending.
-var mu sync.Mutex        // Prevent multiple updates in parallel.
+var (
+	port = flag.Int("port", 8000, "Port on which to serve HTTP")
+
+	toServe atomic.Value   // Always contains valid []byte to serve. May be stale unless wg is zero.
+	wg      sync.WaitGroup // Indicates how many updates are pending.
+	mu      sync.Mutex     // Prevent multiple updates in parallel.
+)
 
 func main() {
+	flag.Parse()
+
 	w, err := fsnotify.NewWatcher()
 	if err != nil {
 		log.Fatalf("Error making watcher: %v", err)
@@ -48,9 +55,14 @@ func main() {
 	ch := make(chan struct{}, 100) // Buffered to ensure we can multiple-increment wg for pending writes
 	go doPopulate(ch, dir)
 
+	go watchFS(ch, w)
+
 	http.HandleFunc("/", serve)
-	go http.ListenAndServe(":8000", nil)
+	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
+
+}
 
+func watchFS(ch chan struct{}, w *fsnotify.Watcher) {
 	for {
 		select {
 		case e := <-w.Events:

From 55f33736b2efdb21507abb0714cd2ff36357334b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 12:04:12 +0100
Subject: [PATCH 051/467] Preserve newlines in wrapped text

---
 templating/build.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/templating/build.py b/templating/build.py
index dadf91ce..258e4a17 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -78,8 +78,10 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
     def wrap(input, wrap=80, initial_indent=""):
         if len(input) == 0:
             return initial_indent
+        input_lines = input.split('\n')
         wrapper = TextWrapper(initial_indent=initial_indent, width=wrap)
-        return wrapper.fill(input)
+        output_lines = [wrapper.fill(line) for line in input_lines]
+        return '\n'.join(output_lines)
 
     # make Jinja aware of the templates and filters
     env = Environment(

From f67e27e843bd812360f095a242be584f40719052 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 13:20:04 +0100
Subject: [PATCH 052/467] Split on double-newlines not single

Otherwise all sorts of tables get horribly broken
---
 templating/build.py | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/templating/build.py b/templating/build.py
index 258e4a17..6756bed9 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -76,12 +76,10 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
         return " "*indent + input
 
     def wrap(input, wrap=80, initial_indent=""):
-        if len(input) == 0:
-            return initial_indent
-        input_lines = input.split('\n')
+        input_lines = input.split('\n\n')
         wrapper = TextWrapper(initial_indent=initial_indent, width=wrap)
         output_lines = [wrapper.fill(line) for line in input_lines]
-        return '\n'.join(output_lines)
+        return '\n\n'.join(output_lines)
 
     # make Jinja aware of the templates and filters
     env = Environment(

From 6a56c8a96559b576aa6925d44d86785a6329f199 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 13:20:56 +0100
Subject: [PATCH 053/467] Add trailing newline

---
 api/client-server/v1/presence.yaml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 16665341..6f8311e4 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -206,4 +206,3 @@ paths:
               title: PresenceEvent
               allOf:
                 - "$ref": "definitions/event.yaml"
-  
\ No newline at end of file

From ad595aea459eedf92f568ff63e0e3522cea8c86c Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 13:21:08 +0100
Subject: [PATCH 054/467] Optionally don't delete intermediate rst files

Useful for debugging
Hackily hackily implemented :)
---
 scripts/gendoc.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index e3e8d3d4..47141d7d 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -71,10 +71,11 @@ def main():
     run_through_template("tmp/howto.rst")
     rst2html("tmp/full_spec.rst", "gen/specification.html")
     rst2html("tmp/howto.rst", "gen/howtos.html")
-    cleanup_env()
+    if "--nodelete" not in sys.argv:
+        cleanup_env()
 
 if __name__ == '__main__':
-    if len(sys.argv) > 1:
+    if len(sys.argv) > 1 and sys.argv[1:] != ["--nodelete"]:
         # we accept no args, so they don't know what they're doing!
         print "gendoc.py - Generate the Matrix specification as HTML."
         print "Usage:"

From a74fb54053eb9c247a4af1f5fa3fe59e891204d5 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 13:44:41 +0100
Subject: [PATCH 055/467] Correct comment

---
 scripts/gendoc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 47141d7d..57a04717 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -76,7 +76,7 @@ def main():
 
 if __name__ == '__main__':
     if len(sys.argv) > 1 and sys.argv[1:] != ["--nodelete"]:
-        # we accept no args, so they don't know what they're doing!
+        # we accept almost no args, so they don't know what they're doing!
         print "gendoc.py - Generate the Matrix specification as HTML."
         print "Usage:"
         print "  python gendoc.py"

From cdc06a8bf1c410147a7cd096c2a9169914285852 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 13:53:38 +0100
Subject: [PATCH 056/467] Document --nodelete

---
 scripts/gendoc.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 57a04717..e3c3797e 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -79,9 +79,11 @@ if __name__ == '__main__':
         # we accept almost no args, so they don't know what they're doing!
         print "gendoc.py - Generate the Matrix specification as HTML."
         print "Usage:"
-        print "  python gendoc.py"
+        print "  python gendoc.py [--nodelete]"
         print ""
         print "The specification can then be found in the gen/ folder."
+        print ("If --nodelete was specified, intermediate files will be "
+               "present in the tmp/ folder.")
         print ""
         print "Requirements:"
         print " - This script requires Jinja2 and rst2html (docutils)."

From 99106bb6fefd0b2f44c1ff418491f7d54c52cc8c Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 13:54:00 +0100
Subject: [PATCH 057/467] Add explanatory comments

Also, remove line which was accidentally removed
---
 templating/build.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/templating/build.py b/templating/build.py
index 6756bed9..ac3d2491 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -76,6 +76,11 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
         return " "*indent + input
 
     def wrap(input, wrap=80, initial_indent=""):
+        if len(input) == 0:
+            return initial_indent
+        # TextWrapper collapses newlines into single spaces; we do our own
+        # splitting on newlines to prevent this, so that newlines can actually
+        # be intentionally inserted in text.
         input_lines = input.split('\n\n')
         wrapper = TextWrapper(initial_indent=initial_indent, width=wrap)
         output_lines = [wrapper.fill(line) for line in input_lines]

From 34091038f39a332c70ac188992437ae7a2e79003 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 14:19:06 +0100
Subject: [PATCH 058/467] speculator: Tool to preview spec pull requests

Not the prettiest thing in the world, and horribly unperformant, but
useful :)
---
 scripts/speculator/README  |   4 +
 scripts/speculator/main.go | 220 +++++++++++++++++++++++++++++++++++++
 2 files changed, 224 insertions(+)
 create mode 100644 scripts/speculator/README
 create mode 100644 scripts/speculator/main.go

diff --git a/scripts/speculator/README b/scripts/speculator/README
new file mode 100644
index 00000000..1da99cfb
--- /dev/null
+++ b/scripts/speculator/README
@@ -0,0 +1,4 @@
+speculator allows you to preview pull requests to the matrix.org specification.
+
+To run it, you must install the `go` tool, and run:
+ `go run main.go`
diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
new file mode 100644
index 00000000..59047e68
--- /dev/null
+++ b/scripts/speculator/main.go
@@ -0,0 +1,220 @@
+// speculator allows you to preview pull requests to the matrix.org specification.
+// It serves two HTTP endpoints:
+//  - /spec/123 which renders the spec as html at pull request 123.
+//  - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
+// It is currently woefully inefficient, and there is a lot of low hanging fruit for improvement.
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"os/exec"
+	"path"
+	"strconv"
+	"strings"
+	"syscall"
+)
+
+type PullRequest struct {
+	Base Commit
+	Head Commit
+	User User
+}
+
+type Commit struct {
+	SHA  string
+	Repo RequestRepo
+}
+
+type RequestRepo struct {
+	CloneURL string `json:"clone_url"`
+}
+
+type User struct {
+	Login string
+}
+
+var (
+	port           = flag.Int("port", 9000, "Port on which to listen for HTTP")
+	allowedMembers map[string]bool
+)
+
+func gitClone(url string) (string, error) {
+	dst := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10))
+	cmd := exec.Command("git", "clone", url, dst)
+	err := cmd.Run()
+	if err != nil {
+		return "", fmt.Errorf("error cloning repo: %v", err)
+	}
+	return dst, nil
+}
+
+func gitCheckout(path, sha string) error {
+	cmd := exec.Command("git", "checkout", sha)
+	cmd.Dir = path
+	err := cmd.Run()
+	if err != nil {
+		return fmt.Errorf("error checking out repo: %v", err)
+	}
+	return nil
+}
+
+func lookupPullRequest(prNumber string) (PullRequest, error) {
+	resp, _ := http.Get("https://api.github.com/repos/matrix-org/matrix-doc/pulls/" + prNumber)
+	defer resp.Body.Close()
+	dec := json.NewDecoder(resp.Body)
+	var pr PullRequest
+	_ = dec.Decode(&pr)
+	return pr, nil
+}
+
+func generate(dir string) error {
+	cmd := exec.Command("python", "gendoc.py", "--nodelete")
+	cmd.Dir = path.Join(dir, "scripts")
+	var b bytes.Buffer
+	cmd.Stderr = &b
+	err := cmd.Run()
+	if err != nil {
+		return fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())
+	}
+	return nil
+}
+
+func writeError(w http.ResponseWriter, err error) {
+	w.WriteHeader(500)
+	io.WriteString(w, fmt.Sprintf("%v\n", err))
+}
+
+// generateAt generates spec from repo at sha.
+// Returns the path where the generation was done.
+func generateAt(repo, sha string) (dst string, err error) {
+	dst, err = gitClone(repo)
+	if err != nil {
+		return
+	}
+
+	if err = gitCheckout(dst, sha); err != nil {
+		return
+	}
+
+	err = generate(dst)
+	return
+}
+
+func serveSpec(w http.ResponseWriter, req *http.Request) {
+	parts := strings.Split(req.URL.Path, "/")
+	if len(parts) != 3 {
+		w.WriteHeader(400)
+		io.WriteString(w, fmt.Sprintf("Invalid path passed: %v expect /pull/123", req.URL.Path))
+		return
+	}
+
+	pr, err := lookupPullRequest(parts[2])
+	if err != nil {
+		writeError(w, err)
+		return
+	}
+
+	if !allowedMembers[pr.User.Login] {
+		w.WriteHeader(403)
+		io.WriteString(w, fmt.Sprintf("%q is not a trusted pull requester", pr.User.Login))
+		return
+	}
+
+	dst, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
+	defer os.RemoveAll(dst)
+	if err != nil {
+		writeError(w, err)
+		return
+	}
+
+	b, err := ioutil.ReadFile(path.Join(dst, "scripts/gen/specification.html"))
+	if err != nil {
+		writeError(w, fmt.Errorf("Error reading spec: %v", err))
+		return
+	}
+	w.Write(b)
+}
+
+func serveRstDiff(w http.ResponseWriter, req *http.Request) {
+	parts := strings.Split(req.URL.Path, "/")
+	if len(parts) != 4 {
+		w.WriteHeader(400)
+		io.WriteString(w, fmt.Sprintf("Invalid path passed: %v expect /diff/rst/123", req.URL.Path))
+		return
+	}
+
+	pr, err := lookupPullRequest(parts[3])
+	if err != nil {
+		writeError(w, err)
+		return
+	}
+
+	if !allowedMembers[pr.User.Login] {
+		w.WriteHeader(403)
+		io.WriteString(w, fmt.Sprintf("%q is not a trusted pull requester", pr.User.Login))
+		return
+	}
+
+	base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA)
+	defer os.RemoveAll(base)
+	if err != nil {
+		writeError(w, err)
+		return
+	}
+
+	head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
+	defer os.RemoveAll(head)
+	if err != nil {
+		writeError(w, err)
+		return
+	}
+
+	diffCmd := exec.Command("diff", path.Join(base, "scripts", "tmp", "full_spec.rst"), path.Join(head, "scripts", "tmp", "full_spec.rst"))
+	var diff bytes.Buffer
+	diffCmd.Stdout = &diff
+	if err := ignoreExitCodeOne(diffCmd.Run()); err != nil {
+		writeError(w, fmt.Errorf("error running diff: %v", err))
+		return
+	}
+	w.Write(diff.Bytes())
+}
+
+func ignoreExitCodeOne(err error) error {
+	if err == nil {
+		return err
+	}
+
+	if exiterr, ok := err.(*exec.ExitError); ok {
+		if status, ok := exiterr.Sys().(syscall.WaitStatus); ok {
+			if status.ExitStatus() == 1 {
+				return nil
+			}
+		}
+	}
+	return err
+}
+
+func main() {
+	flag.Parse()
+	// It would be great to read this from github, but there's no convenient way to do so.
+	// Most of these memberships are "private", so would require some kind of auth.
+	allowedMembers = map[string]bool{
+		"dbkr":          true,
+		"erikjohnston":  true,
+		"illicitonion":  true,
+		"Kegsay":        true,
+		"NegativeMjark": true,
+	}
+	http.HandleFunc("/spec/", serveSpec)
+	http.HandleFunc("/diff/rst/", serveRstDiff)
+	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
+}

From 6e86cb34d2e8864b69c50faa4ab9b165e47857a7 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 14:25:23 +0100
Subject: [PATCH 059/467] continuserv: Adding README

---
 scripts/continuserv/README | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 scripts/continuserv/README

diff --git a/scripts/continuserv/README b/scripts/continuserv/README
new file mode 100644
index 00000000..8ce37850
--- /dev/null
+++ b/scripts/continuserv/README
@@ -0,0 +1,6 @@
+continuserv proactively re-generates the spec on filesystem changes, and serves it over HTTP.
+
+To run it, you must install the `go` tool. You will also need to install fsnotify by running:
+ `go get gopkg.in/fsnotify.v1`
+You can then run continuserv by running:
+ `go run main.go`

From 6ac519d9dce4016b9911b5f6ddc05889b8c2abd2 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 14:25:37 +0100
Subject: [PATCH 060/467] Log on file updates

---
 scripts/continuserv/main.go | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index c5d623a4..b4224772 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -68,6 +68,7 @@ func watchFS(ch chan struct{}, w *fsnotify.Watcher) {
 		case e := <-w.Events:
 			if filter(e) {
 				wg.Add(1)
+				fmt.Printf("Noticed change to %s, re-generating spec\n", e.Name)
 				ch <- struct{}{}
 			}
 		}

From c29aef362e11ed31300f0f66a54536a4644c9a51 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 15:45:21 +0100
Subject: [PATCH 061/467] Error on failure of Watcher.Add

---
 scripts/continuserv/main.go | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index b4224772..e7757c06 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -80,7 +80,9 @@ func makeWalker(w *fsnotify.Watcher) filepath.WalkFunc {
 		if err != nil {
 			log.Fatalf("Error walking: %v", err)
 		}
-		w.Add(path)
+		if err := w.Add(path); err != nil {
+			log.Fatalf("Failed to add watch: %v", err)
+		}
 		return nil
 	}
 }

From 306f91edb318eb6ecc791e8e81c5b6b40ba829c8 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 16:46:22 +0100
Subject: [PATCH 062/467] Specify third party room invitations

SYN-458
---
 api/client-server/v1/membership.yaml | 116 ++++++++++++++++++++++++++-
 templating/matrix_templates/units.py |   2 +-
 2 files changed, 113 insertions(+), 5 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 04922656..17b5795e 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -66,11 +66,16 @@ paths:
         aliases:
           - /join/{roomId}
 
-  "/rooms/{roomId}/invite":
+  # With an extra " " to disambiguate from the 3pid invite endpoint
+  # The extra space makes it sort first for what I'm sure is a good reason.
+  "/rooms/{roomId}/invite ":
     post:
       summary: Invite a user to participate in a particular room.
-      # It's a crying shame that I don't know how to force line breaks.
       description: |-
+        *Note that there are two forms of this API, which are documented separately.
+        This version of the API requires that the inviter knows the Matrix
+        identifier of the invitee.*
+
         This API invites a user to participate in a particular room.
         They do not start participating in the room until they actually join the
         room.
@@ -82,6 +87,9 @@ paths:
 
         Only users currently in a particular room can invite other users to
         join that room.
+
+        If the user was invited to the room, the home server will append a
+        ``m.room.member`` event to the room.
       security:
         - accessToken: []
       parameters:
@@ -92,7 +100,7 @@ paths:
           required: true
           x-example: "!d41d8cd:matrix.org"
         - in: body
-          name: user_id
+          name: body
           required: true
           schema:
             type: object
@@ -112,7 +120,107 @@ paths:
             application/json: |-
               {}
           schema:
-            type: object  # empty json object
+            type: object
+        403:
+          description: |-
+            You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
+              - The invitee has been banned from the room.
+              - The invitee is already a member of the room.
+              - The inviter is not currently in the room.
+              - The inviter's power level is insufficient to invite users to the room.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN", "error": "@cheeky_monkey:matrix.org is banned from the room"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+
+  "/rooms/{roomId}/invite":
+    post:
+      summary: Invite a user to participate in a particular room.
+      description: |-
+        *Note that there are two forms of this API, which are documented separately.
+        This version of the API does not require that the inviter know the Matrix
+        identifier of the invitee, and instead relies on third party identifiers.*
+
+        This API invites a user to participate in a particular room.
+        They do not start participating in the room until they actually join the
+        room.
+
+        This serves two purposes; firstly, to notify the user that the room
+        exists (and that their presence is requested). Secondly, some rooms can
+        only be joined if a user is invited to join it; sending the invite gives
+        that user permission to join the room.
+
+        Only users currently in a particular room can invite other users to
+        join that room.
+
+        If the identity server does not know a Matrix user identifier for the
+        passed third party identifier, the homeserver will issue an invitation
+        which can be accepted upon providing proof of ownership of the third
+        party identifier. This is achieved by requesting a nonce and digest from
+        the identity server. When a user binds the invited third party
+        identifier to a Matrix user ID, the identity server will give the user a
+        list of pending invitations, each containing:
+
+          - The room ID to which they were invited
+
+          - The digest given to the homeserver
+
+          - A secret which, when appended to the nonce, digests to the above digest
+
+        The digest algorithm to be used is SHA256.
+
+        If the identity server did know the Matrix user identifier for the
+        third party identifier, the home server will append a ``m.room.member``
+        event to the room.
+
+        If a digest and nonce are requested from the identity server, the home
+        server will append a ``m.room.token_based_invite`` event to the room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier (not alias) to which to invite the user.
+          required: true
+          x-example: "!d41d8cd:matrix.org"
+        - in: body
+          name: body
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "identity_server": "matrix.org",
+                "medium": "email",
+                "address": "cheeky@monkey.com",
+                "display_name": "A very cheeky monkey"
+              }
+            properties:
+              identity_server:
+                type: string
+                description: The hostname+port of the identity server which should be used for third party identifier lookups.
+              medium:
+                type: string
+                description: The kind of address being passed in the address field.
+              address:
+                type: string
+                description: The invitee's third party identifier.
+              display_name:
+                type: string
+                description: A user-friendly string describing who has been invited. It should not contain the address of the invitee, to avoid leaking mappings between third party identities and matrix user IDs.
+            required: ["identity_server", "medium", "address", "display_name"]
+      responses:
+        200:
+          description: The user has been invited to join the room.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
         403:
           description: |-
             You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index d8cc213d..150b0942 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -110,7 +110,7 @@ class MatrixUnits(Units):
                     "title": single_api.get("summary", ""),
                     "desc": single_api.get("description", single_api.get("summary", "")),
                     "method": method.upper(),
-                    "path": full_path,
+                    "path": full_path.strip(),
                     "requires_auth": "security" in single_api,
                     "rate_limited": 429 in single_api.get("responses", {}),
                     "req_params": [],

From 708f28127057d26e7000f24752dd1a111e1c1d75 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 17:09:44 +0100
Subject: [PATCH 063/467] Add comments about auth

---
 scripts/speculator/main.go | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 59047e68..22fb2ff2 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -123,6 +123,8 @@ func serveSpec(w http.ResponseWriter, req *http.Request) {
 		return
 	}
 
+	// We're going to run whatever Python is specified in the pull request, which
+	// may do bad things, so only trust people we trust.
 	if !allowedMembers[pr.User.Login] {
 		w.WriteHeader(403)
 		io.WriteString(w, fmt.Sprintf("%q is not a trusted pull requester", pr.User.Login))
@@ -158,6 +160,8 @@ func serveRstDiff(w http.ResponseWriter, req *http.Request) {
 		return
 	}
 
+	// We're going to run whatever Python is specified in the pull request, which
+	// may do bad things, so only trust people we trust.
 	if !allowedMembers[pr.User.Login] {
 		w.WriteHeader(403)
 		io.WriteString(w, fmt.Sprintf("%q is not a trusted pull requester", pr.User.Login))

From ca7dd49dc600a8d2f3cc0dbccbfd7f81e15185a2 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 10 Sep 2015 17:14:53 +0100
Subject: [PATCH 064/467] speculator: Add /healthz endpoint

---
 scripts/speculator/main.go | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 22fb2ff2..3775ad86 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -220,5 +220,12 @@ func main() {
 	}
 	http.HandleFunc("/spec/", serveSpec)
 	http.HandleFunc("/diff/rst/", serveRstDiff)
+	http.HandleFunc("/healthz", serveText("ok"))
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 }
+
+func serveText(s string) func(http.ResponseWriter, *http.Request) {
+	return func(w http.ResponseWriter, req *http.Request) {
+		io.WriteString(w, s)
+	}
+}

From bf318ffcd53e4b01ea20c4a3bbe566a653360815 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 11 Sep 2015 12:42:41 +0100
Subject: [PATCH 065/467] Add HTTP endpoints to README

---
 scripts/speculator/README | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/speculator/README b/scripts/speculator/README
index 1da99cfb..ae00c55a 100644
--- a/scripts/speculator/README
+++ b/scripts/speculator/README
@@ -1,4 +1,8 @@
 speculator allows you to preview pull requests to the matrix.org specification.
 
+It serves two HTTP endpoints:
+ - /spec/123 which renders the spec as html at pull request 123.
+ - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
+
 To run it, you must install the `go` tool, and run:
  `go run main.go`

From ba780a589a9311dce856342d978ad3f45d10f392 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 11 Sep 2015 14:40:08 +0100
Subject: [PATCH 066/467] speculator: Add list-pull-request functionality

---
 scripts/speculator/main.go | 43 +++++++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 3775ad86..757c48e7 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -24,9 +24,12 @@ import (
 )
 
 type PullRequest struct {
-	Base Commit
-	Head Commit
-	User User
+	Number  int
+	Base    Commit
+	Head    Commit
+	Title   string
+	User    User
+	HTMLURL string `json:"html_url"`
 }
 
 type Commit struct {
@@ -39,7 +42,8 @@ type RequestRepo struct {
 }
 
 type User struct {
-	Login string
+	Login   string
+	HTMLURL string `json:"html_url"`
 }
 
 var (
@@ -47,6 +51,8 @@ var (
 	allowedMembers map[string]bool
 )
 
+const pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls"
+
 func gitClone(url string) (string, error) {
 	dst := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10))
 	cmd := exec.Command("git", "clone", url, dst)
@@ -68,7 +74,7 @@ func gitCheckout(path, sha string) error {
 }
 
 func lookupPullRequest(prNumber string) (PullRequest, error) {
-	resp, _ := http.Get("https://api.github.com/repos/matrix-org/matrix-doc/pulls/" + prNumber)
+	resp, _ := http.Get(fmt.Sprintf("%s/%s", pullsPrefix, prNumber))
 	defer resp.Body.Close()
 	dec := json.NewDecoder(resp.Body)
 	var pr PullRequest
@@ -192,6 +198,32 @@ func serveRstDiff(w http.ResponseWriter, req *http.Request) {
 	w.Write(diff.Bytes())
 }
 
+func listPulls(w http.ResponseWriter, req *http.Request) {
+	resp, err := http.Get(pullsPrefix)
+	if err != nil {
+		writeError(w, err)
+		return
+	}
+	defer resp.Body.Close()
+	dec := json.NewDecoder(resp.Body)
+	var pulls []PullRequest
+	if err := dec.Decode(&pulls); err != nil {
+		writeError(w, err)
+		return
+	}
+	if len(pulls) == 0 {
+		io.WriteString(w, "No pull requests found")
+		return
+	}
+	s := "<body><ul>"
+	for _, pull := range pulls {
+		s += fmt.Sprintf(`<li>%d: <a href="%s">%s</a>: <a href="%s">%s</a>: <a href="spec/%d">spec</a> <a href="diff/rst/%d">rst diff</a></li>`,
+			pull.Number, pull.User.HTMLURL, pull.User.Login, pull.HTMLURL, pull.Title, pull.Number, pull.Number)
+	}
+	s += "</ul></body>"
+	io.WriteString(w, s)
+}
+
 func ignoreExitCodeOne(err error) error {
 	if err == nil {
 		return err
@@ -221,6 +253,7 @@ func main() {
 	http.HandleFunc("/spec/", serveSpec)
 	http.HandleFunc("/diff/rst/", serveRstDiff)
 	http.HandleFunc("/healthz", serveText("ok"))
+	http.HandleFunc("/", listPulls)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 }
 

From 20f0284155ed5d1a5cfe8146071f395559e93866 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 11 Sep 2015 14:40:52 +0100
Subject: [PATCH 067/467] speculator: Unified diff for rst diff

---
 scripts/speculator/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 3775ad86..d30c199c 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -182,7 +182,7 @@ func serveRstDiff(w http.ResponseWriter, req *http.Request) {
 		return
 	}
 
-	diffCmd := exec.Command("diff", path.Join(base, "scripts", "tmp", "full_spec.rst"), path.Join(head, "scripts", "tmp", "full_spec.rst"))
+	diffCmd := exec.Command("diff", "-u", path.Join(base, "scripts", "tmp", "full_spec.rst"), path.Join(head, "scripts", "tmp", "full_spec.rst"))
 	var diff bytes.Buffer
 	diffCmd.Stdout = &diff
 	if err := ignoreExitCodeOne(diffCmd.Run()); err != nil {

From b6dab0009d5533161dff92b1bdad3c664d3a5c6f Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 11 Sep 2015 14:43:26 +0100
Subject: [PATCH 068/467] speculator: Don't ignore errors

---
 scripts/speculator/main.go | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 3775ad86..1a474277 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -67,13 +67,18 @@ func gitCheckout(path, sha string) error {
 	return nil
 }
 
-func lookupPullRequest(prNumber string) (PullRequest, error) {
-	resp, _ := http.Get("https://api.github.com/repos/matrix-org/matrix-doc/pulls/" + prNumber)
+func lookupPullRequest(prNumber string) (*PullRequest, error) {
+	resp, err := http.Get("https://api.github.com/repos/matrix-org/matrix-doc/pulls/" + prNumber)
 	defer resp.Body.Close()
+	if err != nil {
+		return nil, fmt.Errorf("error getting pulls: %v", err)
+	}
 	dec := json.NewDecoder(resp.Body)
 	var pr PullRequest
-	_ = dec.Decode(&pr)
-	return pr, nil
+	if err := dec.Decode(&pr); err != nil {
+		return nil, fmt.Errorf("error decoding pulls: %v", err)
+	}
+	return &pr, nil
 }
 
 func generate(dir string) error {

From d251f791fbfbe7af675f82689d51b63a79a361ff Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 11 Sep 2015 17:15:03 +0100
Subject: [PATCH 069/467] speculator: Document /

---
 scripts/speculator/README  | 3 ++-
 scripts/speculator/main.go | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/speculator/README b/scripts/speculator/README
index ae00c55a..6ab84f68 100644
--- a/scripts/speculator/README
+++ b/scripts/speculator/README
@@ -1,6 +1,7 @@
 speculator allows you to preview pull requests to the matrix.org specification.
 
-It serves two HTTP endpoints:
+It serves the following HTTP endpoints:
+ - / lists open pull requests
  - /spec/123 which renders the spec as html at pull request 123.
  - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
 
diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index bcbe5514..32c015ac 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -1,5 +1,6 @@
 // speculator allows you to preview pull requests to the matrix.org specification.
-// It serves two HTTP endpoints:
+// It serves the following HTTP endpoints:
+//  - / lists open pull requests
 //  - /spec/123 which renders the spec as html at pull request 123.
 //  - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
 // It is currently woefully inefficient, and there is a lot of low hanging fruit for improvement.

From b2477614658d6b4267ffcb1f35cfc9adb7f388ad Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 10:33:25 +0100
Subject: [PATCH 070/467] .gitignore speculator

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index e2250131..74b1c7d2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
 scripts/gen
 scripts/continuserv/continuserv
+scripts/speculator/speculator
 templating/out
 *.pyc
 supporting-docs/_site

From 0341c30824ed89b64cb09b5eea74f011eed81495 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 10:39:22 +0100
Subject: [PATCH 071/467] Minor typo and formatting fixes

---
 drafts/macaroons_caveats.rst           | 2 +-
 specification/10_client_server_api.rst | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index b6920d6c..2a45fe12 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -1,7 +1,7 @@
 Macaroon Caveats
 ================
 
-Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted, by adding caveats to them.
+Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
 
 Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
 
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index 6a589f8e..440f98e4 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1035,7 +1035,7 @@ user_id
   The fully-qualified Matrix ID that has been registered.
 access_token
   An access token for the account. This token may expire at some point, and if
-  so, it MAY come with a refersh_token, described below.
+  so, it MAY come with a refresh_token, described below.
 refresh_token (optional)
   A refresh token may be exchanged for a new access_token as described in
   `Refreshing access tokens`.
@@ -1064,7 +1064,7 @@ refresh_token (optional)
 There is no specific error message to indicate that a request has failed because
 an access token has expired; instead, if a client has reason to believe its
 access token is valid, and it receives an auth error, they should attempt to
-refresh for a new token on failure, and re-try the request with the new token.
+refresh for a new token on failure, and retry the request with the new token.
 
 Changing Password
 ~~~~~~~~~~~~~~~~~

From d9013cab5fb51d341f96e325b15e207859e4c313 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dan@illicitonion.com>
Date: Mon, 14 Sep 2015 11:03:54 +0100
Subject: [PATCH 072/467] speculator: Add HTML diffing

I started fiddling with re-implementing the perl script in Go to add
some new functionality (and avoid the Perl), but it's not yet usable
---
 scripts/speculator/README      |   1 +
 scripts/speculator/htmldiff.pl | 564 +++++++++++++++++++++++++++++++++
 scripts/speculator/main.go     | 130 ++++++--
 3 files changed, 661 insertions(+), 34 deletions(-)
 create mode 100755 scripts/speculator/htmldiff.pl

diff --git a/scripts/speculator/README b/scripts/speculator/README
index 6ab84f68..0a9f53fd 100644
--- a/scripts/speculator/README
+++ b/scripts/speculator/README
@@ -4,6 +4,7 @@ It serves the following HTTP endpoints:
  - / lists open pull requests
  - /spec/123 which renders the spec as html at pull request 123.
  - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
+ - /diff/html/123 which gives a diff of the spec's HTML at pull request 123.
 
 To run it, you must install the `go` tool, and run:
  `go run main.go`
diff --git a/scripts/speculator/htmldiff.pl b/scripts/speculator/htmldiff.pl
new file mode 100755
index 00000000..b1689067
--- /dev/null
+++ b/scripts/speculator/htmldiff.pl
@@ -0,0 +1,564 @@
+#!/usr/bin/perl
+#
+# htmldiff - present a diff marked version of two html documents
+#
+# Copyright (c) 1998-2006 MACS, Inc.
+#
+# Copyright (c) 2007 SiSco, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# See http://www.themacs.com for more information.
+#
+# usage: htmldiff [[-c] [-l] [-o] oldversion newversion [output]]
+#
+# -c - disable metahtml comment processing
+# -o - disable outputting of old text
+# -l - use navindex to create sequence of diffs
+# oldversion - the previous version of the document
+# newversion - the newer version of the document
+# output - a filename to place the output in. If omitted, the output goes to
+#          standard output.
+#
+# if invoked with no options or arguments, operates as a CGI script. It then
+# takes the following parameters:
+#
+# oldfile - the URL of the original file
+# newfile - the URL of the new file
+# mhtml - a flag to indicate whether it should be aware of MetaHTML comments.
+#
+# requires GNU diff utility
+# also requires the perl modules Getopt::Std
+#
+# NOTE: The markup created by htmldiff may not validate against the HTML 4.0
+# DTD. This is because the algorithm is realtively simple, and there are
+# places in the markup content model where the span element is not allowed.
+# Htmldiff is NOT aware of these places.
+#
+# $Source: /u/sources/public/2009/htmldiff/htmldiff.pl,v $
+# $Revision: 1.1 $
+#
+# $Log: htmldiff.pl,v $
+# Revision 1.1  2014/01/06 08:04:51  dom
+# added copy of htmldiff perl script since aptest.com repo no longer available
+#
+# Revision 1.5  2008/03/05 13:23:16  ahby
+# Fixed a problem with leading whitespace before markup.
+#
+# Revision 1.4  2007/12/13 13:09:16  ahby
+# Updated copyright and license.
+#
+# Revision 1.3  2007/12/13 12:53:34  ahby
+# Changed use of span to ins and del
+#
+# Revision 1.2  2002/02/13 16:27:23  ahby
+# Changed processing model.
+# Improved handling of old text and changed styles.
+#
+# Revision 1.1  2000/07/12 12:20:04  ahby
+# Updated to remove empty spans - this fixes validation problems under
+# strict.
+#
+# Revision 1.11  1999/12/08 19:46:45  ahby
+# Fixed validation errors introduced by placing markup where it didn't
+# belong.
+#
+# Revision 1.10  1999/10/18 13:42:58  ahby
+# Added -o to the usage message.
+#
+# Revision 1.9  1999/05/04 12:29:11  ahby
+# Added an option to turn off the display of old text.
+#
+# Revision 1.8  1999/04/09 14:37:27  ahby
+# Fixed a perl syntax error.
+#
+# Revision 1.7  1999/04/09 14:35:49  ahby
+# Added reference to MACS homepage.
+#
+# Revision 1.6  1999/04/09 14:35:09  ahby
+# Added comment about validity of generated markup.
+#
+# Revision 1.5  1999/02/22 22:17:54  ahby
+# Changed to use stylesheets.
+# Changed to rely upon span.
+# Changed to work around content model problems.
+#
+# Revision 1.4  1999/02/08 02:32:22  ahby
+# Added a copyright statement.
+#
+# Revision 1.3  1999/02/08 02:30:40  ahby
+# Added header processing.
+#
+# Revision 1.2  1998/12/10 17:31:31  ahby
+# Fixed to escape less-thans in change blocks and to not permit change
+# markup within specific elements (like TITLE).
+#
+# Revision 1.1  1998/11/26 00:09:22  ahby
+# Initial revision
+#
+#
+
+use Getopt::Std;
+
+sub usage {
+   print STDERR "htmldiff [-c] [-o] oldversion newversion [output]\n";
+   exit;
+}
+
+sub url_encode {
+    my $str = shift;
+    $str =~ s/([\x00-\x1f\x7F-\xFF])/
+                 sprintf ('%%%02x', ord ($1))/eg;
+    return $str;
+}
+
+# markit - diff-mark the streams
+#
+# markit(file1, file2)
+#
+# markit relies upon GNUdiff to mark up the text.
+#
+# The markup is encoded using special control sequences:
+#
+#   a block wrapped in control-a is deleted text
+#   a block wrapped in control-b is old text
+#   a block wrapped in control-c is new text
+#
+# The main processing loop attempts to wrap the text blocks in appropriate
+# SPANs based upon the type of text that it is.
+#
+# When the loop encounters a < in the text, it stops the span. Then it outputs
+# the element that is defined, then it restarts the span.
+
+sub markit {
+   my $retval = "";
+   my($file1) = shift;
+   my($file2) = shift;
+#  my $old="<span class=\\\"diff-old-a\\\">deleted text: </span>%c'\012'%c'\001'%c'\012'%<%c'\012'%c'\001'%c'\012'";
+   my $old="%c'\012'%c'\001'%c'\012'%<%c'\012'%c'\001'%c'\012'";
+   my $new="%c'\012'%c'\003'%c'\012'%>%c'\012'%c'\003'%c'\012'";
+   my $unchanged="%=";
+   my $changed="%c'\012'%c'\001'%c'\012'%<%c'\012'%c'\001'%c'\012'%c'\004'%c'\012'%>%c'\012'%c'\004'%c'\012'";
+   if ($opt_o) {
+      $old = "";
+      $changed = "%c'\012'%c'\004'%c'\012'%>%c'\012'%c'\004'%c'\012'";
+   }
+#  my $old="%c'\002'<font color=\\\"purple\\\" size=\\\"-2\\\">deleted text:</font><s>%c'\012'%c'\001'%c'\012'%<%c'\012'%c'\001'%c'\012'</s>%c'\012'%c'\002'";
+#  my $new="%c'\002'<font color=\\\"purple\\\"><u>%c'\012'%c'\002'%>%c'\002'</u></font>%c'\002'%c'\012'";
+#  my $unchanged="%=";
+#  my $changed="%c'\002'<s>%c'\012'%c'\001'%c'\012'%<%c'\012'%c'\001'%c'\012'</s><font color=\\\"purple\\\"><u>%c'\002'%c'\012'%>%c'\012'%c'\002'</u></font>%c'\002'%c'\012'";
+
+   my @span;
+   $span[0]="</span>";
+   $span[1]="<del class=\"diff-old\">";
+   $span[2]="<del class=\"diff-old\">";
+   $span[3]="<ins class=\"diff-new\">";
+   $span[4]="<ins class=\"diff-chg\">";
+
+   my @diffEnd ;
+   $diffEnd[1] = '</del>';
+   $diffEnd[2] = '</del>';
+   $diffEnd[3] = '</ins>';
+   $diffEnd[4] = '</ins>';
+
+   my $diffcounter = 0;
+
+   open(FILE, qq(diff -d --old-group-format="$old" --new-group-format="$new" --changed-group-format="$changed" --unchanged-group-format="$unchanged" $file1 $file2 |)) || die("Diff failed: $!");
+#  system (qq(diff --old-group-format="$old" --new-group-format="$new" --changed-group-format="$changed" --unchanged-group-format="$unchanged" $file1 $file2 > /tmp/output));
+
+   my $state = 0;
+   my $inblock = 0;
+   my $temp = "";
+   my $lineCount = 0;
+
+# strategy:
+#
+# process the output of diff...
+#
+# a link with control A-D means the start/end of the corresponding ordinal
+# state (1-4). Resting state is state 0.
+#
+# While in a state, accumulate the contents for that state. When exiting the
+# state, determine if it is appropriate to emit the contents with markup or
+# not (basically, if the accumulated buffer contains only empty lines or lines
+# with markup, then we don't want to emit the wrappers.  We don't need them.
+#
+# Note that if there is markup in the "old" block, that markup is silently
+# removed.  It isn't really that interesting, and it messes up the output
+# something fierce.
+
+   while (<FILE>) {
+      my $anchor = $opt_l ? qq[<a tabindex="$diffcounter">] : "" ;
+      my $anchorEnd = $opt_l ? q[</a>] : "" ;
+      $lineCount ++;
+      if ($state == 0) {   # if we are resting and we find a marker,
+                     # then we must be entering a block
+         if (m/^([\001-\004])/) {
+            $state = ord($1);
+            $_ = "";
+         }
+#        if (m/^\001/) {
+#           $state = 1;
+#           s/^/$span[1]/;
+#        } elsif (m/^\002/) {
+#           $state = 2;
+#           s/^/$span[2]/;
+#        } elsif (m/^\003/) {
+#           $state = 3;
+#           s/^/$span[3]/;
+#        } elsif (m/^\004/) {
+#           $state = 4;
+#           s/^/$span[4]/;
+#        }
+      } else {
+         # if we are in "old" state, remove markup
+         if (($state == 1) || ($state == 2)) {
+            s/\<.*\>//; # get rid of any old markup
+            s/\</&lt;/g; # escape any remaining STAG or ETAGs
+            s/\>/&gt;/g;
+         }
+         # if we found another marker, we must be exiting the state
+         if (m/^([\001-\004])/) {
+            if ($temp ne "") {
+               $_ = $span[$state] . $anchor . $temp . $anchorEnd . $diffEnd[$state] . "\n";
+               $temp = "";
+            } else {
+               $_ = "" ;
+            }
+            $state = 0;
+         } elsif (m/^\s*\</) { # otherwise, is this line markup?
+            # if it is markup AND we haven't seen anything else yet,
+            # then we will emit the markup
+            if ($temp eq "") {
+               $retval .= $_;
+               $_ = "";
+            } else { # we wrap it with the state switches and hold it
+               s/^/$anchorEnd$diffEnd[$state]/;
+               s/$/$span[$state]$anchor/;
+               $temp .= $_;
+               $_ = "";
+            }
+         } else {
+            if (m/.+/) {
+               $temp .= $_;
+               $_ = "";
+            }
+         }
+      }
+
+      s/\001//g;
+      s/\002//g;
+      s/\003//g;
+      s/\004//g;
+      if ($_ !~ m/^$/) {
+         $retval .= $_;
+      }
+      $diffcounter++;
+   }
+   close FILE;
+   $retval =~ s/$span[1]\n+$diffEnd[1]//g;
+   $retval =~ s/$span[2]\n+$diffEnd[2]//g;
+   $retval =~ s/$span[3]\n+$diffEnd[3]//g;
+   $retval =~ s/$span[4]\n+$diffEnd[4]//g;
+   $retval =~ s/$span[1]\n*$//g;
+   $retval =~ s/$span[2]\n*$//g;
+   $retval =~ s/$span[3]\n*$//g;
+   $retval =~ s/$span[4]\n*$//g;
+   return $retval;
+}
+
+sub splitit {
+   my $filename = shift;
+   my $headertmp = shift;
+   my $inheader=0;
+   my $preformatted=0;
+   my $inelement=0;
+   my $retval = "";
+   my $styles = q(<style type='text/css'>
+.diff-old-a {
+  font-size: smaller;
+  color: red;
+}
+
+.diff-new { background-color: yellow; }
+.diff-chg { background-color: lime; }
+.diff-new:before,
+.diff-new:after
+    { content: "\2191" }
+.diff-chg:before, .diff-chg:after
+    { content: "\2195" }
+.diff-old { text-decoration: line-through; background-color: #FBB; }
+.diff-old:before,
+.diff-old:after
+    { content: "\2193" }
+:focus { border: thin red solid}
+</style>
+);
+   if ($opt_t) {
+      $styles .= q(
+<script type="text/javascript">
+<!--
+function setOldDisplay() {
+   for ( var s = 0; s < document.styleSheets.length; s++ ) {
+      var css = document.styleSheets[s];
+      var mydata ;
+      try { mydata = css.cssRules ;
+      if ( ! mydata ) mydata = css.rules;
+      for ( var r = 0; r < mydata.length; r++ ) {
+         if ( mydata[r].selectorText == '.diff-old' ) {
+            mydata[r].style.display = ( mydata[r].style.display == '' ) ? 'none'
+: '';
+            return;
+         }
+      }
+      } catch(e) {} ;
+   }
+}
+-->
+</script>
+);
+
+   }
+
+   if ($stripheader) {
+      open(HEADER, ">$headertmp");
+   }
+
+   my $incomment = 0;
+   my $inhead = 1;
+   open(FILE, $filename) || die("File $filename cannot be opened: $!");
+   while (<FILE>) {
+      if ($inhead == 1) {
+         if (m/\<\/head/i) {
+            print HEADER $styles;
+         }
+         if (m/\<body/i) {
+            $inhead = 0;
+            print HEADER;
+            if ($opt_t) {
+               print HEADER q(
+<form action=""><input type="button" onclick="setOldDisplay()" value="Show/Hide Old Content" /></form>
+);
+            }
+            close HEADER;
+         } else {
+            print HEADER;
+         }
+      } else {
+         if ($incomment) {
+            if (m;-->;) {
+               $incomment = 0;
+               s/.*-->//;
+            } else {
+               next;
+            }
+         }
+         if (m;<!--;) {
+            while (m;<!--.*-->;) {
+               s/<!--.*?-->//;
+            }
+            if (m;<!--; ) {
+               $incomment = 1;
+               s/<!--.*//;
+            }
+         }
+         if (m/\<pre/i) {
+            $preformatted = 1;
+         }
+         if (m/\<\/pre\>/i) {
+            $preformatted = 0;
+         }
+         if ($preformatted) {
+            $retval .= $_;
+         } elsif ($mhtmlcomments && /^;;;/) {
+            $retval .= $_;
+         } else {
+            my @list = split(' ');
+            foreach $element (@list) {
+               if ($element =~ m/\<H[1-6]/i) {
+#                 $inheader = 1;
+               }
+               if ($inheader == 0) {
+                  $element =~ s/</\n</g;
+                  $element =~ s/^\n//;
+                  $element =~ s/>/>\n/g;
+                  $element =~ s/\n$//;
+                  $element =~ s/>\n([.,:!]+)/>$1/g;
+               }
+               if ($element =~ m/\<\/H[1-6]\>/i) {
+                  $inheader = 0;
+               }
+               $retval .= "$element";
+               $inelement += ($element =~ s/</&lt;/g);
+               $inelement -= ($element =~ s/>/&gt;/g);
+               if ($inelement < 0) {
+                  $inelement = 0;
+               }
+               if (($inelement == 0) && ($inheader == 0)) {
+                  $retval .= "\n";
+               } else {
+                  $retval .= " ";
+               }
+            }
+         undef @list;
+         }
+      }
+   }
+   $retval .= "\n";
+   close FILE;
+   return $retval;
+}
+
+$mhtmlcomments = 1;
+
+sub cli {
+   getopts("clto") || usage();
+
+   if ($opt_c) {$mhtmlcomments = 0;}
+
+   if (@ARGV < 2) { usage(); }
+
+   $file1 = $ARGV[0];
+   $file2 = $ARGV[1];
+   $file3 = $ARGV[2];
+
+   $tmp = splitit($file1, $headertmp1);
+   open (FILE, ">$tmp1");
+   print FILE $tmp;
+   close FILE;
+
+   $tmp = splitit($file2, $headertmp2);
+   open (FILE, ">$tmp2");
+   print FILE $tmp;
+   close FILE;
+
+   $output = "";
+
+   if ($stripheader) {
+      open(FILE, $headertmp2);
+      while (<FILE>) {
+         $output .= $_;
+      }
+      close(FILE);
+   }
+
+   $output .= markit($tmp1, $tmp2);
+
+   if ($file3) {
+      open(FILE, ">$file3");
+      print FILE $output;
+      close FILE;
+   } else {
+      print $output;
+   }
+}
+
+sub cgi {
+#  use LWP::UserAgent;
+#  use CGI;
+
+   my $query = new CGI;
+   my $url1 = $query->param("oldfile");
+   my $url2 = $query->param("newfile");
+   my $mhtml = $query->param("mhtml");
+
+   my $file1 = "/tmp/htdcgi1.$$";
+   my $file2 = "/tmp/htdcgi2.$$";
+
+   my $ua = new LWP::UserAgent;
+   $ua->agent("MACS, Inc. HTMLdiff/0.9 " . $ua->agent);
+
+   # Create a request
+
+   my $req1 = new HTTP::Request GET => $url1;
+
+   my $res1 = $ua->request($req1, $file1);
+   if ($res1->is_error) {
+      print $res1->error_as_HTML();
+      print "<p>The URL $url1 could not be found.  Please check it and try again.</p>";
+      return;
+   }
+
+   my $req2 = new HTTP::Request GET => $url2;
+
+   my $res2 = $ua->request($req2, $file2);
+   if ($res2->is_error) {
+      print $res2->error_as_HTML();
+      print "<p>The URL $url2 could not be found.  Please check it and try again.</p>";
+      return;
+   }
+
+   $split1 = splitit($file1, $headertmp1);
+   open (FILE, ">$tmp1");
+   print FILE $split1;
+   close FILE;
+
+   $split2 = splitit($file2, $headertmp2);
+   open (FILE, ">$tmp2");
+   print FILE $split2;
+   close FILE;
+
+   $output = "";
+
+   if ($stripheader) {
+      open(FILE, $headertmp2);
+      while (<FILE>) {
+         $output .= $_;
+      }
+      close(FILE);
+   }
+
+   $output .= markit($tmp1, $tmp2);
+
+   my $base=$res2->base;
+
+   if ($base !~ /\/$/) {
+      $base =~ s/[^\/]*$//;
+   }
+
+   if ( $output !~ /<base/i ) {
+      $output =~ s/<head>/<head>\n<base href="$base">/i ||
+      $output =~ s/<html>/<html>\n<base href="$base">/i ;
+   }
+
+   print $query->header(-type=>'text/html',-nph=>1);
+   print $output;
+
+   unlink $file1;
+   unlink $file2;
+
+}
+
+$tmp1="/tmp/htdtmp1.$$";
+$headertmp1="/tmp/htdhtmp1.$$";
+$tmp2="/tmp/htdtmp2.$$";
+$headertmp2="/tmp/htdhtmp2.$$";
+$stripheader = 1;
+
+if (@ARGV == 0) {
+   cgi();      # if no arguments, we must be operating as a cgi script
+} else {
+   cli();      # if there are arguments, then we are operating as a CLI
+}
+
+unlink $tmp1;
+unlink $headertmp1;
+unlink $tmp2;
+unlink $headertmp2;
diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 32c015ac..ab88c84e 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -3,6 +3,7 @@
 //  - / lists open pull requests
 //  - /spec/123 which renders the spec as html at pull request 123.
 //  - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
+//  - /diff/html/123 which gives a diff of the spec's HTML at pull request 123.
 // It is currently woefully inefficient, and there is a lot of low hanging fruit for improvement.
 package main
 
@@ -16,6 +17,7 @@ import (
 	"log"
 	"math/rand"
 	"net/http"
+	"net/url"
 	"os"
 	"os/exec"
 	"path"
@@ -52,6 +54,10 @@ var (
 	allowedMembers map[string]bool
 )
 
+func (u *User) IsTrusted() bool {
+	return allowedMembers[u.Login]
+}
+
 const pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls"
 
 func gitClone(url string) (string, error) {
@@ -74,7 +80,15 @@ func gitCheckout(path, sha string) error {
 	return nil
 }
 
-func lookupPullRequest(prNumber string) (*PullRequest, error) {
+func lookupPullRequest(url url.URL, pathPrefix string) (*PullRequest, error) {
+	if !strings.HasPrefix(url.Path, pathPrefix+"/") {
+		return nil, fmt.Errorf("invalid path passed: %s expect %s/123", url.Path, pathPrefix)
+	}
+	prNumber := url.Path[len(pathPrefix)+1:]
+	if strings.Contains(prNumber, "/") {
+		return nil, fmt.Errorf("invalid path passed: %s expect %s/123", url.Path, pathPrefix)
+	}
+
 	resp, err := http.Get(fmt.Sprintf("%s/%s", pullsPrefix, prNumber))
 	defer resp.Body.Close()
 	if err != nil {
@@ -100,8 +114,8 @@ func generate(dir string) error {
 	return nil
 }
 
-func writeError(w http.ResponseWriter, err error) {
-	w.WriteHeader(500)
+func writeError(w http.ResponseWriter, code int, err error) {
+	w.WriteHeader(code)
 	io.WriteString(w, fmt.Sprintf("%v\n", err))
 }
 
@@ -122,75 +136,66 @@ func generateAt(repo, sha string) (dst string, err error) {
 }
 
 func serveSpec(w http.ResponseWriter, req *http.Request) {
-	parts := strings.Split(req.URL.Path, "/")
-	if len(parts) != 3 {
-		w.WriteHeader(400)
-		io.WriteString(w, fmt.Sprintf("Invalid path passed: %v expect /pull/123", req.URL.Path))
-		return
-	}
-
-	pr, err := lookupPullRequest(parts[2])
+	pr, err := lookupPullRequest(*req.URL, "/spec")
 	if err != nil {
-		writeError(w, err)
+		writeError(w, 400, err)
 		return
 	}
 
 	// We're going to run whatever Python is specified in the pull request, which
 	// may do bad things, so only trust people we trust.
-	if !allowedMembers[pr.User.Login] {
-		w.WriteHeader(403)
-		io.WriteString(w, fmt.Sprintf("%q is not a trusted pull requester", pr.User.Login))
+	if err := checkAuth(pr); err != nil {
+		writeError(w, 403, err)
 		return
 	}
 
 	dst, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
 	defer os.RemoveAll(dst)
 	if err != nil {
-		writeError(w, err)
+		writeError(w, 500, err)
 		return
 	}
 
 	b, err := ioutil.ReadFile(path.Join(dst, "scripts/gen/specification.html"))
 	if err != nil {
-		writeError(w, fmt.Errorf("Error reading spec: %v", err))
+		writeError(w, 500, fmt.Errorf("Error reading spec: %v", err))
 		return
 	}
 	w.Write(b)
 }
 
-func serveRstDiff(w http.ResponseWriter, req *http.Request) {
-	parts := strings.Split(req.URL.Path, "/")
-	if len(parts) != 4 {
-		w.WriteHeader(400)
-		io.WriteString(w, fmt.Sprintf("Invalid path passed: %v expect /diff/rst/123", req.URL.Path))
-		return
+func checkAuth(pr *PullRequest) error {
+	if !pr.User.IsTrusted() {
+		return fmt.Errorf("%q is not a trusted pull requester", pr.User.Login)
 	}
+	return nil
+}
 
-	pr, err := lookupPullRequest(parts[3])
+func serveRSTDiff(w http.ResponseWriter, req *http.Request) {
+	pr, err := lookupPullRequest(*req.URL, "/diff/rst")
 	if err != nil {
-		writeError(w, err)
+		writeError(w, 400, err)
 		return
 	}
 
 	// We're going to run whatever Python is specified in the pull request, which
 	// may do bad things, so only trust people we trust.
-	if !allowedMembers[pr.User.Login] {
-		w.WriteHeader(403)
-		io.WriteString(w, fmt.Sprintf("%q is not a trusted pull requester", pr.User.Login))
+	if err := checkAuth(pr); err != nil {
+		writeError(w, 403, err)
 		return
 	}
 
 	base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA)
 	defer os.RemoveAll(base)
 	if err != nil {
-		writeError(w, err)
+		writeError(w, 500, err)
 		return
 	}
 
 	head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
 	defer os.RemoveAll(head)
 	if err != nil {
-		writeError(w, err)
+		writeError(w, 500, err)
 		return
 	}
 
@@ -198,23 +203,79 @@ func serveRstDiff(w http.ResponseWriter, req *http.Request) {
 	var diff bytes.Buffer
 	diffCmd.Stdout = &diff
 	if err := ignoreExitCodeOne(diffCmd.Run()); err != nil {
-		writeError(w, fmt.Errorf("error running diff: %v", err))
+		writeError(w, 500, fmt.Errorf("error running diff: %v", err))
 		return
 	}
 	w.Write(diff.Bytes())
 }
 
+func serveHTMLDiff(w http.ResponseWriter, req *http.Request) {
+	pr, err := lookupPullRequest(*req.URL, "/diff/html")
+	if err != nil {
+		writeError(w, 400, err)
+		return
+	}
+
+	// We're going to run whatever Python is specified in the pull request, which
+	// may do bad things, so only trust people we trust.
+	if err := checkAuth(pr); err != nil {
+		writeError(w, 403, err)
+		return
+	}
+
+	base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA)
+	defer os.RemoveAll(base)
+	if err != nil {
+		writeError(w, 500, err)
+		return
+	}
+
+	head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
+	defer os.RemoveAll(head)
+	if err != nil {
+		writeError(w, 500, err)
+		return
+	}
+
+	htmlDiffer, err := findHTMLDiffer()
+	if err != nil {
+		writeError(w, 500, fmt.Errorf("could not find HTML differ"))
+		return
+	}
+
+	cmd := exec.Command(htmlDiffer, path.Join(base, "scripts", "gen", "specification.html"), path.Join(head, "scripts", "gen", "specification.html"))
+	var b bytes.Buffer
+	cmd.Stdout = &b
+	if err := cmd.Run(); err != nil {
+		writeError(w, 500, fmt.Errorf("error running HTML differ: %v", err))
+		return
+	}
+	w.Write(b.Bytes())
+}
+
+func findHTMLDiffer() (string, error) {
+	wd, err := os.Getwd()
+	if err != nil {
+		return "", err
+	}
+	differ := path.Join(wd, "htmldiff.pl")
+	if _, err := os.Stat(differ); err != nil {
+		return differ, nil
+	}
+	return "", fmt.Errorf("unable to find htmldiff.pl")
+}
+
 func listPulls(w http.ResponseWriter, req *http.Request) {
 	resp, err := http.Get(pullsPrefix)
 	if err != nil {
-		writeError(w, err)
+		writeError(w, 500, err)
 		return
 	}
 	defer resp.Body.Close()
 	dec := json.NewDecoder(resp.Body)
 	var pulls []PullRequest
 	if err := dec.Decode(&pulls); err != nil {
-		writeError(w, err)
+		writeError(w, 500, err)
 		return
 	}
 	if len(pulls) == 0 {
@@ -257,7 +318,8 @@ func main() {
 		"NegativeMjark": true,
 	}
 	http.HandleFunc("/spec/", serveSpec)
-	http.HandleFunc("/diff/rst/", serveRstDiff)
+	http.HandleFunc("/diff/rst/", serveRSTDiff)
+	http.HandleFunc("/diff/html/", serveHTMLDiff)
 	http.HandleFunc("/healthz", serveText("ok"))
 	http.HandleFunc("/", listPulls)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))

From 521d625aace474375d73b30fb24d0cd09244b7c3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dan@illicitonion.com>
Date: Mon, 14 Sep 2015 11:06:57 +0100
Subject: [PATCH 073/467] Fix typo

---
 scripts/speculator/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index ab88c84e..8d7500a7 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -259,7 +259,7 @@ func findHTMLDiffer() (string, error) {
 		return "", err
 	}
 	differ := path.Join(wd, "htmldiff.pl")
-	if _, err := os.Stat(differ); err != nil {
+	if _, err := os.Stat(differ); err == nil {
 		return differ, nil
 	}
 	return "", fmt.Errorf("unable to find htmldiff.pl")

From c55917d99c68397729f6d910bf164120d998578d Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dan@illicitonion.com>
Date: Mon, 14 Sep 2015 11:08:16 +0100
Subject: [PATCH 074/467] Add spec diff to index page

---
 scripts/speculator/main.go | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 8d7500a7..b4fe2aef 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -284,8 +284,8 @@ func listPulls(w http.ResponseWriter, req *http.Request) {
 	}
 	s := "<body><ul>"
 	for _, pull := range pulls {
-		s += fmt.Sprintf(`<li>%d: <a href="%s">%s</a>: <a href="%s">%s</a>: <a href="spec/%d">spec</a> <a href="diff/rst/%d">rst diff</a></li>`,
-			pull.Number, pull.User.HTMLURL, pull.User.Login, pull.HTMLURL, pull.Title, pull.Number, pull.Number)
+		s += fmt.Sprintf(`<li>%d: <a href="%s">%s</a>: <a href="%s">%s</a>: <a href="spec/%d">spec</a> <a href="diff/html/%d">spec diff</a> <a href="diff/rst/%d">rst diff</a></li>`,
+			pull.Number, pull.User.HTMLURL, pull.User.Login, pull.HTMLURL, pull.Title, pull.Number, pull.Number, pull.Number)
 	}
 	s += "</ul></body>"
 	io.WriteString(w, s)

From b8e2ac5c1b9408a4dc3d72b4bc2935dd278cfeda Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 13:49:27 +0100
Subject: [PATCH 075/467] Swaggerify /login

---
 api/client-server/v1/registration.yaml  | 79 +++++++++++++++++++++++++
 specification/10_client_server_api.rst  | 27 +--------
 templating/matrix_templates/sections.py |  6 ++
 3 files changed, 86 insertions(+), 26 deletions(-)
 create mode 100644 api/client-server/v1/registration.yaml

diff --git a/api/client-server/v1/registration.yaml b/api/client-server/v1/registration.yaml
new file mode 100644
index 00000000..2df695be
--- /dev/null
+++ b/api/client-server/v1/registration.yaml
@@ -0,0 +1,79 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Registration and Login API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/login":
+    post:
+      summary: Authenticates the user.
+      description: |-
+        Authenticates the user by password, and issues an access token they can
+        use to authorize themself in subsequent requests.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: body
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "username": "cheeky_monkey",
+                "password": "ilovebananas"
+              }
+            properties:
+              username:
+                type: string
+                description: The fully qualified user ID or just local part of the user ID, to log in.
+              password:
+                type: string
+                description: The user's password.
+            required: ["username", "password"]
+      responses:
+        200:
+          description: The user has been authenticated.
+          examples:
+            application/json: |-
+              {
+                "user_id": "@cheeky_monkey:matrix.org",
+                "access_token": "abc123",
+                "home_server": "matrix.org"
+              }
+          schema:
+            type: object
+            properties:
+              user_id:
+                type: string
+                description: The fully-qualified Matrix ID that has been registered.
+              access_token:
+                type: string
+                description: An access token for the account. This access token can then be used to authorize other requests.
+              home_server:
+                type: string
+                description: The hostname of the Home Server on which the account has been registered.
+        403:
+          description: |-
+            The login attempt failed. For example, the password may have been incorrect.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index f0d6be42..b55118be 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1011,32 +1011,7 @@ was registered whilst the client was performing authentication.
 
 Old V1 API docs: |register|_
 
-Login
-~~~~~
-This section refers to API Version 1.
-
-API docs: |login|_
-
-Obtaining an access token for an existing user account is done using the
-request::
-
-  POST $PREFIX/login
-
-The body of the POST request is a JSON object containing:
-
-username
-  The full qualified or local part of the Matrix ID to log in with.
-password
-  The password for the account.
-
-On success, this returns a JSON object with keys:
-
-user_id
-  The fully-qualified Matrix ID that has been registered.
-access_token
-  An access token for the new account.
-home_server
-  The hostname of the Home Server on which the account has been registered.
+{{registration_http_api}}
 
 Changing Password
 ~~~~~~~~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 7e21434f..f28fe22d 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -96,6 +96,12 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
+    def render_registration_http_api(self):
+        return self._render_http_api_group(
+            "registration",
+            title_kind="~"
+        )
+
     def render_room_events(self):
         def filterFn(eventType):
             return (

From f483340033f86a878daf82a2501e3049b74a392f Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 15:16:30 +0100
Subject: [PATCH 076/467] Clarify digest computation

---
 api/client-server/v1/membership.yaml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 17b5795e..c4d658fb 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -168,7 +168,8 @@ paths:
 
           - The digest given to the homeserver
 
-          - A secret which, when appended to the nonce, digests to the above digest
+          - A secret which, when appended to the nonce, digests to the above digest,
+            i.e. digest = sha256(nonce + secret)
 
         The digest algorithm to be used is SHA256.
 

From 59a4e843612cd8b5cf954b1f1cac6bf9fdb732e8 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 17:10:46 +0100
Subject: [PATCH 077/467] s/registration/login/

---
 api/client-server/v1/{registration.yaml => login.yaml} | 0
 specification/10_client_server_api.rst                 | 2 +-
 templating/matrix_templates/sections.py                | 4 ++--
 3 files changed, 3 insertions(+), 3 deletions(-)
 rename api/client-server/v1/{registration.yaml => login.yaml} (100%)

diff --git a/api/client-server/v1/registration.yaml b/api/client-server/v1/login.yaml
similarity index 100%
rename from api/client-server/v1/registration.yaml
rename to api/client-server/v1/login.yaml
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index b55118be..fe11d199 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -1011,7 +1011,7 @@ was registered whilst the client was performing authentication.
 
 Old V1 API docs: |register|_
 
-{{registration_http_api}}
+{{login_http_api}}
 
 Changing Password
 ~~~~~~~~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index f28fe22d..729157bb 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -96,9 +96,9 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
-    def render_registration_http_api(self):
+    def render_login_http_api(self):
         return self._render_http_api_group(
-            "registration",
+            "login",
             title_kind="~"
         )
 

From 3b44fb92f972104bc781999303631d7ba5d649f2 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 14 Sep 2015 17:13:54 +0100
Subject: [PATCH 078/467] Clarify == case

---
 drafts/macaroons_caveats.rst | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 2a45fe12..791d217a 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -17,14 +17,16 @@ And these are joined by single space characters.
 
 Specified caveats:
 
-+-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
-| Caveat name | Description                                      | Legal Values                                                                               |
-+-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
-| gen         | Generation of the macaroon caveat spec.          | 1                                                                                          |
-| user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                   |
-| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                 |
-|                                                                   refresh - only used to authorize a token refresh                                          |
-| time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                |
-|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server. |
-|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.  |
-+-------------+--------------------------------------------------+--------------------------------------------------------------------------------------------+
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
+| Caveat name | Description                                      | Legal Values                                                                                   |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
+| gen         | Generation of the macaroon caveat spec.          | 1                                                                                              |
+| user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                       |
+| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                     |
+|                                                                   refresh - only used to authorize a token refresh                                              |
+| time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                    |
+|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server.     |
+|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.      |
+|                                                                  Operator == means the macaroon is valid at exactly the timestamp, as interpreted by the server.|
+|                                                                  Note that exact equality of time is largely meaningless.                                       |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+

From c3eab1cc3c81cc8a589ed4343dbbbeb2a0027a94 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 15 Sep 2015 11:41:40 +0100
Subject: [PATCH 079/467] Set the current working directory in gendoc.py to the
 script directory as that is where it expects to be run

---
 scripts/gendoc.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index e3c3797e..10c3fa90 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -8,6 +8,8 @@ import shutil
 import subprocess
 import sys
 
+os.chdir(os.path.dirname(__file__))
+
 stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]
 }

From 7eb8b4fde275834496580cf642de6dd5eca8c377 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 15 Sep 2015 15:52:36 +0100
Subject: [PATCH 080/467] Add new-style docs for the APIs for getting events
 for a room

---
 api/client-server/v1/rooms.yaml         | 433 ++++++++++++++++++++++++
 specification/10_client_server_api.rst  |  44 +--
 templating/matrix_templates/sections.py |   6 +
 templating/matrix_templates/units.py    |   4 +-
 4 files changed, 442 insertions(+), 45 deletions(-)
 create mode 100644 api/client-server/v1/rooms.yaml

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
new file mode 100644
index 00000000..9a73bb05
--- /dev/null
+++ b/api/client-server/v1/rooms.yaml
@@ -0,0 +1,433 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Rooms API"
+  version: "1.0.0"
+host: example.com:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/state/{eventType}/{stateKey}":
+    get:
+      summary: Get the state identified by the type and key.
+      description: |-
+        Looks up the contents of a state event in a room. If the user is
+        joined to the room then the state is taken from the current
+        state of the room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to look up the state in.
+          required: true
+          x-example: "!room:example.com"
+        - in: path
+          type: string
+          name: eventType
+          description: The type of state to look up.
+          required: true
+          x-example: "m.room.name"
+        - in: path
+          type: string
+          name: stateKey
+          description: |-
+            The key of the state to look up. Defaults to the empty string.
+          required: false
+          x-example: ""
+      responses:
+        200:
+          description: The content of the state event.
+          examples:
+            application/json: |-
+              {"name": "Example room name"}
+          schema:
+            type: object
+        404:
+          description: The room has no state with the given type or key.
+        403:
+          description: You are not joined to the room.
+
+  "/rooms/{roomId}/state":
+    get:
+      summary: Get all state events in the current state of a room.
+      description: |-
+        Get the state events for the current state of a room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to look up the state for.
+          required: true
+          x-example: "!room:example.com"
+      responses:
+        200:
+          description: The current state of the room
+          examples:
+            application/json: |-
+              [
+                {
+                    "age": 7148266897,
+                    "content": {
+                        "join_rule": "public"
+                    },
+                    "event_id": "$14259997323TLwtb:example.com",
+                    "origin_server_ts": 1425999732392,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.join_rules",
+                    "user_id": "@alice:example.com"
+                },
+                {
+                    "age": 6547561012,
+                    "content": {
+                        "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto",
+                        "displayname": null,
+                        "membership": "join"
+                    },
+                    "event_id": "$1426600438280zExKY:example.com",
+                    "membership": "join",
+                    "origin_server_ts": 1426600438277,
+                    "room_id": "!room:example.com",
+                    "state_key": "@alice:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@alice:example.com"
+                },
+                {
+                    "age": 7148267200,
+                    "content": {
+                        "creator": "@alice:example.com"
+                    },
+                    "event_id": "$14259997320KhbwJ:example.com",
+                    "origin_server_ts": 1425999732089,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.create",
+                    "user_id": "@alice:example.com"
+                },
+                {
+                    "age": 1622568720,
+                    "content": {
+                        "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto",
+                        "displayname": "Bob",
+                        "membership": "join"
+                    },
+                    "event_id": "$1431525430134MxlLX:example.com",
+                    "origin_server_ts": 1431525430569,
+                    "replaces_state": "$142652023736BSXcM:example.com",
+                    "room_id": "!room:example.com",
+                    "state_key": "@bob:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@bob:example.com"
+                },
+                {
+                    "age": 7148267004,
+                    "content": {
+                        "ban": 50,
+                        "events": {
+                            "m.room.name": 100,
+                            "m.room.power_levels": 100
+                         },
+                         "events_default": 0,
+                         "kick": 50,
+                         "redact": 50,
+                         "state_default": 50,
+                         "users": {
+                             "@alice:example.com": 100
+                         },
+                         "users_default": 0
+                    },
+                    "event_id": "$14259997322mqfaq:example.com",
+                    "origin_server_ts": 1425999732285,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.power_levels",
+                    "user_id": "@alice:example.com"
+                }
+              ]
+            schema:
+              type: array
+              title: RoomState
+              description: |-
+                If the user is a member of the room this will be the
+                current state of the room as a list of events.
+              items:
+                title: StateEvent
+                type: object
+                allOf:
+                  - "$ref": "definitions/state_event.yaml"
+        403:
+          description: You are not joined to the room.
+
+  "/rooms/{roomId}/initialSync":
+    get:
+      summary: Snapshot the current state of a room and its most recent messages.
+      description: |-
+        Get a copy of the current state and the most recent messages in a room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to get the data.
+          required: true
+          x-example: "!room:example.com"
+      responses:
+        200:
+          description: The current state of the room
+          examples:
+            application/json: |-
+              {
+                "membership": "join",
+                "messages": {
+                  "chunk": [
+                    {
+                      "age": 343513403,
+                      "content": {
+                        "body": "foo",
+                        "msgtype": "m.text"
+                      },
+                      "event_id": "$14328044851tzTJS:example.com",
+                      "origin_server_ts": 1432804485886,
+                      "room_id": "!room:example.com",
+                      "type": "m.room.message",
+                      "user_id": "@alice:example.com"
+                    },
+                    {
+                      "age": 343511809,
+                      "content": {
+                        "body": "bar",
+                        "msgtype": "m.text"
+                      },
+                      "event_id": "$14328044872spjFg:example.com",
+                      "origin_server_ts": 1432804487480,
+                      "room_id": "!room:example.com",
+                      "type": "m.room.message",
+                      "user_id": "@bob:example.com"
+                    }
+                  ],
+                  "end": "s3456_9_0",
+                  "start": "t44-3453_9_0"
+                },
+                "room_id": "!room:example.com",
+                "state": [
+                  {
+                    "age": 7148266897,
+                    "content": {
+                      "join_rule": "public"
+                    },
+                    "event_id": "$14259997323TLwtb:example.com",
+                    "origin_server_ts": 1425999732392,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.join_rules",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 6547561012,
+                    "content": {
+                      "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto",
+                      "displayname": null,
+                      "membership": "join"
+                    },
+                    "event_id": "$1426600438280zExKY:example.com",
+                    "membership": "join",
+                    "origin_server_ts": 1426600438277,
+                    "room_id": "!room:example.com",
+                    "state_key": "@alice:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 7148267200,
+                    "content": {
+                        "creator": "@alice:example.com"
+                    },
+                    "event_id": "$14259997320KhbwJ:example.com",
+                    "origin_server_ts": 1425999732089,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.create",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 1622568720,
+                    "content": {
+                        "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto",
+                        "displayname": "Bob",
+                        "membership": "join"
+                    },
+                    "event_id": "$1431525430134MxlLX:example.com",
+                    "origin_server_ts": 1431525430569,
+                    "replaces_state": "$142652023736BSXcM:example.com",
+                    "room_id": "!room:example.com",
+                    "state_key": "@bob:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@bob:example.com"
+                  },
+                  {
+                    "age": 7148267004,
+                    "content": {
+                      "ban": 50,
+                      "events": {
+                        "m.room.name": 100,
+                        "m.room.power_levels": 100
+                      },
+                      "events_default": 0,
+                      "kick": 50,
+                      "redact": 50,
+                      "state_default": 50,
+                      "users": {
+                        "@alice:example.com": 100
+                      },
+                      "users_default": 0
+                    },
+                    "event_id": "$14259997322mqfaq:example.com",
+                    "origin_server_ts": 1425999732285,
+                    "room_id": "!room:example.com",
+                    "state_key": "",
+                    "type": "m.room.power_levels",
+                    "user_id": "@alice:example.com"
+                  }
+                ],
+                "visibility": "private"
+              }
+          schema:
+            title: RoomInfo
+            type: object
+            properties:
+              room_id:
+                type: string
+                description: "The ID of this room."
+              membership:
+                type: string
+                description: "The user's membership state in this room."
+                enum: ["invite", "join", "leave", "ban"]
+              messages:
+                type: object
+                title: PaginationChunk
+                description: "The pagination chunk for this room."
+                properties:
+                  start:
+                    type: string
+                    description: |-
+                      A token which correlates to the first value in ``chunk``.
+                      Used for pagination.
+                  end:
+                    type: string
+                    description: |-
+                      A token which correlates to the last value in ``chunk``.
+                      Used for pagination.
+                  chunk:
+                    type: array
+                    description: |-
+                      If the user is a member of the room this will be a
+                      list of the most recent messages for this room. If
+                      the user has left the room this will be the
+                      messages that preceeded them leaving. This array
+                      will consist of at most ``limit`` elements.
+                    items:
+                      type: object
+                      title: RoomEvent
+                      allOf:
+                        - "$ref": "definitions/room_event.yaml"
+                required: ["start", "end", "chunk"]
+              state:
+                type: array
+                description: |-
+                  If the user is a member of the room this will be the
+                  current state of the room as a list of events. If the
+                  user has left the room this will be the state of the
+                  room when they left it.
+                items:
+                  title: StateEvent
+                  type: object
+                  allOf:
+                    - "$ref": "definitions/state_event.yaml"
+              visibility:
+                type: string
+                enum: ["private", "public"]
+                description: |-
+                  Whether this room is visible to the ``/publicRooms`` API
+                  or not."
+            required: ["room_id", "membership"]
+
+  "/rooms/{roomId}/members":
+    get:
+      summary: Get the m.room.member events for the room.
+      description:
+        Get the list of members of the room.
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to get the member events for.
+          required: true,
+          x-example: "!room:example.com"
+      responses:
+        200:
+          description: The members of the room.
+          examples:
+            application/json: |-
+              {
+                "chunk": [
+                  {
+                    "age": 6547561012,
+                    "content": {
+                      "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto",
+                      "displayname": null,
+                      "membership": "join"
+                    },
+                    "event_id": "$1426600438280zExKY:example.com",
+                    "membership": "join",
+                    "origin_server_ts": 1426600438277,
+                    "room_id": "!room:example.com",
+                    "state_key": "@alice:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@alice:example.com"
+                  },
+                  {
+                    "age": 1622568720,
+                    "content": {
+                        "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto",
+                        "displayname": "Bob",
+                        "membership": "join"
+                    },
+                    "event_id": "$1431525430134MxlLX:example.com",
+                    "origin_server_ts": 1431525430569,
+                    "replaces_state": "$142652023736BSXcM:example.com",
+                    "room_id": "!room:example.com",
+                    "state_key": "@bob:example.com",
+                    "type": "m.room.member",
+                    "user_id": "@bob:example.com"
+                  },
+                ]
+              }
+          schema:
+            object:
+            properities:
+              chunk:
+                type: array
+                items:
+                  title: MemberEvent
+                  type: object
+                  allOf:
+                    - "$ref": "../../event-schemas/schema/v1/m.room.member"
+              visibility:
+                type: string
+
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index fe11d199..2c85a4c2 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -631,49 +631,7 @@ Getting events for a room
 
 There are several APIs provided to ``GET`` events for a room:
 
-``/rooms/<room id>/state/<event type>/<state key>``
-  Description:
-    Get the state event identified.
-  Response format:
-    A JSON object representing the state event **content**.
-  Example:
-    ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }``
-
-|/rooms/<room_id>/state|_
-  Description:
-    Get all state events for a room.
-  Response format:
-    ``[ { state event }, { state event }, ... ]``
-  Example:
-    TODO-doc
-
-|/rooms/<room_id>/members|_
-  Description:
-    Get all ``m.room.member`` state events.
-  Response format:
-    ``{ "start": "<token>", "end": "<token>", "chunk": [ { m.room.member event }, ... ] }``
-  Example:
-    TODO-doc
-
-|/rooms/<room_id>/messages|_
-  Description:
-    Get all events from the room's timeline. This API supports
-    pagination using ``from`` and ``to`` query parameters, coupled with the
-    ``start`` and ``end`` tokens from an |initialSync|_ API.
-    
-  Response format:
-    ``{ "start": "<token>", "end": "<token>" }``
-  Example:
-    TODO-doc
-
-|/rooms/<room_id>/initialSync|_
-  Description:
-    Get all relevant events for a room. This includes state events, paginated
-    non-state events and presence events.
-  Response format:
-    `` { TODO-doc } ``
-  Example:
-    TODO-doc
+{{rooms_http_api}}
 
 Redactions
 ~~~~~~~~~~
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 729157bb..2a591c11 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -102,6 +102,12 @@ class MatrixSections(Sections):
             title_kind="~"
         )
 
+    def render_rooms_http_api(self):
+        return self._render_http_api_group(
+            "rooms",
+            title_kind="+"
+        )
+
     def render_room_events(self):
         def filterFn(eventType):
             return (
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 0c072df3..98f0f3a9 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -149,8 +149,8 @@ class MatrixUnits(Units):
                     # object with some keys; we'll add entries f.e one)
                     if "schema" not in param:
                         raise Exception(
-                            "API endpoint group=%s path=%s method=%s param=%s"+
-                            " has no valid parameter value." % (
+                            ("API endpoint group=%s path=%s method=%s param=%s"+
+                            " has no valid parameter value.") % (
                                 group_name, path, method, param
                             )
                         )

From 2e9d3d283a351dae22bf2d88e8dfed821442d6de Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 15 Sep 2015 16:23:19 +0100
Subject: [PATCH 081/467] Swagger refresh tokens

---
 api/client-server/v1/login.yaml | 71 ++++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 2df695be..0852db6b 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -63,7 +63,19 @@ paths:
                 description: The fully-qualified Matrix ID that has been registered.
               access_token:
                 type: string
-                description: An access token for the account. This access token can then be used to authorize other requests.
+                description: |-
+                  An access token for the account.
+                  This access token can then be used to authorize other requests.
+                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+                  There is no specific error message to indicate that a request has failed because
+                  an access token has expired; instead, if a client has reason to believe its
+                  access token is valid, and it receives an auth error, they should attempt to
+                  refresh for a new token on failure, and retry the request with the new token.
+              refresh_token:
+                type: string
+                # TODO: Work out how to linkify /tokenrefresh
+                description: |-
+                  (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the /tokenrefresh API endpoint.
               home_server:
                 type: string
                 description: The hostname of the Home Server on which the account has been registered.
@@ -77,3 +89,60 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
+  "/tokenrefresh":
+    post:
+      summary: Exchanges a refresh token for an access token.
+      description: |-
+        Exchanges a refresh token for a new access token.
+        This is intended to be used if the access token has expired.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: body
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "refresh_token": "a1b2c3"
+              }
+            properties:
+              refresh_token:
+                type: string
+                description: The refresh token which was issued by the server.
+            required: ["refresh_token"]
+      responses:
+        200:
+          description: |-
+            The refresh token was accepted, and a new access token has been issued.
+            The passed refresh token is no longer valid, and cannot be used.
+            A new refresh token may have been returned.
+          examples:
+            application/json: |-
+              {
+                "access_token": "bearwithme123",
+                "refresh_token": "exchangewithme987"
+              }
+          schema:
+            type: object
+            properties:
+              access_token:
+                type: string
+                description: |-
+                  An access token for the account.
+                  This access token can then be used to authorize other requests.
+                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+              refresh_token:
+                type: string
+                description: (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the TODO Linkify /tokenrefresh API endpoint.
+        403:
+          description: |-
+            The exchange attempt failed. For example, the refresh token may have already been used.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"

From 21411309e0569ffa8a0b95bff06e1e989821de2d Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 15 Sep 2015 16:27:10 +0100
Subject: [PATCH 082/467] Fix chdir path

---
 scripts/gendoc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 10c3fa90..ed172726 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -8,7 +8,7 @@ import shutil
 import subprocess
 import sys
 
-os.chdir(os.path.dirname(__file__))
+os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
 stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]

From 6147562d14a0efb7fe14b64c1cf8637a0c86bf8f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 10:09:37 +0100
Subject: [PATCH 083/467] Split the "core" definition files into separate
 files. To make it easier to use the schema files with tools that don't
 support deep links.

---
 event-schemas/schema/v1/core                  | 88 -------------------
 event-schemas/schema/v1/core/event.json       | 24 +++++
 .../v1/core/msgtype_infos/image_info.json     | 23 +++++
 event-schemas/schema/v1/core/room_event.json  | 16 ++++
 event-schemas/schema/v1/core/state_event.json | 20 +++++
 event-schemas/schema/v1/m.call.answer         |  2 +-
 event-schemas/schema/v1/m.call.candidates     |  2 +-
 event-schemas/schema/v1/m.call.hangup         |  2 +-
 event-schemas/schema/v1/m.call.invite         |  2 +-
 event-schemas/schema/v1/m.room.aliases        |  2 +-
 .../schema/v1/m.room.canonical_alias          |  2 +-
 event-schemas/schema/v1/m.room.create         |  2 +-
 .../schema/v1/m.room.history_visibility       |  2 +-
 event-schemas/schema/v1/m.room.join_rules     |  2 +-
 event-schemas/schema/v1/m.room.member         |  2 +-
 event-schemas/schema/v1/m.room.message        |  2 +-
 .../schema/v1/m.room.message#m.audio          |  2 +-
 .../schema/v1/m.room.message#m.emote          |  2 +-
 event-schemas/schema/v1/m.room.message#m.file |  4 +-
 .../schema/v1/m.room.message#m.image          |  4 +-
 .../schema/v1/m.room.message#m.location       |  4 +-
 .../schema/v1/m.room.message#m.notice         |  2 +-
 event-schemas/schema/v1/m.room.message#m.text |  2 +-
 .../schema/v1/m.room.message#m.video          |  4 +-
 .../schema/v1/m.room.message.feedback         |  2 +-
 event-schemas/schema/v1/m.room.name           |  2 +-
 event-schemas/schema/v1/m.room.power_levels   |  2 +-
 event-schemas/schema/v1/m.room.redaction      |  2 +-
 event-schemas/schema/v1/m.room.topic          |  2 +-
 29 files changed, 111 insertions(+), 116 deletions(-)
 delete mode 100644 event-schemas/schema/v1/core
 create mode 100644 event-schemas/schema/v1/core/event.json
 create mode 100644 event-schemas/schema/v1/core/msgtype_infos/image_info.json
 create mode 100644 event-schemas/schema/v1/core/room_event.json
 create mode 100644 event-schemas/schema/v1/core/state_event.json

diff --git a/event-schemas/schema/v1/core b/event-schemas/schema/v1/core
deleted file mode 100644
index ed374a69..00000000
--- a/event-schemas/schema/v1/core
+++ /dev/null
@@ -1,88 +0,0 @@
-{
-  "$schema": "http://json-schema.org/draft-04/schema#",
-  "definitions": {
-    "event": {
-      "title": "Event",
-      "description": "The basic set of fields all events must have.",
-      "type": "object",
-      "properties": {
-        "event_id": {
-          "type": "string",
-          "description": "The globally unique event identifier."
-        },
-        "user_id": {
-          "type": "string",
-          "description": "Contains the fully-qualified ID of the user who *sent* this event."
-        },
-        "content": {
-          "type": "object",
-          "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
-        },
-        "type": {
-          "type": "string",
-          "description": "The type of event. This SHOULD be namespaced similar to Java package naming conventions e.g. 'com.example.subdomain.event.type'"
-        }
-      },
-      "required": ["event_id", "user_id", "content", "type"]
-    },
-    "room_event": {
-        "type": "object",
-        "title": "Room Event",
-        "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
-        "allOf":[{
-            "$ref": "#/definitions/event"
-        }],
-        "properties": {
-            "room_id": {
-                "type": "string",
-                "description": "The ID of the room associated with this event."
-            }
-        },
-        "required": ["room_id"]
-    },
-    "state_event": {
-        "type": "object",
-        "title": "State Event",
-        "description": "In addition to the Room Event fields, State Events have the following additional fields.",
-        "allOf":[{
-            "$ref": "#/definitions/room_event"
-        }],
-        "properties": {
-            "state_key": {
-                "type": "string",
-                "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
-            },
-            "prev_content": {
-                "type": "object",
-                "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
-            }
-        },
-        "required": ["state_key"]
-    },
-    "msgtype_infos": {
-      "image_info": {
-        "type": "object",
-        "title": "ImageInfo",
-        "description": "Metadata about an image.",
-        "properties": {
-            "size": {
-                "type": "integer",
-                "description": "Size of the image in bytes."
-            },
-            "w": {
-                "type": "integer",
-                "description": "The width of the image in pixels."
-            },
-            "h": {
-                "type": "integer",
-                "description": "The height of the image in pixels."
-            },
-            "mimetype": {
-                "type": "string",
-                "description": "The mimetype of the image, e.g. ``image/jpeg``."
-            }
-        }
-      }
-    }
-  }
-}
diff --git a/event-schemas/schema/v1/core/event.json b/event-schemas/schema/v1/core/event.json
new file mode 100644
index 00000000..b4824079
--- /dev/null
+++ b/event-schemas/schema/v1/core/event.json
@@ -0,0 +1,24 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "type": "object",
+  "title": "Event",
+  "description": "The basic set of fields all events must have.",
+  "properties": {
+    "event_id": {
+      "type": "string",
+      "description": "The globally unique event identifier."
+    },
+    "user_id": {
+      "type": "string",
+      "description": "Contains the fully-qualified ID of the user who *sent* this event."
+    },
+    "content": {
+      "type": "object",
+      "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
+    },
+    "type": {
+      "type": "string",
+      "description": "The type of event. This SHOULD be namespaced similar to Java package naming conventions e.g. 'com.example.subdomain.event.type'"
+    }
+  }
+}
diff --git a/event-schemas/schema/v1/core/msgtype_infos/image_info.json b/event-schemas/schema/v1/core/msgtype_infos/image_info.json
new file mode 100644
index 00000000..ee75745e
--- /dev/null
+++ b/event-schemas/schema/v1/core/msgtype_infos/image_info.json
@@ -0,0 +1,23 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "title": "ImageInfo",
+  "description": "Metadata about an image.",
+  "properties": {
+    "size": {
+      "type": "integer",
+      "description": "Size of the image in bytes."
+    },
+    "w": {
+      "type": "integer",
+      "description": "The width of the image in pixels."
+    },
+    "h": {
+      "type": "integer",
+      "description": "The height of the image in pixels."
+    },
+    "mimetype": {
+      "type": "string",
+      "description": "The mimetype of the image, e.g. ``image/jpeg``."
+    }
+  }
+}
diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core/room_event.json
new file mode 100644
index 00000000..717cdaae
--- /dev/null
+++ b/event-schemas/schema/v1/core/room_event.json
@@ -0,0 +1,16 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "type": "object",
+  "title": "Room Event",
+  "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
+  "allOf":[{
+    "$ref": "core/event.json"
+  }],
+  "properties": {
+    "room_id": {
+      "type": "string",
+      "description": "The ID of the room associated with this event."
+    }
+  },
+  "required": ["room_id"]
+}
diff --git a/event-schemas/schema/v1/core/state_event.json b/event-schemas/schema/v1/core/state_event.json
new file mode 100644
index 00000000..a237f4d1
--- /dev/null
+++ b/event-schemas/schema/v1/core/state_event.json
@@ -0,0 +1,20 @@
+{
+    "$schema": "http://json-schema.org/draft-04/schema#",
+        "type": "object",
+        "title": "State Event",
+        "description": "In addition to the Room Event fields, State Events have the following additional fields.",
+        "allOf":[{
+            "$ref": "core/room_event.json"
+        }],
+        "properties": {
+            "state_key": {
+                "type": "string",
+                "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
+            },
+            "prev_content": {
+                "type": "object",
+                "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
+            }
+        },
+        "required": ["state_key"]
+}
diff --git a/event-schemas/schema/v1/m.call.answer b/event-schemas/schema/v1/m.call.answer
index 598554f0..abdf4a9b 100644
--- a/event-schemas/schema/v1/m.call.answer
+++ b/event-schemas/schema/v1/m.call.answer
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the callee when they wish to answer the call.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.candidates b/event-schemas/schema/v1/m.call.candidates
index 6475cf51..052ead0b 100644
--- a/event-schemas/schema/v1/m.call.candidates
+++ b/event-schemas/schema/v1/m.call.candidates
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by callers after sending an invite and by the callee after answering. Its purpose is to give the other party additional ICE candidates to try using to communicate.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.hangup b/event-schemas/schema/v1/m.call.hangup
index 3704ec75..383952d8 100644
--- a/event-schemas/schema/v1/m.call.hangup
+++ b/event-schemas/schema/v1/m.call.hangup
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.invite b/event-schemas/schema/v1/m.call.invite
index 2be01466..a852ff43 100644
--- a/event-schemas/schema/v1/m.call.invite
+++ b/event-schemas/schema/v1/m.call.invite
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the caller when they wish to establish a call.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.aliases b/event-schemas/schema/v1/m.room.aliases
index 6585f13d..79fd259b 100644
--- a/event-schemas/schema/v1/m.room.aliases
+++ b/event-schemas/schema/v1/m.room.aliases
@@ -4,7 +4,7 @@
     "title": "Informs the room about what room aliases it has been given.",
     "description": "This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 9f044e58..17caa321 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -4,7 +4,7 @@
     "title": "Informs the room as to which alias is the canonical one.",
     "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index 8f74e69b..ecc2ed63 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -4,7 +4,7 @@
     "title": "The first event in the room.",
     "description": "This is the first event in a room and cannot be changed. It acts as the root of all other events.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 07bd39ff..48b0a5c4 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -4,7 +4,7 @@
     "title": "Controls visibility of history.",
     "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.join_rules b/event-schemas/schema/v1/m.room.join_rules
index 2ba20b89..567247eb 100644
--- a/event-schemas/schema/v1/m.room.join_rules
+++ b/event-schemas/schema/v1/m.room.join_rules
@@ -4,7 +4,7 @@
     "title": "Describes how users are allowed to join the room.",
     "description": "A room may be ``public`` meaning anyone can join the room without any prior action. Alternatively, it can be ``invite`` meaning that a user who wishes to join the room must first receive an invite to the room from someone already inside of the room. Currently, ``knock`` and ``private`` are reserved keywords which are not implemented.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index ab616176..67e10d51 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -4,7 +4,7 @@
     "title": "The current membership state of a user in the room.",
     "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index a849f07f..61b6256f 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -4,7 +4,7 @@
     "title": "Message",
     "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.audio b/event-schemas/schema/v1/m.room.message#m.audio
index 9910cb0c..b236e5b0 100644
--- a/event-schemas/schema/v1/m.room.message#m.audio
+++ b/event-schemas/schema/v1/m.room.message#m.audio
@@ -4,7 +4,7 @@
     "title": "AudioMessage",
     "description": "This message represents a single audio clip.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.emote b/event-schemas/schema/v1/m.room.message#m.emote
index 886abafc..9f17180c 100644
--- a/event-schemas/schema/v1/m.room.message#m.emote
+++ b/event-schemas/schema/v1/m.room.message#m.emote
@@ -4,7 +4,7 @@
     "title": "EmoteMessage",
     "description": "This message is similar to ``m.text`` except that the sender is 'performing' the action contained in the ``body`` key, similar to ``/me`` in IRC. This message should be prefixed by the name of the sender. This message could also be represented in a different colour to distinguish it from regular ``m.text`` messages.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.file b/event-schemas/schema/v1/m.room.message#m.file
index cf6d6552..d8c63b33 100644
--- a/event-schemas/schema/v1/m.room.message#m.file
+++ b/event-schemas/schema/v1/m.room.message#m.file
@@ -4,7 +4,7 @@
     "title": "FileMessage",
     "description": "This message represents a generic file.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -50,7 +50,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core#/definitions/msgtype_infos/image_info"
+                        "$ref": "core/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.image b/event-schemas/schema/v1/m.room.message#m.image
index 8c010b0d..ca22654f 100644
--- a/event-schemas/schema/v1/m.room.message#m.image
+++ b/event-schemas/schema/v1/m.room.message#m.image
@@ -4,7 +4,7 @@
     "title": "ImageMessage",
     "description": "This message represents a single image and an optional thumbnail.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -31,7 +31,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core#/definitions/msgtype_infos/image_info"
+                        "$ref": "core/msgtype_infos/image_info.json"
                     }]
                 },
                 "info": {
diff --git a/event-schemas/schema/v1/m.room.message#m.location b/event-schemas/schema/v1/m.room.message#m.location
index cd15f31e..919c3325 100644
--- a/event-schemas/schema/v1/m.room.message#m.location
+++ b/event-schemas/schema/v1/m.room.message#m.location
@@ -4,7 +4,7 @@
     "title": "LocationMessage",
     "description": "This message represents a real-world location.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -30,7 +30,7 @@
                     "type": "object",
                     "title": "ImageInfo",
                     "allOf": [{
-                        "$ref": "core#/definitions/msgtype_infos/image_info"
+                        "$ref": "core/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.notice b/event-schemas/schema/v1/m.room.message#m.notice
index 0f8044f5..e6ce6f89 100644
--- a/event-schemas/schema/v1/m.room.message#m.notice
+++ b/event-schemas/schema/v1/m.room.message#m.notice
@@ -4,7 +4,7 @@
     "title": "NoticeMessage",
     "description": "A m.notice message should be considered similar to a plain m.text message except that clients should visually distinguish it in some way. It is intended to be used by automated clients, such as bots, bridges, and other entities, rather than humans. Additionally, such automated agents which watch a room for messages and respond to them ought to ignore m.notice messages. This helps to prevent infinite-loop situations where two automated clients continuously exchange messages, as each responds to the other.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.text b/event-schemas/schema/v1/m.room.message#m.text
index 054d4a96..ceadbcd0 100644
--- a/event-schemas/schema/v1/m.room.message#m.text
+++ b/event-schemas/schema/v1/m.room.message#m.text
@@ -4,7 +4,7 @@
     "title": "TextMessage",
     "description": "This message is the most basic message and is used to represent text.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.video b/event-schemas/schema/v1/m.room.message#m.video
index 22266506..667832ff 100644
--- a/event-schemas/schema/v1/m.room.message#m.video
+++ b/event-schemas/schema/v1/m.room.message#m.video
@@ -4,7 +4,7 @@
     "title": "VideoMessage",
     "description": "This message represents a single video clip.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
@@ -55,7 +55,7 @@
                             "type": "object",
                             "title": "ImageInfo",
                             "allOf": [{
-                                "$ref": "core#/definitions/msgtype_infos/image_info"
+                                "$ref": "core/msgtype_infos/image_info.json"
                             }]
                         }
                     }
diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index 4cfd44d1..b662e9e6 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -4,7 +4,7 @@
     "title": "MessageFeedback",
     "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index 5077774c..5565147c 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -4,7 +4,7 @@
     "description": "A room has an opaque room ID which is not human-friendly to read. A room alias is human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not unique, as multiple rooms can have the same room name set. The room name can also be set when creating a room using ``/createRoom`` with the ``name`` key.",
     "type": "object",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.power_levels b/event-schemas/schema/v1/m.room.power_levels
index 9e0cb81a..95850fcb 100644
--- a/event-schemas/schema/v1/m.room.power_levels
+++ b/event-schemas/schema/v1/m.room.power_levels
@@ -4,7 +4,7 @@
     "title": "Defines the power levels (privileges) of users in the room.",
     "description": "This event specifies the minimum level a user must have in order to perform a certain action. It also specifies the levels of each user in the room. If a ``user_id`` is in the ``users`` list, then that ``user_id`` has the associated power level. Otherwise they have the default level ``users_default``. If ``users_default`` is not supplied, it is assumed to be 0. The level required to send a certain event is governed by ``events``, ``state_default`` and ``events_default``. If an event type is specified in ``events``, then the user must have at least the level specified in order to send that event. If the event type is not supplied, it defaults to ``events_default`` for Message Events and ``state_default`` for State Events.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.redaction b/event-schemas/schema/v1/m.room.redaction
index 2896c0da..b095b17d 100644
--- a/event-schemas/schema/v1/m.room.redaction
+++ b/event-schemas/schema/v1/m.room.redaction
@@ -4,7 +4,7 @@
     "title": "Redaction",
     "description": "Events can be redacted by either room or server admins. Redacting an event means that all keys not required by the protocol are stripped off, allowing admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data.  There is also a concept of a moderator hiding a message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key.",
     "allOf": [{
-        "$ref": "core#/definitions/room_event"
+        "$ref": "core/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.topic b/event-schemas/schema/v1/m.room.topic
index 266df8fa..32319c38 100644
--- a/event-schemas/schema/v1/m.room.topic
+++ b/event-schemas/schema/v1/m.room.topic
@@ -4,7 +4,7 @@
     "title": "Topic",
     "description": "A topic is a short message detailing what is currently being discussed in the room.  It can also be used as a way to display extra information about the room, which may not be suitable for the room name. The room topic can also be set when creating a room using ``/createRoom`` with the ``topic`` key.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core/state_event.json"
     }],
     "properties": {
         "content": {

From 2cf8da6b20dabe3d97ac9c97a6d356b4aa2c4d5d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 10:28:57 +0100
Subject: [PATCH 084/467] Update the gendoc script to load the core event
 schema from separate files.

---
 templating/matrix_templates/units.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 0c072df3..6bb9b7d7 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -283,17 +283,25 @@ class MatrixUnits(Units):
     def load_common_event_fields(self):
         path = "../event-schemas/schema/v1/core"
         event_types = {}
-        with open(path, "r") as f:
-            core_json = json.loads(f.read())
-            for event_type in core_json["definitions"]:
+
+        for (root, dirs, files) in os.walk(path):
+            for filename in files:
+                if not filename.endswith(".json"):
+                    continue
+
+                event_type = filename[:-5]  # strip the ".json"
+                with open(os.path.join(root, filename)) as f:
+                    event_info = json.loads(f.read())
+
                 if "event" not in event_type:
                     continue  # filter ImageInfo and co
-                event_info = core_json["definitions"][event_type]
+
                 table = {
                     "title": event_info["title"],
                     "desc": event_info["description"],
                     "rows": []
                 }
+
                 for prop in sorted(event_info["properties"]):
                     row = {
                         "key": prop,
@@ -301,6 +309,7 @@ class MatrixUnits(Units):
                         "desc": event_info["properties"][prop].get("description","")
                     }
                     table["rows"].append(row)
+
                 event_types[event_type] = table
         return event_types
 

From c49338006aa1fe8da43185ea77afa7510673c99d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 13:09:35 +0100
Subject: [PATCH 085/467] replace definitions/*event.yaml with symlinks to
 /event-schemas/

---
 api/client-server/v1/core                     |   1 +
 api/client-server/v1/definitions/event.yaml   |   7 ----
 .../v1/definitions/room_event.yaml            |   9 -----
 .../v1/definitions/state_event.yaml           |  11 ------
 api/client-server/v1/events                   |   1 +
 api/client-server/v1/presence.yaml            |   2 +-
 api/client-server/v1/sync.yaml                |  10 ++---
 .../schema/v1/core/.room_event.json.swp       | Bin 0 -> 12288 bytes
 event-schemas/schema/v1/core/event.json       |   9 -----
 event-schemas/schema/v1/core/room_event.json  |  11 +++++-
 event-schemas/schema/v1/core/state_event.json |  35 +++++++++---------
 11 files changed, 34 insertions(+), 62 deletions(-)
 create mode 120000 api/client-server/v1/core
 delete mode 100644 api/client-server/v1/definitions/event.yaml
 delete mode 100644 api/client-server/v1/definitions/room_event.yaml
 delete mode 100644 api/client-server/v1/definitions/state_event.yaml
 create mode 120000 api/client-server/v1/events
 create mode 100644 event-schemas/schema/v1/core/.room_event.json.swp

diff --git a/api/client-server/v1/core b/api/client-server/v1/core
new file mode 120000
index 00000000..9fc0cd87
--- /dev/null
+++ b/api/client-server/v1/core
@@ -0,0 +1 @@
+events/core
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/event.yaml b/api/client-server/v1/definitions/event.yaml
deleted file mode 100644
index cfb1b924..00000000
--- a/api/client-server/v1/definitions/event.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-type: object
-description: A Matrix Event
-properties:
-  event_id:
-    type: string
-    description: An event ID.
-required: ["event_id"]
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/room_event.yaml b/api/client-server/v1/definitions/room_event.yaml
deleted file mode 100644
index 5a9e3a09..00000000
--- a/api/client-server/v1/definitions/room_event.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-type: object
-description: A Matrix Room Event
-properties:
-  event_id:
-    type: string
-    description: An event ID.
-  room_id:
-    type: string
-required: ["event_id", "room_id"]
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/state_event.yaml b/api/client-server/v1/definitions/state_event.yaml
deleted file mode 100644
index f8e6f4d4..00000000
--- a/api/client-server/v1/definitions/state_event.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-type: object
-description: A Matrix State Event
-properties:
-  event_id:
-    type: string
-    description: An event ID.
-  room_id:
-    type: string
-  state_key:
-    type: string
-required: ["event_id", "room_id", "state_key"]
\ No newline at end of file
diff --git a/api/client-server/v1/events b/api/client-server/v1/events
new file mode 120000
index 00000000..7a0d0326
--- /dev/null
+++ b/api/client-server/v1/events
@@ -0,0 +1 @@
+../../../event-schemas/schema/v1
\ No newline at end of file
diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 6f8311e4..472a2d7f 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -205,4 +205,4 @@ paths:
               type: object
               title: PresenceEvent
               allOf:
-                - "$ref": "definitions/event.yaml"
+                - "$ref": "events/core/event.json"
diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 777e27db..b9525491 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -82,7 +82,7 @@ paths:
                   type: object
                   title: RoomEvent
                   allOf:
-                   - "$ref": "definitions/room_event.yaml"
+                   - "$ref": "events/core/room_event.json"
         400:
           description: "Bad pagination ``from`` parameter."
   "/initialSync":
@@ -253,7 +253,7 @@ paths:
                   type: object
                   title: Event
                   allOf:
-                   - "$ref": "definitions/event.yaml"
+                   - "$ref": "events/core/event.json"
               rooms:
                 type: array
                 items:
@@ -294,7 +294,7 @@ paths:
                             type: object
                             title: RoomEvent
                             allOf:
-                              - "$ref": "definitions/room_event.yaml"
+                              - "$ref": "events/core/room_event.json"
                       required: ["start", "end", "chunk"]
                     state:
                       type: array
@@ -307,7 +307,7 @@ paths:
                         title: StateEvent
                         type: object
                         allOf:
-                          - "$ref": "definitions/state_event.yaml"
+                          - "$ref": "events/core/state_event.json"
                     visibility:
                       type: string
                       enum: ["private", "public"]
@@ -350,6 +350,6 @@ paths:
               }
           schema:
             allOf:
-              - "$ref": "definitions/event.yaml"
+              - "$ref": "events/core/event.json"
         404:
           description: The event was not found or you do not have permission to read this event.
diff --git a/event-schemas/schema/v1/core/.room_event.json.swp b/event-schemas/schema/v1/core/.room_event.json.swp
new file mode 100644
index 0000000000000000000000000000000000000000..fce0fd5f873a95140af3b68ffd4843fa3cb522c4
GIT binary patch
literal 12288
zcmeI2&2G~`5XZN0E}$a8g~PA~AyrM>@S#fOz=01bhpG}ZNT{l+jlGUHI9_LWolsQ~
z2i^b>7oH&!2e|PPyatcZ17g;85~+t!j-XlTpS3;XotfX9tW0<7WqTL4oA(*6D~#>E
zKkeL}-nlq!F*Xq9bwBZj<tr=Pa);U)SQ7?4<&vTs41_c4O+%aB>@w0}AV)HD4I3tM
zAZ$?W!Kf94+Q`6YJvb^ontiLYbA==)B0vP@1XkI-yR9vMyUuUI)wRytLKj4U2oM1x
zKm>>Y5g-CYfC&8m1YEwt-eQ>->Qb-Q`^uTUzNL)_5CI}U1c(3;AOb{y2oM1xKm>>Y
z5%>oQh={QptBid_`TzfC-~YcZGxihp0~MoQp&p_hpw>}qsMAY~eML=CpHc5n&rpw0
zP1H5icRcqS>I>=<%E$SDqD2IV01+SpM1Tko0U|&IhyW2F0*p-<05@`2C?g}j0|&eu
zeO3`aM9X9of94x{M$(2x<xc4go%v&(IiWHOE|CxyX?olk79v%#k`c6@fR0N`VWoj_
zqG7}0%=HF7Qx>Y}YF-AwlihJH{gD=DwX+_t0(1Z75hk+(C(AS0#{_z*?h4ESicAd)
zSw#R9VN9QqY5pNtnbEnUEz0x6Kv=6oB^(w$#<BCisBs3os*&eL=h8SOZC&-`u!eEV
z$k-z)C)5|i>AotCb?>3^z+*bwrFI4)ij;2}I9+z+sULm^gpx*f6Q275%10J<J1_Pj
q5hGdp#X3#(7#lJ_L8R4d&9X(@DVLUgTbxlF>VE2OUxp5!ov`26(Di%(

literal 0
HcmV?d00001

diff --git a/event-schemas/schema/v1/core/event.json b/event-schemas/schema/v1/core/event.json
index b4824079..e73aec80 100644
--- a/event-schemas/schema/v1/core/event.json
+++ b/event-schemas/schema/v1/core/event.json
@@ -1,17 +1,8 @@
 {
-  "$schema": "http://json-schema.org/draft-04/schema#",
   "type": "object",
   "title": "Event",
   "description": "The basic set of fields all events must have.",
   "properties": {
-    "event_id": {
-      "type": "string",
-      "description": "The globally unique event identifier."
-    },
-    "user_id": {
-      "type": "string",
-      "description": "Contains the fully-qualified ID of the user who *sent* this event."
-    },
     "content": {
       "type": "object",
       "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core/room_event.json
index 717cdaae..a64ba2ea 100644
--- a/event-schemas/schema/v1/core/room_event.json
+++ b/event-schemas/schema/v1/core/room_event.json
@@ -1,5 +1,4 @@
 {
-  "$schema": "http://json-schema.org/draft-04/schema#",
   "type": "object",
   "title": "Room Event",
   "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
@@ -10,7 +9,15 @@
     "room_id": {
       "type": "string",
       "description": "The ID of the room associated with this event."
-    }
+    },
+    "event_id": {
+      "type": "string",
+      "description": "The globally unique event identifier."
+    },
+    "user_id": {
+      "type": "string",
+      "description": "Contains the fully-qualified ID of the user who *sent* this event."
+    },
   },
   "required": ["room_id"]
 }
diff --git a/event-schemas/schema/v1/core/state_event.json b/event-schemas/schema/v1/core/state_event.json
index a237f4d1..60de9413 100644
--- a/event-schemas/schema/v1/core/state_event.json
+++ b/event-schemas/schema/v1/core/state_event.json
@@ -1,20 +1,19 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
-        "type": "object",
-        "title": "State Event",
-        "description": "In addition to the Room Event fields, State Events have the following additional fields.",
-        "allOf":[{
-            "$ref": "core/room_event.json"
-        }],
-        "properties": {
-            "state_key": {
-                "type": "string",
-                "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
-            },
-            "prev_content": {
-                "type": "object",
-                "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
-            }
-        },
-        "required": ["state_key"]
+  "type": "object",
+  "title": "State Event",
+  "description": "In addition to the Room Event fields, State Events have the following additional fields.",
+  "allOf":[{
+    "$ref": "core/room_event.json"
+  }],
+  "properties": {
+    "state_key": {
+      "type": "string",
+      "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
+    },
+    "prev_content": {
+      "type": "object",
+      "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
+    }
+  },
+  "required": ["state_key"]
 }

From 380f186273b4b7e045c76b0078b091063347aef6 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 14:42:43 +0100
Subject: [PATCH 086/467] Log which file a json parse error occurred in

---
 templating/matrix_templates/units.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 6bb9b7d7..4b8127ae 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -290,8 +290,14 @@ class MatrixUnits(Units):
                     continue
 
                 event_type = filename[:-5]  # strip the ".json"
-                with open(os.path.join(root, filename)) as f:
-                    event_info = json.loads(f.read())
+                filepath = os.path.join(root, filename)
+                with open(filepath) as f:
+                    try:
+                        event_info = json.load(f)
+                    except Exception as e:
+                        raise ValueError(
+                            "Error reading file %r" % (filepath,), e
+                        )
 
                 if "event" not in event_type:
                     continue  # filter ImageInfo and co

From d0b018c46ecee5ee01d11a3c244d5e4fd3e30bef Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 14:43:09 +0100
Subject: [PATCH 087/467] Fix syntax error in JSON

---
 .../schema/v1/core/.room_event.json.swp       | Bin 12288 -> 12288 bytes
 event-schemas/schema/v1/core/room_event.json  |   2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/core/.room_event.json.swp b/event-schemas/schema/v1/core/.room_event.json.swp
index fce0fd5f873a95140af3b68ffd4843fa3cb522c4..8140e9ceb080ae0bf455a04a57e8d6966854a472 100644
GIT binary patch
delta 73
zcmZojXh@JsG6?hZRWR2xVE_UF28JgGe}$^8RcAgcyHV<uJgY1(1H;tGoC?vr+kmW%
XKs+5Jvbj(}g?I84UA4`-bzPYN>!=h&

delta 77
zcmZojXh@JsG6?hZRWR2xVE_UF1_u90zd}{lsxz-O+$i-*o>hjIfnmyIPK9XRtw7cW
bAf5&i*<7fg!aI4IuIgkLJ&DbiboH135Umu8

diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core/room_event.json
index a64ba2ea..7ac66b30 100644
--- a/event-schemas/schema/v1/core/room_event.json
+++ b/event-schemas/schema/v1/core/room_event.json
@@ -17,7 +17,7 @@
     "user_id": {
       "type": "string",
       "description": "Contains the fully-qualified ID of the user who *sent* this event."
-    },
+    }
   },
   "required": ["room_id"]
 }

From fd3401fc7a78b3ccbfd9825f01272225d7858407 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 14:44:17 +0100
Subject: [PATCH 088/467] Remove accidentally committed swp file

---
 .../schema/v1/core/.room_event.json.swp         | Bin 12288 -> 0 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 event-schemas/schema/v1/core/.room_event.json.swp

diff --git a/event-schemas/schema/v1/core/.room_event.json.swp b/event-schemas/schema/v1/core/.room_event.json.swp
deleted file mode 100644
index 8140e9ceb080ae0bf455a04a57e8d6966854a472..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 12288
zcmeI2&1w`u5XU=uj!FbQ1<H;ngzQe@2SPk}@B{V`1kDN(Lg=08+39t5dbhi0*ANyD
zK7xn`pCJc9JbCc}JopZte1sgt>YmwMh!EpZsfPdb%v4QR)vr#=R%iWLYYXl)Z!ugK
z82kR^sD175`q{6`jP-?i(aXHy<W-dJWQW=sSrbQH<+8F9^@TI)WkZ|p^fu9PBuBDv
z4I5{&FKkrp(P$%zwULq0y1y4ZnmwzFQ-vfaB0vOY1QyxN8yoBXafM%ji%ad9g>Hxd
z5g-CYfCvx)B0vO)01^2A3An)mdxd45sY|_F?+eHF`jIvwKm>>Y5g-CYfCvx)B0vO)
z01+SpMBpDJAQHxw7a4nt^8f!&fB*kH&)5&tH&lvxj=GDwjaor1p^nZm_7Qc6dXIXI
zdW5=%YN9ToKI6HcP#;k5P(IEZ6fGh^1c(3;AOb{y2oM1xKm>>Y5nyaGVF297VX2Hv
z_$KV|F#D_$zKfR08eVV<S9DCIjg1;Or3>`s_jTcfDlE86LR#keeq&gQT%}4T(0Twm
z4VKbM1LI7?s>PY>ReYx`RMXXbda~OeNPncoS?!|BtGL{Mxre!I!a;DX^Z7lKZA_q>
z>yE%2pe)p|lvM;!3C8ppndTpYg^W%eZC;)S2Etk$E8(#4F^--8MU7M5RgHXL^gtS?
zq^+x-?A9=@8ku@TVM2XAobIdQSodQz?s!bcyVNQ`B#H7(1E)hr9{S;TKqz@)*WigS
yAiT1$)qc7SnHb67m+Cy%V{FLm1d&&tHA9QIQ!WpEo1ak|>VE1@PsR@49k5>;A@-yI


From 9b18fd252ebab7244c3ebdd005e013d54b3a263a Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:46:37 +0100
Subject: [PATCH 089/467] Log port being listened on

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index e7757c06..dc7705a8 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-
+    fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From 9c3a333a6dc1cad4decd8e38c2b64feea6506131 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:49:01 +0100
Subject: [PATCH 090/467] Apparently tabs are A Thing

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index dc7705a8..658ae0fb 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-    fmt.Printf("Listening on port %d\n", *port)
+	fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From 654ed9b99e7dfae5cd227e192c5a5ace111dacf0 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 16:00:06 +0100
Subject: [PATCH 091/467] Remove  keys from the m.room.* schema files so that
 they may be used in the node swagger validator

---
 event-schemas/schema/v1/m.call.answer             | 1 -
 event-schemas/schema/v1/m.call.candidates         | 1 -
 event-schemas/schema/v1/m.call.hangup             | 1 -
 event-schemas/schema/v1/m.call.invite             | 1 -
 event-schemas/schema/v1/m.presence                | 1 -
 event-schemas/schema/v1/m.receipt                 | 1 -
 event-schemas/schema/v1/m.room.aliases            | 1 -
 event-schemas/schema/v1/m.room.canonical_alias    | 1 -
 event-schemas/schema/v1/m.room.create             | 1 -
 event-schemas/schema/v1/m.room.history_visibility | 1 -
 event-schemas/schema/v1/m.room.join_rules         | 1 -
 event-schemas/schema/v1/m.room.member             | 1 -
 event-schemas/schema/v1/m.room.message            | 1 -
 event-schemas/schema/v1/m.room.message#m.audio    | 1 -
 event-schemas/schema/v1/m.room.message#m.emote    | 1 -
 event-schemas/schema/v1/m.room.message#m.file     | 1 -
 event-schemas/schema/v1/m.room.message#m.image    | 1 -
 event-schemas/schema/v1/m.room.message#m.location | 1 -
 event-schemas/schema/v1/m.room.message#m.notice   | 1 -
 event-schemas/schema/v1/m.room.message#m.text     | 1 -
 event-schemas/schema/v1/m.room.message#m.video    | 1 -
 event-schemas/schema/v1/m.room.message.feedback   | 1 -
 event-schemas/schema/v1/m.room.name               | 1 -
 event-schemas/schema/v1/m.room.power_levels       | 1 -
 event-schemas/schema/v1/m.room.redaction          | 1 -
 event-schemas/schema/v1/m.room.topic              | 1 -
 26 files changed, 26 deletions(-)

diff --git a/event-schemas/schema/v1/m.call.answer b/event-schemas/schema/v1/m.call.answer
index abdf4a9b..3b2c48d4 100644
--- a/event-schemas/schema/v1/m.call.answer
+++ b/event-schemas/schema/v1/m.call.answer
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "This event is sent by the callee when they wish to answer the call.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.call.candidates b/event-schemas/schema/v1/m.call.candidates
index 052ead0b..f3c86296 100644
--- a/event-schemas/schema/v1/m.call.candidates
+++ b/event-schemas/schema/v1/m.call.candidates
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "This event is sent by callers after sending an invite and by the callee after answering. Its purpose is to give the other party additional ICE candidates to try using to communicate.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.call.hangup b/event-schemas/schema/v1/m.call.hangup
index 383952d8..0f253ef9 100644
--- a/event-schemas/schema/v1/m.call.hangup
+++ b/event-schemas/schema/v1/m.call.hangup
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.call.invite b/event-schemas/schema/v1/m.call.invite
index a852ff43..393e2455 100644
--- a/event-schemas/schema/v1/m.call.invite
+++ b/event-schemas/schema/v1/m.call.invite
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "description": "This event is sent by the caller when they wish to establish a call.",
     "allOf": [{
diff --git a/event-schemas/schema/v1/m.presence b/event-schemas/schema/v1/m.presence
index bb69ce40..79852ac6 100644
--- a/event-schemas/schema/v1/m.presence
+++ b/event-schemas/schema/v1/m.presence
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Presence Event",
     "description": "Informs the client of a user's presence state change.",
diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index 4bd9f17b..0f365eed 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Receipt Event",
     "description": "Informs the client of new receipts.",
diff --git a/event-schemas/schema/v1/m.room.aliases b/event-schemas/schema/v1/m.room.aliases
index 79fd259b..38fd7d18 100644
--- a/event-schemas/schema/v1/m.room.aliases
+++ b/event-schemas/schema/v1/m.room.aliases
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Informs the room about what room aliases it has been given.",
     "description": "This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.",
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 17caa321..f79ce234 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Informs the room as to which alias is the canonical one.",
     "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index ecc2ed63..9561f19b 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "The first event in the room.",
     "description": "This is the first event in a room and cannot be changed. It acts as the root of all other events.",
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 48b0a5c4..22b39623 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Controls visibility of history.",
     "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
diff --git a/event-schemas/schema/v1/m.room.join_rules b/event-schemas/schema/v1/m.room.join_rules
index 567247eb..095ccfa7 100644
--- a/event-schemas/schema/v1/m.room.join_rules
+++ b/event-schemas/schema/v1/m.room.join_rules
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Describes how users are allowed to join the room.",
     "description": "A room may be ``public`` meaning anyone can join the room without any prior action. Alternatively, it can be ``invite`` meaning that a user who wishes to join the room must first receive an invite to the room from someone already inside of the room. Currently, ``knock`` and ``private`` are reserved keywords which are not implemented.",
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 67e10d51..5acc5f7c 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "The current membership state of a user in the room.",
     "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index 61b6256f..c8c5931e 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Message",
     "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
diff --git a/event-schemas/schema/v1/m.room.message#m.audio b/event-schemas/schema/v1/m.room.message#m.audio
index b236e5b0..f943fc7f 100644
--- a/event-schemas/schema/v1/m.room.message#m.audio
+++ b/event-schemas/schema/v1/m.room.message#m.audio
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "AudioMessage",
     "description": "This message represents a single audio clip.",
diff --git a/event-schemas/schema/v1/m.room.message#m.emote b/event-schemas/schema/v1/m.room.message#m.emote
index 9f17180c..3379d635 100644
--- a/event-schemas/schema/v1/m.room.message#m.emote
+++ b/event-schemas/schema/v1/m.room.message#m.emote
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "EmoteMessage",
     "description": "This message is similar to ``m.text`` except that the sender is 'performing' the action contained in the ``body`` key, similar to ``/me`` in IRC. This message should be prefixed by the name of the sender. This message could also be represented in a different colour to distinguish it from regular ``m.text`` messages.",
diff --git a/event-schemas/schema/v1/m.room.message#m.file b/event-schemas/schema/v1/m.room.message#m.file
index d8c63b33..b1cdff22 100644
--- a/event-schemas/schema/v1/m.room.message#m.file
+++ b/event-schemas/schema/v1/m.room.message#m.file
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "FileMessage",
     "description": "This message represents a generic file.",
diff --git a/event-schemas/schema/v1/m.room.message#m.image b/event-schemas/schema/v1/m.room.message#m.image
index ca22654f..5d4f811b 100644
--- a/event-schemas/schema/v1/m.room.message#m.image
+++ b/event-schemas/schema/v1/m.room.message#m.image
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "ImageMessage",
     "description": "This message represents a single image and an optional thumbnail.",
diff --git a/event-schemas/schema/v1/m.room.message#m.location b/event-schemas/schema/v1/m.room.message#m.location
index 919c3325..f02bea78 100644
--- a/event-schemas/schema/v1/m.room.message#m.location
+++ b/event-schemas/schema/v1/m.room.message#m.location
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "LocationMessage",
     "description": "This message represents a real-world location.",
diff --git a/event-schemas/schema/v1/m.room.message#m.notice b/event-schemas/schema/v1/m.room.message#m.notice
index e6ce6f89..972eb45e 100644
--- a/event-schemas/schema/v1/m.room.message#m.notice
+++ b/event-schemas/schema/v1/m.room.message#m.notice
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "NoticeMessage",
     "description": "A m.notice message should be considered similar to a plain m.text message except that clients should visually distinguish it in some way. It is intended to be used by automated clients, such as bots, bridges, and other entities, rather than humans. Additionally, such automated agents which watch a room for messages and respond to them ought to ignore m.notice messages. This helps to prevent infinite-loop situations where two automated clients continuously exchange messages, as each responds to the other.",
diff --git a/event-schemas/schema/v1/m.room.message#m.text b/event-schemas/schema/v1/m.room.message#m.text
index ceadbcd0..00aa1818 100644
--- a/event-schemas/schema/v1/m.room.message#m.text
+++ b/event-schemas/schema/v1/m.room.message#m.text
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "TextMessage",
     "description": "This message is the most basic message and is used to represent text.",
diff --git a/event-schemas/schema/v1/m.room.message#m.video b/event-schemas/schema/v1/m.room.message#m.video
index 667832ff..b91fd48a 100644
--- a/event-schemas/schema/v1/m.room.message#m.video
+++ b/event-schemas/schema/v1/m.room.message#m.video
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "VideoMessage",
     "description": "This message represents a single video clip.",
diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index b662e9e6..3a069654 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "MessageFeedback",
     "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index 5565147c..2a72feeb 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "title": "RoomName",
     "description": "A room has an opaque room ID which is not human-friendly to read. A room alias is human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not unique, as multiple rooms can have the same room name set. The room name can also be set when creating a room using ``/createRoom`` with the ``name`` key.",
     "type": "object",
diff --git a/event-schemas/schema/v1/m.room.power_levels b/event-schemas/schema/v1/m.room.power_levels
index 95850fcb..b5cb1049 100644
--- a/event-schemas/schema/v1/m.room.power_levels
+++ b/event-schemas/schema/v1/m.room.power_levels
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Defines the power levels (privileges) of users in the room.",
     "description": "This event specifies the minimum level a user must have in order to perform a certain action. It also specifies the levels of each user in the room. If a ``user_id`` is in the ``users`` list, then that ``user_id`` has the associated power level. Otherwise they have the default level ``users_default``. If ``users_default`` is not supplied, it is assumed to be 0. The level required to send a certain event is governed by ``events``, ``state_default`` and ``events_default``. If an event type is specified in ``events``, then the user must have at least the level specified in order to send that event. If the event type is not supplied, it defaults to ``events_default`` for Message Events and ``state_default`` for State Events.",
diff --git a/event-schemas/schema/v1/m.room.redaction b/event-schemas/schema/v1/m.room.redaction
index b095b17d..970679c7 100644
--- a/event-schemas/schema/v1/m.room.redaction
+++ b/event-schemas/schema/v1/m.room.redaction
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Redaction",
     "description": "Events can be redacted by either room or server admins. Redacting an event means that all keys not required by the protocol are stripped off, allowing admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data.  There is also a concept of a moderator hiding a message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key.",
diff --git a/event-schemas/schema/v1/m.room.topic b/event-schemas/schema/v1/m.room.topic
index 32319c38..a9956228 100644
--- a/event-schemas/schema/v1/m.room.topic
+++ b/event-schemas/schema/v1/m.room.topic
@@ -1,5 +1,4 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema#",
     "type": "object",
     "title": "Topic",
     "description": "A topic is a short message detailing what is currently being discussed in the room.  It can also be used as a way to display extra information about the room, which may not be suitable for the room name. The room topic can also be set when creating a room using ``/createRoom`` with the ``topic`` key.",

From 5b59c67510be4d17a81dd437af0d7d1235da24c1 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 16:34:20 +0100
Subject: [PATCH 092/467] Minor formatting fixes. Fix state event templating.

---
 specification/00_basis.rst             |  5 ++--
 specification/10_client_server_api.rst |  7 ++---
 specification/20_events.rst            | 39 +++++++++++++-------------
 templating/matrix_templates/units.py   |  5 ++--
 4 files changed, 26 insertions(+), 30 deletions(-)

diff --git a/specification/00_basis.rst b/specification/00_basis.rst
index 10709f1d..f6d50802 100644
--- a/specification/00_basis.rst
+++ b/specification/00_basis.rst
@@ -171,13 +171,14 @@ All data exchanged over Matrix is expressed as an "event". Typically each client
 action (e.g. sending a message) correlates with exactly one event. Each event
 has a ``type`` which is used to differentiate different kinds of data. ``type``
 values MUST be uniquely globally namespaced following Java's `package naming
-conventions
-<http://docs.oracle.com/javase/specs/jls/se5.0/html/packages.html#7.7>`, e.g.
+conventions`_, e.g.
 ``com.example.myapp.event``. The special top-level namespace ``m.`` is reserved
 for events defined in the Matrix specification - for instance ``m.room.message``
 is the event type for instant messages. Events are usually sent in the context
 of a "Room".
 
+.. _package naming conventions: https://en.wikipedia.org/wiki/Java_package#Package_naming_conventions
+
 Event Graphs
 ~~~~~~~~~~~~
 
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index fe11d199..f0416c02 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -932,11 +932,8 @@ directly by sending the following request to
     "membership": "leave"
   }
 
-See the `Room events`_ section for more information on ``m.room.member``.
-
-Once a user has left a room, that room will no longer appear on the
-|initialSync|_ API.
-
+See the `Room events`_ section for more information on ``m.room.member``. Once a
+user has left a room, that room will no longer appear on the |initialSync|_ API.
 If all members in a room leave, that room becomes eligible for deletion.
 
 Banning users in a room
diff --git a/specification/20_events.rst b/specification/20_events.rst
index b0d5d716..ffc23ec4 100644
--- a/specification/20_events.rst
+++ b/specification/20_events.rst
@@ -92,11 +92,10 @@ Voice over IP
 -------------
 Matrix can also be used to set up VoIP calls. This is part of the core
 specification, although is at a relatively early stage. Voice (and video) over
-Matrix is built on the WebRTC 1.0 standard.
-
-Call events are sent to a room, like any other event. This means that clients
-must only send call events to rooms with exactly two participants as currently
-the WebRTC standard is based around two-party communication.
+Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
+any other event. This means that clients must only send call events to rooms
+with exactly two participants as currently the WebRTC standard is based around
+two-party communication.
 
 {{voip_events}}
 
@@ -106,25 +105,26 @@ A call is set up with messages exchanged as follows:
 
 ::
 
-   Caller                   Callee
+   Caller                    Callee
+ [Place Call]
  m.call.invite ----------->
  m.call.candidate -------->
- [more candidates events]
-                         User answers call
-                  <------ m.call.answer
-               [...]
-                  <------ m.call.hangup
+ [..candidates..] -------->
+                           [Answers call]
+          <--------------- m.call.answer
+    [Call is active and ongoing]
+          <--------------- m.call.hangup
 
 Or a rejected call:
 
 ::
 
-   Caller                   Callee
- m.call.invite ----------->
- m.call.candidate -------->
- [more candidates events]
-                        User rejects call
-                 <------- m.call.hangup
+   Caller                      Callee
+ m.call.invite ------------>
+ m.call.candidate --------->
+ [..candidates..] --------->
+                            [Rejects call]
+            <-------------- m.call.hangup
 
 Calls are negotiated according to the WebRTC specification.
 
@@ -137,9 +137,8 @@ better experience for the users if their calls are connected if it is clear
 that their intention is to set up a call with one another.
 
 In Matrix, calls are to rooms rather than users (even if those rooms may only
-contain one other user) so we consider calls which are to the same room.
-
-The rules for dealing with such a situation are as follows:
+contain one other user) so we consider calls which are to the same room. The
+rules for dealing with such a situation are as follows:
 
  - If an invite to a room is received whilst the client is preparing to send an
    invite to the same room, the client should cancel its outgoing call and
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 4b8127ae..3a8825ae 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -361,8 +361,8 @@ class MatrixUnits(Units):
 
                 # add typeof
                 base_defs = {
-                    "core#/definitions/room_event": "Message Event",
-                    "core#/definitions/state_event": "State Event"
+                    "core/room_event.json": "Message Event",
+                    "core/state_event.json": "State Event"
                 }
                 if type(json_schema.get("allOf")) == list:
                     schema["typeof"] = base_defs.get(
@@ -399,7 +399,6 @@ class MatrixUnits(Units):
                         "`m.room.message msgtypes`_."
                     )
 
-
                 # Assign state key info if it has some
                 if schema["typeof"] == "State Event":
                     skey_desc = Units.prop(

From af32ec194a0fdad546df6f6e35bbd13d8b476c23 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 16:38:58 +0100
Subject: [PATCH 093/467] Move VoIP events to 22_voip_events

20_events is a huge as it is, this makes it easier to find what section
you actually want when editing.
---
 specification/20_events.rst      | 66 --------------------------------
 specification/22_voip_events.rst | 65 +++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 66 deletions(-)
 create mode 100644 specification/22_voip_events.rst

diff --git a/specification/20_events.rst b/specification/20_events.rst
index ffc23ec4..ce36b040 100644
--- a/specification/20_events.rst
+++ b/specification/20_events.rst
@@ -88,69 +88,3 @@ users, they should include the displayname and avatar URL fields in these
 events so that clients already have these details to hand, and do not have to
 perform extra roundtrips to query it.
 
-Voice over IP
--------------
-Matrix can also be used to set up VoIP calls. This is part of the core
-specification, although is at a relatively early stage. Voice (and video) over
-Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
-any other event. This means that clients must only send call events to rooms
-with exactly two participants as currently the WebRTC standard is based around
-two-party communication.
-
-{{voip_events}}
-
-Message Exchange
-~~~~~~~~~~~~~~~~
-A call is set up with messages exchanged as follows:
-
-::
-
-   Caller                    Callee
- [Place Call]
- m.call.invite ----------->
- m.call.candidate -------->
- [..candidates..] -------->
-                           [Answers call]
-          <--------------- m.call.answer
-    [Call is active and ongoing]
-          <--------------- m.call.hangup
-
-Or a rejected call:
-
-::
-
-   Caller                      Callee
- m.call.invite ------------>
- m.call.candidate --------->
- [..candidates..] --------->
-                            [Rejects call]
-            <-------------- m.call.hangup
-
-Calls are negotiated according to the WebRTC specification.
-
-
-Glare
-~~~~~
-This specification aims to address the problem of two users calling each other
-at roughly the same time and their invites crossing on the wire. It is a far
-better experience for the users if their calls are connected if it is clear
-that their intention is to set up a call with one another.
-
-In Matrix, calls are to rooms rather than users (even if those rooms may only
-contain one other user) so we consider calls which are to the same room. The
-rules for dealing with such a situation are as follows:
-
- - If an invite to a room is received whilst the client is preparing to send an
-   invite to the same room, the client should cancel its outgoing call and
-   instead automatically accept the incoming call on behalf of the user.
- - If an invite to a room is received after the client has sent an invite to
-   the same room and is waiting for a response, the client should perform a
-   lexicographical comparison of the call IDs of the two calls and use the
-   lesser of the two calls, aborting the greater. If the incoming call is the
-   lesser, the client should accept this call on behalf of the user.
-
-The call setup should appear seamless to the user as if they had simply placed
-a call and the other party had accepted. Thusly, any media stream that had been
-setup for use on a call should be transferred and used for the call that
-replaces it.
-
diff --git a/specification/22_voip_events.rst b/specification/22_voip_events.rst
new file mode 100644
index 00000000..d75db7e7
--- /dev/null
+++ b/specification/22_voip_events.rst
@@ -0,0 +1,65 @@
+Voice over IP
+-------------
+Matrix can also be used to set up VoIP calls. This is part of the core
+specification, although is at a relatively early stage. Voice (and video) over
+Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
+any other event. This means that clients must only send call events to rooms
+with exactly two participants as currently the WebRTC standard is based around
+two-party communication.
+
+{{voip_events}}
+
+Message Exchange
+~~~~~~~~~~~~~~~~
+A call is set up with messages exchanged as follows:
+
+::
+
+   Caller                    Callee
+ [Place Call]
+ m.call.invite ----------->
+ m.call.candidate -------->
+ [..candidates..] -------->
+                           [Answers call]
+          <--------------- m.call.answer
+    [Call is active and ongoing]
+          <--------------- m.call.hangup
+
+Or a rejected call:
+
+::
+
+   Caller                      Callee
+ m.call.invite ------------>
+ m.call.candidate --------->
+ [..candidates..] --------->
+                            [Rejects call]
+            <-------------- m.call.hangup
+
+Calls are negotiated according to the WebRTC specification.
+
+
+Glare
+~~~~~
+This specification aims to address the problem of two users calling each other
+at roughly the same time and their invites crossing on the wire. It is a far
+better experience for the users if their calls are connected if it is clear
+that their intention is to set up a call with one another.
+
+In Matrix, calls are to rooms rather than users (even if those rooms may only
+contain one other user) so we consider calls which are to the same room. The
+rules for dealing with such a situation are as follows:
+
+ - If an invite to a room is received whilst the client is preparing to send an
+   invite to the same room, the client should cancel its outgoing call and
+   instead automatically accept the incoming call on behalf of the user.
+ - If an invite to a room is received after the client has sent an invite to
+   the same room and is waiting for a response, the client should perform a
+   lexicographical comparison of the call IDs of the two calls and use the
+   lesser of the two calls, aborting the greater. If the incoming call is the
+   lesser, the client should accept this call on behalf of the user.
+
+The call setup should appear seamless to the user as if they had simply placed
+a call and the other party had accepted. Thusly, any media stream that had been
+setup for use on a call should be transferred and used for the call that
+replaces it.

From 46d29e9eea31d32e4865554e0c36badcd4006091 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 17 Sep 2015 16:48:07 +0100
Subject: [PATCH 094/467] fix the room api swagger to be valid swagger

---
 api/client-server/v1/rooms.yaml | 41 +++++++++++++++------------------
 1 file changed, 19 insertions(+), 22 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 9a73bb05..c038dba5 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -43,9 +43,8 @@ paths:
         - in: path
           type: string
           name: stateKey
-          description: |-
-            The key of the state to look up. Defaults to the empty string.
-          required: false
+          description: The key of the state to look up. Defaults to the empty string.
+          required: true
           x-example: ""
       responses:
         200:
@@ -159,17 +158,17 @@ paths:
                     "user_id": "@alice:example.com"
                 }
               ]
-            schema:
-              type: array
-              title: RoomState
-              description: |-
-                If the user is a member of the room this will be the
-                current state of the room as a list of events.
-              items:
-                title: StateEvent
-                type: object
-                allOf:
-                  - "$ref": "definitions/state_event.yaml"
+          schema:
+            type: array
+            title: RoomState
+            description: |-
+              If the user is a member of the room this will be the
+              current state of the room as a list of events.
+            items:
+              title: StateEvent
+              type: object
+              allOf:
+                - "$ref": "events/core/state_event.json"
         403:
           description: You are not joined to the room.
 
@@ -345,7 +344,7 @@ paths:
                       type: object
                       title: RoomEvent
                       allOf:
-                        - "$ref": "definitions/room_event.yaml"
+                        - "$ref": "events/core/room_event.json"
                 required: ["start", "end", "chunk"]
               state:
                 type: array
@@ -358,7 +357,7 @@ paths:
                   title: StateEvent
                   type: object
                   allOf:
-                    - "$ref": "definitions/state_event.yaml"
+                    - "$ref": "events/core/state_event.json"
               visibility:
                 type: string
                 enum: ["private", "public"]
@@ -377,7 +376,7 @@ paths:
           type: string
           name: roomId
           description: The room to get the member events for.
-          required: true,
+          required: true
           x-example: "!room:example.com"
       responses:
         200:
@@ -419,15 +418,13 @@ paths:
                 ]
               }
           schema:
-            object:
-            properities:
+            type: object
+            properties:
               chunk:
                 type: array
                 items:
                   title: MemberEvent
                   type: object
                   allOf:
-                    - "$ref": "../../event-schemas/schema/v1/m.room.member"
-              visibility:
-                type: string
+                    - "$ref": "events/m.room.member"
 

From faa95e172f0ec7f1f40416b1df094380a72f2c98 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 17:49:57 +0100
Subject: [PATCH 095/467] More formatting fixes; typos; etc

---
 specification/22_voip_events.rst             |  1 +
 specification/25_application_service_api.rst | 11 ++--
 specification/30_server_server_api.rst       | 55 ++++++++++----------
 specification/31_event_signing.rst           |  2 +-
 specification/40_content_repository.rst      |  8 +--
 specification/43_push_cs_api.rst             | 20 +++----
 6 files changed, 50 insertions(+), 47 deletions(-)

diff --git a/specification/22_voip_events.rst b/specification/22_voip_events.rst
index d75db7e7..a5468237 100644
--- a/specification/22_voip_events.rst
+++ b/specification/22_voip_events.rst
@@ -63,3 +63,4 @@ The call setup should appear seamless to the user as if they had simply placed
 a call and the other party had accepted. Thusly, any media stream that had been
 setup for use on a call should be transferred and used for the call that
 replaces it.
+
diff --git a/specification/25_application_service_api.rst b/specification/25_application_service_api.rst
index 8236b2de..9b58c861 100644
--- a/specification/25_application_service_api.rst
+++ b/specification/25_application_service_api.rst
@@ -66,13 +66,13 @@ An example HS configuration required to pass traffic to the AS is:
   application service is merely augmenting the room itself (e.g. providing
   logging or searching facilities).
 - Namespaces are represented by POSIX extended regular expressions, 
-  e.g.:
+  e.g:
 
 .. code-block:: yaml
 
    users:
      - exclusive: true
-       regex: @irc.freenode.net/.*
+       regex: @irc.freenode.net_.*
 
 
 Home Server -> Application Service API
@@ -326,7 +326,7 @@ but only if the application service has defined the namespace as ``exclusive``.
 
 ID conventions
 ~~~~~~~~~~~~~~
-.. NOTE::
+.. TODO-spec
   - Giving HSes the freedom to namespace still feels like the Right Thing here.
   - Exposing a public API provides the consistency which was the main complaint
     against namespacing.
@@ -345,7 +345,7 @@ types, including:
 - MSISDNs (``tel``)
 - Email addresses (``mailto``)
 - IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04)
-- XMPP (xep-0032)
+- XMPP (XEP-0032)
 - SIP URIs (RFC 3261)
 
 As a result, virtual user IDs SHOULD relate to their URI counterpart. This
@@ -403,6 +403,8 @@ blog comment traffic in & out of matrix
 
 Active Application Services
 ----------------------------
+.. NOTE::
+  This section is a work in progress.
 
 .. TODO-spec
   API that provides hooks into the server so that you can intercept and
@@ -419,3 +421,4 @@ Policy Servers
 
 Enforcing policies
 ------------------
+
diff --git a/specification/30_server_server_api.rst b/specification/30_server_server_api.rst
index 1c7bf3ed..f5cadabf 100644
--- a/specification/30_server_server_api.rst
+++ b/specification/30_server_server_api.rst
@@ -2,10 +2,9 @@ Federation API
 ==============
 
 Matrix home servers use the Federation APIs (also known as server-server APIs)
-to communicate with each other.
-Home servers use these APIs to push messages to each other in real-time, to
-request historic messages from each other, and to query profile and presence
-information about users on each other's servers.
+to communicate with each other. Home servers use these APIs to push messages to
+each other in real-time, to request historic messages from each other, and to
+query profile and presence information about users on each other's servers.
 
 The APIs are implemented using HTTPS GETs and PUTs between each of the
 servers. These HTTPS requests are strongly authenticated using public key
@@ -21,7 +20,7 @@ Persisted Data Units (PDUs):
     context.
 
     Like email, it is the responsibility of the originating server of a PDU
-    to deliver that event to its recepient servers. However PDUs are signed
+    to deliver that event to its recipient servers. However PDUs are signed
     using the originating server's public key so that it is possible to
     deliver them through third-party servers.
 
@@ -84,18 +83,19 @@ directly or by querying an intermediate notary server using a
 response with their own key. A server may query multiple notary servers to
 ensure that they all report the same public keys.
 
-This approach is borrowed from the Perspectives Project
-(http://perspectives-project.org/), but modified to include the NACL keys and to
-use JSON instead of XML. It has the advantage of avoiding a single trust-root
-since each server is free to pick which notary servers they trust and can
-corroborate the keys returned by a given notary server by querying other
-servers.
+This approach is borrowed from the `Perspectives Project`_, but modified to
+include the NACL keys and to use JSON instead of XML. It has the advantage of
+avoiding a single trust-root since each server is free to pick which notary
+servers they trust and can corroborate the keys returned by a given notary
+server by querying other servers.
+
+.. _Perspectives Project: http://perspectives-project.org/
 
 Publishing Keys
 _______________
 
 Home servers publish the allowed TLS fingerprints and signing keys in a JSON
-object at ``/_matrix/key/v2/server/${key_id}``. The response contains a list of
+object at ``/_matrix/key/v2/server/{key_id}``. The response contains a list of
 ``verify_keys`` that are valid for signing federation requests made by the
 server and for signing events. It contains a list of ``old_verify_keys``
 which are only valid for signing events. Finally the response contains a list
@@ -510,7 +510,7 @@ To backfill events on a given context::
 
 Retrieves a sliding-window history of previous PDUs that occurred on the given
 context. Starting from the PDU ID(s) given in the "v" argument, the PDUs that
-preceeded it are retrieved, up to a total number given by the "limit" argument.
+preceded it are retrieved, up to a total number given by the "limit" argument.
 These are then returned in a new Transaction containing all of the PDUs.
 
 
@@ -554,9 +554,7 @@ Every HTTP request made by a homeserver is authenticated using public key
 digital signatures. The request method, target and body are signed by wrapping
 them in a JSON object and signing it using the JSON signing algorithm. The
 resulting signatures are added as an Authorization header with an auth scheme
-of X-Matrix.
-
-Note that the target field should include the full path starting with
+of X-Matrix. Note that the target field should include the full path starting with
 ``/_matrix/...``, including the ``?`` and any query parameters if present, but
 should not include the leading ``https:``, nor the destination server's
 hostname.
@@ -656,12 +654,12 @@ State Conflict Resolution
   - How does this work with deleting current state
   - How do we reject invalid federation traffic?
 
-[[TODO(paul): At this point we should probably have a long description of how
-State management works, with descriptions of clobbering rules, power levels, etc
-etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
-so on. This part needs refining. And writing in its own document as the details
-relate to the server/system as a whole, not specifically to server-server
-federation.]]
+  [[TODO(paul): At this point we should probably have a long description of how
+  State management works, with descriptions of clobbering rules, power levels, etc
+  etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
+  so on. This part needs refining. And writing in its own document as the details
+  relate to the server/system as a whole, not specifically to server-server
+  federation.]]
 
 Presence
 --------
@@ -677,8 +675,8 @@ Performing a presence update and poll subscription request::
       Each should be an object with the following keys:
         user_id: string containing a User ID
         presence: "offline"|"unavailable"|"online"|"free_for_chat"
-        status_msg: (optional) string of freeform text
-        last_active_ago: miliseconds since the last activity by the user
+        status_msg: (optional) string of free-form text
+        last_active_ago: milliseconds since the last activity by the user
 
     poll: (optional): list of strings giving User IDs
 
@@ -696,7 +694,7 @@ removed until explicitly requested by a later ``unpoll``.
 On receipt of a message containing a non-empty ``poll`` list, the receiving
 server should immediately send the sending server a presence update EDU of its
 own, containing in a ``push`` list the current state of every user that was in
-the orginal EDU's ``poll`` list.
+the original EDU's ``poll`` list.
 
 Sending a presence invite::
 
@@ -721,7 +719,7 @@ Rejecting a presence invite::
   Content keys - as for m.presence_invite
 
 .. TODO-doc
-  - Explain the timing-based roundtrip reduction mechanism for presence
+  - Explain the timing-based round-trip reduction mechanism for presence
     messages
   - Explain the zero-byte presence inference logic
   See also: docs/client-server/model/presence
@@ -742,8 +740,8 @@ Querying profile information::
     field: (optional) string giving a field name
 
   Returns: JSON object containing the following keys:
-    displayname: string of freeform text
-    avatar_url: string containing an http-scheme URL
+    displayname: string of free-form text
+    avatar_url: string containing an HTTP-scheme URL
 
 If the query contains the optional ``field`` key, it should give the name of a
 result field. If such is present, then the result should contain only a field
@@ -769,3 +767,4 @@ Querying directory information::
 The list of join candidates is a list of server names that are likely to hold
 the given room; these are servers that the requesting server may wish to try
 joining with. This list may or may not include the server answering the query.
+
diff --git a/specification/31_event_signing.rst b/specification/31_event_signing.rst
index 9351cd57..f3a22d1d 100644
--- a/specification/31_event_signing.rst
+++ b/specification/31_event_signing.rst
@@ -30,7 +30,7 @@ using this representation.
          value,
          # Encode code-points outside of ASCII as UTF-8 rather than \u escapes
          ensure_ascii=False,
-         # Remove unecessary white space.
+         # Remove unnecessary white space.
          separators=(',',':'),
          # Sort the keys of dictionaries.
          sort_keys=True,
diff --git a/specification/40_content_repository.rst b/specification/40_content_repository.rst
index df04605a..2c45ced7 100644
--- a/specification/40_content_repository.rst
+++ b/specification/40_content_repository.rst
@@ -39,10 +39,10 @@ thumbnailing method::
 
        <thumbnail>
 
-The thumbnail methods are "crop" and "scale". "scale" trys to return an
+The thumbnail methods are "crop" and "scale". "scale" tries to return an
 image where either the width or the height is smaller than the requested
 size. The client should then scale and letterbox the image if it needs to
-fit within a given rectangle. "crop" trys to return an image where the
+fit within a given rectangle. "crop" tries to return an image where the
 width and height are close to the requested size and the aspect matches
 the requested size. The client should scale the image if it needs to fit
 within a given rectangle.
@@ -53,8 +53,8 @@ the content. Homeservers may return thumbnails of a different size to that
 requested. However homeservers should provide exact matches where reasonable.
 Homeservers must never upscale images.
 
-Security
---------
+Security considerations
+-----------------------
 
 Clients may try to upload very large files. Homeservers should not store files
 that are too large and should not serve them to clients.
diff --git a/specification/43_push_cs_api.rst b/specification/43_push_cs_api.rst
index 022f50c3..b78b44fd 100644
--- a/specification/43_push_cs_api.rst
+++ b/specification/43_push_cs_api.rst
@@ -70,7 +70,7 @@ Room Rules
 Sender
   These rules configure notification behaviour for messages from a specific,
   named Matrix user ID. The rule_id of Sender rules is always the Matrix user
-  ID of the user whose messages theyt apply to.
+  ID of the user whose messages they'd apply to.
 Underride
   These are identical to override rules, but have a lower priority than content,
   room and sender rules.
@@ -112,7 +112,7 @@ In addition, all rules may be enabled or disabled. Disabled rules never match.
 
 If no rules match an event, the Home Server should not notify for the message
 (that is to say, the default action is "dont-notify"). Events that the user sent
-themself are never alerted for.
+themselves are never alerted for.
 
 Predefined Rules
 ----------------
@@ -128,7 +128,7 @@ with these IDs, their semantics should match those given below:
 
     {
         "rule_id": ".m.rule.contains_user_name"
-        "pattern": "[the lcoal part of the user's Matrix ID]",
+        "pattern": "[the local part of the user's Matrix ID]",
         "actions": [
             "notify",
             {
@@ -220,7 +220,7 @@ with these IDs, their semantics should match those given below:
 Push Rules: Actions:
 --------------------
 All rules have an associated list of 'actions'. An action affects if and how a
-notification is delievered for a matching event. This standard defines the
+notification is delivered for a matching event. This standard defines the
 following actions, although if Home servers wish to support more, they are free
 to do so:
 
@@ -241,11 +241,11 @@ set_tweak
 
 Actions that have no parameters are represented as a string. Otherwise, they are
 represented as a dictionary with a key equal to their name and other keys as
-their parameters, eg. { "set_tweak": "sound", "value": "default" }
+their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
 
 Push Rules: Actions: Tweaks
 ---------------------------
-The 'set_tweak' key action is used to add an entry to the 'tweaks' dictionary
+The ``set_tweak`` key action is used to add an entry to the 'tweaks' dictionary
 that is sent in the notification poke. The following tweaks are defined:
 
 sound
@@ -275,7 +275,7 @@ do so:
 
 event_match
   This is a glob pattern match on a field of the event. Parameters:
-   * 'key': The dot-separated field of the event to match, eg. content.body
+   * 'key': The dot-separated field of the event to match, e.g. content.body
    * 'pattern': The glob-style pattern to match against. Patterns with no
                 special glob characters should be treated as having asterisks
                 prepended and appended when testing the condition.
@@ -295,7 +295,7 @@ room_member_count
      '>=' or '<='. A prefix of '<' matches rooms where the member count is
      strictly less than the given number and so forth. If no prefix is present,
      this matches rooms where the member count is exactly equal to the given
-     number (ie. the same as '==').
+     number (i.e. the same as '==').
 
 Room, Sender, User and Content rules do not have conditions in the same way,
 but instead have predefined conditions, the behaviour of which can be configured
@@ -314,7 +314,7 @@ scope
   Either 'global' or 'device/<profile_tag>' to specify global rules or
   device rules for the given profile_tag.
 kind
-  The kind of rule, ie. 'override', 'underride', 'sender', 'room', 'content'.
+  The kind of rule, i.e. 'override', 'underride', 'sender', 'room', 'content'.
 rule_id
   The identifier for the rule.
 
@@ -330,7 +330,7 @@ after
   rule.
 
 All requests to the push rules API also require an access_token as a query
-paraemter.
+parameter.
 
 The content of the PUT request is a JSON object with a list of actions under the
 'actions' key and either conditions (under the 'conditions' key) or the

From 620d3dcb266c432e1bc471ad63b6bed57354e46f Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 17:51:44 +0100
Subject: [PATCH 096/467] Test commit

---
 specification/44_push_push_gw_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/44_push_push_gw_api.rst b/specification/44_push_push_gw_api.rst
index b182503b..d7f16144 100644
--- a/specification/44_push_push_gw_api.rst
+++ b/specification/44_push_push_gw_api.rst
@@ -1,7 +1,7 @@
 HTTP Notification Protocol
 --------------------------
 
-This describes the format used by "http" pushers to send notifications of
+This describes the format used by "HTTP" pushers to send notifications of
 events.
 
 Notifications are sent as HTTP POST requests to the URL configured when the

From 3f9d183c2aaaf5f6d2f6c9bc39a57317145dca1b Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 09:25:40 +0100
Subject: [PATCH 097/467] More typo/layout fixes

Split out identity servers to a separate file
---
 specification/44_push_push_gw_api.rst     |  8 ++++----
 specification/45_typing_notifications.rst | 11 +++++++----
 specification/46_receipts.rst             | 18 +++++++-----------
 specification/49_other_non_core_apis.rst  |  4 ++--
 specification/50_appendices.rst           |  8 --------
 specification/60_identity_servers.rst     |  8 ++++++++
 6 files changed, 28 insertions(+), 29 deletions(-)
 create mode 100644 specification/60_identity_servers.rst

diff --git a/specification/44_push_push_gw_api.rst b/specification/44_push_push_gw_api.rst
index d7f16144..c970e93e 100644
--- a/specification/44_push_push_gw_api.rst
+++ b/specification/44_push_push_gw_api.rst
@@ -77,10 +77,10 @@ counts
   This is a dictionary of the current number of unacknowledged communications
   for the recipient user. Counts whose value is zero are omitted.
 unread
-  The number of unread messages a user has accross all of the rooms they are a
+  The number of unread messages a user has across all of the rooms they are a
   member of.
 missed_calls
-  The number of unacknowledged missed calls a user has accross all rooms of
+  The number of unacknowledged missed calls a user has across all rooms of
   which they are a member.
 device
   This is an array of devices that the notification should be sent to.
@@ -104,13 +104,13 @@ And additional key is defined but only present on member events:
 
 user_is_target
   This is true if the user receiving the notification is the subject of a member
-  event (ie. the state_key of the member event is equal to the user's Matrix
+  event (i.e. the state_key of the member event is equal to the user's Matrix
   ID).
 
 The recipient of an HTTP notification should respond with an HTTP 2xx response
 when the notification has been processed. If the endpoint returns an HTTP error
 code, the Home Server should retry for a reasonable amount of time with a
-reasonable backoff scheme.
+reasonable back-off scheme.
 
 The endpoint should return a JSON dictionary as follows::
 
diff --git a/specification/45_typing_notifications.rst b/specification/45_typing_notifications.rst
index 7df9a238..89f06b48 100644
--- a/specification/45_typing_notifications.rst
+++ b/specification/45_typing_notifications.rst
@@ -5,9 +5,10 @@ Client APIs
 -----------
 
 To set "I am typing for the next N msec"::
+
   PUT .../rooms/<room_id>/typing/<user_id>
   Content:  { "typing": true, "timeout": N }
-  # timeout is in msec; I suggest no more than 20 or 30 seconds
+  # timeout is in milliseconds; suggested no more than 20 or 30 seconds
 
 This should be re-sent by the client to continue informing the server the user
 is still typing; I suggest a safety margin of 5 seconds before the expected
@@ -15,6 +16,7 @@ timeout runs out. Just keep declaring a new timeout, it will replace the old
 one.
 
 To set "I am no longer typing"::
+
   PUT ../rooms/<room_id>/typing/<user_id>
   Content: { "typing": false }
 
@@ -46,13 +48,14 @@ Servers will emit EDUs in the following form::
     "content": {
       "room_id": "!room-id-here:matrix.org",
       "user_id": "@user-id-here:matrix.org",
-      "typing": true/false,
+      "typing": true/false
     }
   }
 
 Server EDUs don't (currently) contain timing information; it is up to
 originating HSes to ensure they eventually send "stop" notifications.
 
-((This will eventually need addressing, as part of the wider typing/presence
-timer addition work))
+.. TODO
+  ((This will eventually need addressing, as part of the wider typing/presence
+  timer addition work))
 
diff --git a/specification/46_receipts.rst b/specification/46_receipts.rst
index 6428d6b5..a0a90c16 100644
--- a/specification/46_receipts.rst
+++ b/specification/46_receipts.rst
@@ -2,9 +2,8 @@ Receipts
 ========
 
 Receipts are used to publish which events in a room the user or their devices
-have interacted with. For example, which events the user has read.
-
-For efficiency this is done as "up to" markers, i.e. marking a particular event
+have interacted with. For example, which events the user has read. For
+efficiency this is done as "up to" markers, i.e. marking a particular event
 as, say, ``read`` indicates the user has read all events *up to* that event.
 
 Client-Server API
@@ -43,14 +42,11 @@ For example::
     }
 
 For efficiency, receipts are batched into one event per room. In the initialSync
-and v2 sync APIs the receipts are listed in a seperate top level ``receipts``
-key.
-
-Each ``user_id``, ``receipt_type`` pair must be associated with only a single
-``event_id``.
-
-New receipts that come down the event streams are deltas. Deltas update
-existing mappings, clobbering based on ``user_id``, ``receipt_type`` pairs.
+and v2 sync APIs the receipts are listed in a separate top level ``receipts``
+key. Each ``user_id``, ``receipt_type`` pair must be associated with only a
+single ``event_id``. New receipts that come down the event streams are deltas.
+Deltas update existing mappings, clobbering based on ``user_id``,
+``receipt_type`` pairs.
 
 
 A client can update the markers for its user by issuing a request::
diff --git a/specification/49_other_non_core_apis.rst b/specification/49_other_non_core_apis.rst
index 8855ce9e..d6315a96 100644
--- a/specification/49_other_non_core_apis.rst
+++ b/specification/49_other_non_core_apis.rst
@@ -3,10 +3,10 @@ Address book repository
 
 .. NOTE::
   This section is a work in progress.
-  Do we even need it?  Clients can use out-of-band addressbook servers for now;
-  this should definitely not be core.
 
 .. TODO-spec
+  Do we even need it?  Clients can use out-of-band addressbook servers for now;
+  this should definitely not be core.
   - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET.
   - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of
     contacts), etc.
diff --git a/specification/50_appendices.rst b/specification/50_appendices.rst
index 295c8f69..de1ac290 100644
--- a/specification/50_appendices.rst
+++ b/specification/50_appendices.rst
@@ -128,11 +128,3 @@ Threat: Disclosure to Servers Within Chatroom
 An attacker could take control of a server within a chatroom to expose message
 contents or metadata for messages in that room.
 
-
-Identity Servers
-================
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-doc Dave
-  - 3PIDs and identity server, functions
diff --git a/specification/60_identity_servers.rst b/specification/60_identity_servers.rst
new file mode 100644
index 00000000..6ec013bd
--- /dev/null
+++ b/specification/60_identity_servers.rst
@@ -0,0 +1,8 @@
+Identity Servers
+================
+.. NOTE::
+  This section is a work in progress.
+
+.. TODO-doc Dave
+  - 3PIDs and identity server, functions
+

From 8e7b33ac9989c985ceac1cf7fccddd84c1346ec8 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 09:40:53 +0100
Subject: [PATCH 098/467] Fix more typos/spelling errors

---
 specification/00_basis.rst             | 13 ++++++-----
 specification/10_client_server_api.rst | 32 +++++++++++++-------------
 2 files changed, 23 insertions(+), 22 deletions(-)

diff --git a/specification/00_basis.rst b/specification/00_basis.rst
index f6d50802..384d583b 100644
--- a/specification/00_basis.rst
+++ b/specification/00_basis.rst
@@ -79,7 +79,7 @@ The functionality that Matrix provides includes:
 - Extensible user management (inviting, joining, leaving, kicking, banning)
   mediated by a power-level based user privilege system.
 - Extensible room state management (room naming, aliasing, topics, bans)
-- Extensible user profile management (avatars, displaynames, etc)
+- Extensible user profile management (avatars, display names, etc)
 - Managing user accounts (registration, login, logout)
 - Use of 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
   Facebook accounts to authenticate, identify and discover users on Matrix.
@@ -91,7 +91,7 @@ The functionality that Matrix provides includes:
 The end goal of Matrix is to be a ubiquitous messaging layer for synchronising
 arbitrary data between sets of people, devices and services - be that for
 instant messages, VoIP call setups, or any other objects that need to be
-reliably and persistently pushed from A to B in an interoperable and federated
+reliably and persistently pushed from A to B in an inter-operable and federated
 manner.
 
 Overview
@@ -185,7 +185,7 @@ Event Graphs
 Events exchanged in the context of a room are stored in a directed acyclic graph
 (DAG) called an ``event graph``. The partial ordering of this graph gives the
 chronological ordering of events within the room. Each event in the graph has a
-list of zero or more ``parent`` events, which refer to any preceeding events
+list of zero or more ``parent`` events, which refer to any preceding events
 which have no chronological successor from the perspective of the homeserver
 which created the event.
 
@@ -368,7 +368,8 @@ room). An example of a non-proactive client activity would be a client setting
 key called ``last_active_ago``, which gives the relative number of milliseconds
 since the message is generated/emitted that the user was last seen active.
 
-N.B. in v1 API, status/online/idle state are muxed into a single 'presence' field on the m.presence event.
+N.B. in v1 API, status/online/idle state are muxed into a single 'presence'
+field on the ``m.presence`` event.
 
 Presence Lists
 ~~~~~~~~~~~~~~
@@ -386,7 +387,7 @@ Profiles
 
 Users may publish arbitrary key/value data associated with their account - such
 as a human readable ``display name``, a profile photo URL, contact information
-(email address, phone nubers, website URLs etc).
+(email address, phone numbers, website URLs etc).
 
 In Client-Server API v2, profile data is typed using namespaced keys for
 interoperability, much like events - e.g. ``m.profile.display_name``.
@@ -443,7 +444,7 @@ The ``error`` string will be a human-readable error message, usually a sentence
 explaining what went wrong. The ``errcode`` string will be a unique string
 which can be used to handle an error message e.g. ``M_FORBIDDEN``. These error
 codes should have their namespace first in ALL CAPS, followed by a single _ to
-ease seperating the namespace from the error code.. For example, if there was a
+ease separating the namespace from the error code.. For example, if there was a
 custom namespace ``com.mydomain.here``, and a
 ``FORBIDDEN`` code, the error code should look like
 ``COM.MYDOMAIN.HERE_FORBIDDEN``. There may be additional keys depending on the
diff --git a/specification/10_client_server_api.rst b/specification/10_client_server_api.rst
index f0416c02..ef77a7dc 100644
--- a/specification/10_client_server_api.rst
+++ b/specification/10_client_server_api.rst
@@ -8,11 +8,11 @@ The client-server API provides a simple lightweight API to let clients send
 messages, control rooms and synchronise conversation history. It is designed to
 support both lightweight clients which store no state and lazy-load data from
 the server as required - as well as heavyweight clients which maintain a full
-local peristent copy of server state.
+local persistent copy of server state.
 
 This mostly describes v1 of the Client-Server API as featured in the original September
 2014 launch of Matrix, apart from user-interactive authentication where it is
-encouraged to move to V2, therefore this is the version documented here.
+encouraged to move to v2, therefore this is the version documented here.
 Version 2 is currently in development (as of Jan-March 2015) as an incremental
 but backwards-incompatible refinement of Version 1 and will be released
 shortly.
@@ -154,7 +154,7 @@ Matrix client, for example, an email confirmation may be completed when the user
 clicks on the link in the email. In this case, the client retries the request
 with an auth dict containing only the session key. The response to this will be
 the same as if the client were attempting to complete an auth state normally,
-ie. the request will either complete or request auth, with the presence or
+i.e. the request will either complete or request auth, with the presence or
 absence of that login stage type in the 'completed' array indicating whether
 that stage is complete.
 
@@ -204,7 +204,7 @@ Password-based
 :Type:
   ``m.login.password``
 :Description:
-  The client submits a username and secret password, both sent in plaintext.
+  The client submits a username and secret password, both sent in plain-text.
 
 To respond to this type, reply with an auth dict as follows::
 
@@ -247,10 +247,10 @@ service which the home server accepts when logging in, this indirection can be
 skipped and the "uri" key can be the ``Authorization Request URI``.
 
 The client then visits the ``Authorization Request URI``, which then shows the
-OAuth2 Allow/Deny prompt. Hitting 'Allow' returns the [XXX: redirects to the?]``redirect URI`` with the
-auth code. Home servers can choose any path for the ``redirect URI``. Once the
-OAuth flow has completed, the client retries the request with the session only,
-as above.
+OAuth2 Allow/Deny prompt. Hitting 'Allow' redirects to the ``redirect URI`` with
+the auth code. Home servers can choose any path for the ``redirect URI``. Once
+the OAuth flow has completed, the client retries the request with the session
+only, as above.
 
 Email-based (identity server)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -308,7 +308,7 @@ Where ``stage type`` is the type name of the stage it is attempting and
 ``session id`` is the ID of the session given by the home server.
 
 This MUST return an HTML page which can perform this authentication stage. This
-page must attempt to call the Javascript function ``window.onAuthDone`` when
+page must attempt to call the JavaScript function ``window.onAuthDone`` when
 the authentication has been completed.
 
 Pagination
@@ -373,7 +373,7 @@ now show page 3 (rooms R11 -> 15)::
                              Returns: R11,R12,R13,R14,R15
                          
 Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 
-token from the intial request was '9' which corresponded to R10. When the 2nd
+token from the initial request was '9' which corresponded to R10. When the 2nd
 request was made, R10 did not appear again, even though from=9 was specified. If
 you know the token, you already have the data.
 
@@ -425,9 +425,9 @@ You can visualise the range of events being returned as::
                               |                             |
                         start: '1-2-3'                end: 'a-b-c'
                              
-Now, to receive future events in realtime on the eventstream, you simply GET
+Now, to receive future events in real-time on the eventstream, you simply GET
 $PREFIX/events with a ``from`` parameter of 'a-b-c': in other words passing in the
-``end`` token returned by initialsync. The request blocks until new events are
+``end`` token returned by initial sync. The request blocks until new events are
 available or until your specified timeout elapses, and then returns a
 new paginatable chunk of events alongside new start and end parameters::
 
@@ -467,7 +467,7 @@ event stream. When the request returns, an ``end`` token is included in the
 response. This token can be used in the next request to continue where the
 last request left off.
 
-All events must be deduplicated based on their event ID.
+All events must be de-duplicated based on their event ID.
 
 .. TODO
   is deduplication actually a hard requirement in CS v2?
@@ -493,7 +493,7 @@ Room events are split into two categories:
 
 :Message events:
   These are events which describe transient "once-off" activity in a room:
-  typically communication such as sending an instant messaage or setting up a
+  typically communication such as sending an instant message or setting up a
   VoIP call. These used to be called 'non-state' events.
 
 This specification outlines several events, all with the event type prefix
@@ -971,7 +971,7 @@ Registering for a user account is done using the request::
   POST $V2PREFIX/register
 
 This API endpoint uses the User-Interactive Authentication API.
-This API endoint does not require an access token.
+This API endpoint does not require an access token.
 
 The body of the POST request is a JSON object containing:
 
@@ -1059,7 +1059,7 @@ The third party identifier credentials object comprises:
 
 id_server
   The colon-separated hostname and port of the Identity Server used to
-  authenticate the third party identifer. If the port is the default, it and the
+  authenticate the third party identifier. If the port is the default, it and the
   colon should be omitted.
 sid
   The session ID given by the Identity Server

From ad26b7f8cb89b2a32e009bdd1d061ea6b7887b9d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 10:03:58 +0100
Subject: [PATCH 099/467] Fix a subtle rendering bug when catting spec sections

Throw in gendoc.py if a spec section doesn't end with \n\n

There needs to be TWO new lines at the end of each spec section else the
title of the next section merges into the last paragraph of the earlier
section. This happens without rst2html producing a warning, and results
in the section heading of a file disappearing(!)
---
 scripts/gendoc.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index ed172726..e871055e 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -18,7 +18,14 @@ def glob_spec_to(out_file_name):
     with open(out_file_name, "wb") as outfile:
         for f in sorted(glob.glob("../specification/*.rst")):
             with open(f, "rb") as infile:
-                outfile.write(infile.read())
+                section = infile.read()
+                # we need TWO new lines else the next file's title gets merged
+                # the last paragraph *WITHOUT RST PRODUCING A WARNING*
+                if not section[-2:] == '\n\n':
+                    raise Exception(
+                        "The file " + f + " does not end with 2 new lines."
+                    )
+                outfile.write(section)
 
 
 def rst2html(i, o):

From 08defafd98727b0011dddd6c0e8e271f61d147b2 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 18 Sep 2015 10:26:13 +0100
Subject: [PATCH 100/467] Print errors to stderr

This means that continuserv will actually display them
---
 scripts/gendoc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index ed172726..0e9626af 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -49,7 +49,7 @@ def run_through_template(input):
             )
     except subprocess.CalledProcessError as e:
         with open(tmpfile, 'r') as f:
-            print f.read()
+            sys.stderr.write(f.read() + "\n")
         raise
 
 def prepare_env():

From ef473b4161d029b35ef193309c9b4ad23f43e4f1 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:05:48 +0100
Subject: [PATCH 101/467] Start enforcing title styles

This depends on the number of the file.
---
 scripts/gendoc.py | 55 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 47 insertions(+), 8 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index e871055e..6bf7c1a6 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -4,6 +4,7 @@ from docutils.core import publish_file
 import fileinput
 import glob
 import os
+import re
 import shutil
 import subprocess
 import sys
@@ -14,17 +15,55 @@ stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]
 }
 
-def glob_spec_to(out_file_name):
+title_style_matchers = {
+    "=": re.compile("^=+$"),
+    "-": re.compile("^-+$")
+}
+TOP_LEVEL = "="
+SECOND_LEVEL = "-"
+
+
+def check_valid_section(filename, section):
+    # we need TWO new lines else the next file's title gets merged
+    # the last paragraph *WITHOUT RST PRODUCING A WARNING*
+    if not section[-2:] == '\n\n':
+        raise Exception(
+            "The file " + filename + " does not end with 2 new lines."
+        )
+
+    # Enforce some rules to reduce the risk of having mismatched title
+    # styles.
+    title_line = section.split("\n")[1]
+    if title_line != (len(title_line) * title_line[0]):
+        raise Exception(
+            "The file " + filename + " doesn't have a title style line on line 2"
+        )
+
+    # anything marked as x0_ is the start of a new top-level section
+    if re.match("^[0-9]+0_", filename):
+        if not title_style_matchers[TOP_LEVEL].match(title_line):
+            raise Exception(
+                "The file " + filename + " is a top-level section because it matches " +
+                "the filename format x0_something.rst but has the wrong title " +
+                "style: expected '" + TOP_LEVEL + "' but got '" +
+                title_line[0] + "'"
+            )
+    # anything marked as xx_ is the start of a sub-section
+    elif re.match("^[0-9]+_", filename):
+        if not title_style_matchers[SECOND_LEVEL].match(title_line):
+            raise Exception(
+                "The file " + filename + " is a 2nd-level section because it matches " +
+                "the filename format xx_something.rst but has the wrong title " +
+                "style: expected '" + SECOND_LEVEL + "' but got '" +
+                title_line[0] + "'"
+            )
+
+def cat_spec_sections_to(out_file_name):
     with open(out_file_name, "wb") as outfile:
         for f in sorted(glob.glob("../specification/*.rst")):
             with open(f, "rb") as infile:
                 section = infile.read()
-                # we need TWO new lines else the next file's title gets merged
-                # the last paragraph *WITHOUT RST PRODUCING A WARNING*
-                if not section[-2:] == '\n\n':
-                    raise Exception(
-                        "The file " + f + " does not end with 2 new lines."
-                    )
+                check_valid_section(f.split("/")[-1], section)
                 outfile.write(section)
 
 
@@ -74,7 +113,7 @@ def cleanup_env():
 
 def main():
     prepare_env()
-    glob_spec_to("tmp/full_spec.rst")
+    cat_spec_sections_to("tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
     run_through_template("tmp/howto.rst")

From 9ed39665c8f70d9c704738d07a39877267e0a4da Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:09:04 +0100
Subject: [PATCH 102/467] PR feedback

---
 specification/00_basis.rst                | 2 +-
 specification/45_typing_notifications.rst | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/specification/00_basis.rst b/specification/00_basis.rst
index 384d583b..a5196e99 100644
--- a/specification/00_basis.rst
+++ b/specification/00_basis.rst
@@ -444,7 +444,7 @@ The ``error`` string will be a human-readable error message, usually a sentence
 explaining what went wrong. The ``errcode`` string will be a unique string
 which can be used to handle an error message e.g. ``M_FORBIDDEN``. These error
 codes should have their namespace first in ALL CAPS, followed by a single _ to
-ease separating the namespace from the error code.. For example, if there was a
+ease separating the namespace from the error code. For example, if there was a
 custom namespace ``com.mydomain.here``, and a
 ``FORBIDDEN`` code, the error code should look like
 ``COM.MYDOMAIN.HERE_FORBIDDEN``. There may be additional keys depending on the
diff --git a/specification/45_typing_notifications.rst b/specification/45_typing_notifications.rst
index 89f06b48..ff00165d 100644
--- a/specification/45_typing_notifications.rst
+++ b/specification/45_typing_notifications.rst
@@ -11,9 +11,9 @@ To set "I am typing for the next N msec"::
   # timeout is in milliseconds; suggested no more than 20 or 30 seconds
 
 This should be re-sent by the client to continue informing the server the user
-is still typing; I suggest a safety margin of 5 seconds before the expected
-timeout runs out. Just keep declaring a new timeout, it will replace the old
-one.
+is still typing; a safety margin of 5 seconds before the expected
+timeout runs out is recommended. Just keep declaring a new timeout, it will
+replace the old one.
 
 To set "I am no longer typing"::
 

From 06e46c1899755d9ae46b66a8b316b4c41b664328 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:20:34 +0100
Subject: [PATCH 103/467] Fix on a better numbering format

---
 scripts/gendoc.py | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 6bf7c1a6..1348679a 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -21,9 +21,16 @@ title_style_matchers = {
 }
 TOP_LEVEL = "="
 SECOND_LEVEL = "-"
+FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}_.*\.rst$")
 
 
 def check_valid_section(filename, section):
+    if not re.match(FILE_FORMAT_MATCHER, filename):
+        raise Exception(
+            "The filename of " + filename +" does not match the expected format " +
+            "of '##_##_words-go-here.rst'"
+        )
+
     # we need TWO new lines else the next file's title gets merged
     # the last paragraph *WITHOUT RST PRODUCING A WARNING*
     if not section[-2:] == '\n\n':
@@ -39,21 +46,21 @@ def check_valid_section(filename, section):
             "The file " + filename + " doesn't have a title style line on line 2"
         )
 
-    # anything marked as x0_ is the start of a new top-level section
-    if re.match("^[0-9]+0_", filename):
+    # anything marked as xx_00_ is the start of a new top-level section
+    if re.match("^[0-9]+_00_", filename):
         if not title_style_matchers[TOP_LEVEL].match(title_line):
             raise Exception(
                 "The file " + filename + " is a top-level section because it matches " +
-                "the filename format x0_something.rst but has the wrong title " +
+                "the filename format ##_00_something.rst but has the wrong title " +
                 "style: expected '" + TOP_LEVEL + "' but got '" +
                 title_line[0] + "'"
             )
-    # anything marked as xx_ is the start of a sub-section
-    elif re.match("^[0-9]+_", filename):
+    # anything marked as xx_x0_ is the start of a sub-section
+    elif re.match("^[0-9]+_0[0-9]{1}_", filename):
         if not title_style_matchers[SECOND_LEVEL].match(title_line):
             raise Exception(
                 "The file " + filename + " is a 2nd-level section because it matches " +
-                "the filename format xx_something.rst but has the wrong title " +
+                "the filename format ##_#0_something.rst but has the wrong title " +
                 "style: expected '" + SECOND_LEVEL + "' but got '" +
                 title_line[0] + "'"
             )

From 7f81501762cb700615de837b7501c6bde9deda11 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 11:39:44 +0100
Subject: [PATCH 104/467] Allow relative references to schema to work in python
 and node.

Rename "schema/v1/core" to "schema/v1/core-event-schema".
Add self-referential symlinks to schema/v1/core-event-schema

The python json schema libraries expect that relative references are
relative to the file they are in. The node json schema libraries
expect that relateive references are relative to the first file loaded.

To support both kinds we reference the core event schema using
"core-event-schema/event.json". We then symlink the core-event-schema
directory to both the location of the file refering to "event.json" so
that it will work in python and to the location of the top level file
so that it will work in node.
---
 api/client-server/v1/core                              |  1 -
 api/client-server/v1/core-event-schema                 |  1 +
 api/client-server/v1/presence.yaml                     |  2 +-
 api/client-server/v1/sync.yaml                         | 10 +++++-----
 api/client-server/v1/{events => v1-event-schema}       |  0
 .../schema/v1/core-event-schema/core-event-schema      |  1 +
 .../schema/v1/{core => core-event-schema}/event.json   |  0
 .../msgtype_infos/image_info.json                      |  0
 .../v1/{core => core-event-schema}/room_event.json     |  2 +-
 .../v1/{core => core-event-schema}/state_event.json    |  2 +-
 event-schemas/schema/v1/m.call.answer                  |  2 +-
 event-schemas/schema/v1/m.call.candidates              |  2 +-
 event-schemas/schema/v1/m.call.hangup                  |  2 +-
 event-schemas/schema/v1/m.call.invite                  |  2 +-
 event-schemas/schema/v1/m.room.aliases                 |  2 +-
 event-schemas/schema/v1/m.room.canonical_alias         |  2 +-
 event-schemas/schema/v1/m.room.create                  |  2 +-
 event-schemas/schema/v1/m.room.history_visibility      |  2 +-
 event-schemas/schema/v1/m.room.join_rules              |  2 +-
 event-schemas/schema/v1/m.room.member                  |  2 +-
 event-schemas/schema/v1/m.room.message                 |  2 +-
 event-schemas/schema/v1/m.room.message#m.audio         |  2 +-
 event-schemas/schema/v1/m.room.message#m.emote         |  2 +-
 event-schemas/schema/v1/m.room.message#m.file          |  4 ++--
 event-schemas/schema/v1/m.room.message#m.image         |  4 ++--
 event-schemas/schema/v1/m.room.message#m.location      |  4 ++--
 event-schemas/schema/v1/m.room.message#m.notice        |  2 +-
 event-schemas/schema/v1/m.room.message#m.text          |  2 +-
 event-schemas/schema/v1/m.room.message#m.video         |  4 ++--
 event-schemas/schema/v1/m.room.message.feedback        |  2 +-
 event-schemas/schema/v1/m.room.name                    |  2 +-
 event-schemas/schema/v1/m.room.power_levels            |  2 +-
 event-schemas/schema/v1/m.room.redaction               |  2 +-
 event-schemas/schema/v1/m.room.topic                   |  2 +-
 event-schemas/schema/v1/v1-event-schema                |  1 +
 templating/matrix_templates/units.py                   |  2 +-
 36 files changed, 40 insertions(+), 38 deletions(-)
 delete mode 120000 api/client-server/v1/core
 create mode 120000 api/client-server/v1/core-event-schema
 rename api/client-server/v1/{events => v1-event-schema} (100%)
 create mode 120000 event-schemas/schema/v1/core-event-schema/core-event-schema
 rename event-schemas/schema/v1/{core => core-event-schema}/event.json (100%)
 rename event-schemas/schema/v1/{core => core-event-schema}/msgtype_infos/image_info.json (100%)
 rename event-schemas/schema/v1/{core => core-event-schema}/room_event.json (93%)
 rename event-schemas/schema/v1/{core => core-event-schema}/state_event.json (93%)
 create mode 120000 event-schemas/schema/v1/v1-event-schema

diff --git a/api/client-server/v1/core b/api/client-server/v1/core
deleted file mode 120000
index 9fc0cd87..00000000
--- a/api/client-server/v1/core
+++ /dev/null
@@ -1 +0,0 @@
-events/core
\ No newline at end of file
diff --git a/api/client-server/v1/core-event-schema b/api/client-server/v1/core-event-schema
new file mode 120000
index 00000000..045aecb0
--- /dev/null
+++ b/api/client-server/v1/core-event-schema
@@ -0,0 +1 @@
+v1-event-schema/core-event-schema
\ No newline at end of file
diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 472a2d7f..717e9c47 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -205,4 +205,4 @@ paths:
               type: object
               title: PresenceEvent
               allOf:
-                - "$ref": "events/core/event.json"
+                - "$ref": "core-event-schema/event.json"
diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index b9525491..27c7073c 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -82,7 +82,7 @@ paths:
                   type: object
                   title: RoomEvent
                   allOf:
-                   - "$ref": "events/core/room_event.json"
+                   - "$ref": "core-event-schema/room_event.json"
         400:
           description: "Bad pagination ``from`` parameter."
   "/initialSync":
@@ -253,7 +253,7 @@ paths:
                   type: object
                   title: Event
                   allOf:
-                   - "$ref": "events/core/event.json"
+                   - "$ref": "core-event-schema/event.json"
               rooms:
                 type: array
                 items:
@@ -294,7 +294,7 @@ paths:
                             type: object
                             title: RoomEvent
                             allOf:
-                              - "$ref": "events/core/room_event.json"
+                              - "$ref": "core-event-schema/room_event.json"
                       required: ["start", "end", "chunk"]
                     state:
                       type: array
@@ -307,7 +307,7 @@ paths:
                         title: StateEvent
                         type: object
                         allOf:
-                          - "$ref": "events/core/state_event.json"
+                          - "$ref": "core-event-schema/state_event.json"
                     visibility:
                       type: string
                       enum: ["private", "public"]
@@ -350,6 +350,6 @@ paths:
               }
           schema:
             allOf:
-              - "$ref": "events/core/event.json"
+              - "$ref": "core-event-schema/event.json"
         404:
           description: The event was not found or you do not have permission to read this event.
diff --git a/api/client-server/v1/events b/api/client-server/v1/v1-event-schema
similarity index 100%
rename from api/client-server/v1/events
rename to api/client-server/v1/v1-event-schema
diff --git a/event-schemas/schema/v1/core-event-schema/core-event-schema b/event-schemas/schema/v1/core-event-schema/core-event-schema
new file mode 120000
index 00000000..945c9b46
--- /dev/null
+++ b/event-schemas/schema/v1/core-event-schema/core-event-schema
@@ -0,0 +1 @@
+.
\ No newline at end of file
diff --git a/event-schemas/schema/v1/core/event.json b/event-schemas/schema/v1/core-event-schema/event.json
similarity index 100%
rename from event-schemas/schema/v1/core/event.json
rename to event-schemas/schema/v1/core-event-schema/event.json
diff --git a/event-schemas/schema/v1/core/msgtype_infos/image_info.json b/event-schemas/schema/v1/core-event-schema/msgtype_infos/image_info.json
similarity index 100%
rename from event-schemas/schema/v1/core/msgtype_infos/image_info.json
rename to event-schemas/schema/v1/core-event-schema/msgtype_infos/image_info.json
diff --git a/event-schemas/schema/v1/core/room_event.json b/event-schemas/schema/v1/core-event-schema/room_event.json
similarity index 93%
rename from event-schemas/schema/v1/core/room_event.json
rename to event-schemas/schema/v1/core-event-schema/room_event.json
index 7ac66b30..d5413f8a 100644
--- a/event-schemas/schema/v1/core/room_event.json
+++ b/event-schemas/schema/v1/core-event-schema/room_event.json
@@ -3,7 +3,7 @@
   "title": "Room Event",
   "description": "In addition to the Event fields, Room Events MUST have the following additional field.",
   "allOf":[{
-    "$ref": "core/event.json"
+    "$ref": "core-event-schema/event.json"
   }],
   "properties": {
     "room_id": {
diff --git a/event-schemas/schema/v1/core/state_event.json b/event-schemas/schema/v1/core-event-schema/state_event.json
similarity index 93%
rename from event-schemas/schema/v1/core/state_event.json
rename to event-schemas/schema/v1/core-event-schema/state_event.json
index 60de9413..f70118f4 100644
--- a/event-schemas/schema/v1/core/state_event.json
+++ b/event-schemas/schema/v1/core-event-schema/state_event.json
@@ -3,7 +3,7 @@
   "title": "State Event",
   "description": "In addition to the Room Event fields, State Events have the following additional fields.",
   "allOf":[{
-    "$ref": "core/room_event.json"
+    "$ref": "core-event-schema/room_event.json"
   }],
   "properties": {
     "state_key": {
diff --git a/event-schemas/schema/v1/m.call.answer b/event-schemas/schema/v1/m.call.answer
index abdf4a9b..f8980a8f 100644
--- a/event-schemas/schema/v1/m.call.answer
+++ b/event-schemas/schema/v1/m.call.answer
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the callee when they wish to answer the call.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.candidates b/event-schemas/schema/v1/m.call.candidates
index 052ead0b..ad84fe86 100644
--- a/event-schemas/schema/v1/m.call.candidates
+++ b/event-schemas/schema/v1/m.call.candidates
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by callers after sending an invite and by the callee after answering. Its purpose is to give the other party additional ICE candidates to try using to communicate.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.hangup b/event-schemas/schema/v1/m.call.hangup
index 383952d8..ded16a0d 100644
--- a/event-schemas/schema/v1/m.call.hangup
+++ b/event-schemas/schema/v1/m.call.hangup
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.call.invite b/event-schemas/schema/v1/m.call.invite
index a852ff43..bfeb3b7d 100644
--- a/event-schemas/schema/v1/m.call.invite
+++ b/event-schemas/schema/v1/m.call.invite
@@ -3,7 +3,7 @@
     "type": "object",
     "description": "This event is sent by the caller when they wish to establish a call.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.aliases b/event-schemas/schema/v1/m.room.aliases
index 79fd259b..d1ad0fc7 100644
--- a/event-schemas/schema/v1/m.room.aliases
+++ b/event-schemas/schema/v1/m.room.aliases
@@ -4,7 +4,7 @@
     "title": "Informs the room about what room aliases it has been given.",
     "description": "This event is sent by a homeserver directly to inform of changes to the list of aliases it knows about for that room. The ``state_key`` for this event is set to the homeserver which owns the room alias. The entire set of known aliases for the room is the union of all the ``m.room.aliases`` events, one for each homeserver. Clients **should** check the validity of any room alias given in this list before presenting it to the user as trusted fact. The lists given by this event should be considered simply as advice on which aliases might exist, for which the client can perform the lookup to confirm whether it receives the correct room ID.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.canonical_alias b/event-schemas/schema/v1/m.room.canonical_alias
index 17caa321..6d56e27e 100644
--- a/event-schemas/schema/v1/m.room.canonical_alias
+++ b/event-schemas/schema/v1/m.room.canonical_alias
@@ -4,7 +4,7 @@
     "title": "Informs the room as to which alias is the canonical one.",
     "description": "This event is used to inform the room about which alias should be considered the canonical one. This could be for display purposes or as suggestion to users which alias to use to advertise the room.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index ecc2ed63..bd9231f4 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -4,7 +4,7 @@
     "title": "The first event in the room.",
     "description": "This is the first event in a room and cannot be changed. It acts as the root of all other events.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index 48b0a5c4..a35fe779 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -4,7 +4,7 @@
     "title": "Controls visibility of history.",
     "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.join_rules b/event-schemas/schema/v1/m.room.join_rules
index 567247eb..8c70dcaf 100644
--- a/event-schemas/schema/v1/m.room.join_rules
+++ b/event-schemas/schema/v1/m.room.join_rules
@@ -4,7 +4,7 @@
     "title": "Describes how users are allowed to join the room.",
     "description": "A room may be ``public`` meaning anyone can join the room without any prior action. Alternatively, it can be ``invite`` meaning that a user who wishes to join the room must first receive an invite to the room from someone already inside of the room. Currently, ``knock`` and ``private`` are reserved keywords which are not implemented.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 67e10d51..a8a442bc 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -4,7 +4,7 @@
     "title": "The current membership state of a user in the room.",
     "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index 61b6256f..59edb676 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -4,7 +4,7 @@
     "title": "Message",
     "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.audio b/event-schemas/schema/v1/m.room.message#m.audio
index b236e5b0..41587c1d 100644
--- a/event-schemas/schema/v1/m.room.message#m.audio
+++ b/event-schemas/schema/v1/m.room.message#m.audio
@@ -4,7 +4,7 @@
     "title": "AudioMessage",
     "description": "This message represents a single audio clip.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.emote b/event-schemas/schema/v1/m.room.message#m.emote
index 9f17180c..e6bf5e68 100644
--- a/event-schemas/schema/v1/m.room.message#m.emote
+++ b/event-schemas/schema/v1/m.room.message#m.emote
@@ -4,7 +4,7 @@
     "title": "EmoteMessage",
     "description": "This message is similar to ``m.text`` except that the sender is 'performing' the action contained in the ``body`` key, similar to ``/me`` in IRC. This message should be prefixed by the name of the sender. This message could also be represented in a different colour to distinguish it from regular ``m.text`` messages.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.file b/event-schemas/schema/v1/m.room.message#m.file
index d8c63b33..ed996694 100644
--- a/event-schemas/schema/v1/m.room.message#m.file
+++ b/event-schemas/schema/v1/m.room.message#m.file
@@ -4,7 +4,7 @@
     "title": "FileMessage",
     "description": "This message represents a generic file.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -50,7 +50,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core/msgtype_infos/image_info.json"
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.image b/event-schemas/schema/v1/m.room.message#m.image
index ca22654f..8baee04e 100644
--- a/event-schemas/schema/v1/m.room.message#m.image
+++ b/event-schemas/schema/v1/m.room.message#m.image
@@ -4,7 +4,7 @@
     "title": "ImageMessage",
     "description": "This message represents a single image and an optional thumbnail.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -31,7 +31,7 @@
                     "title": "ImageInfo",
                     "description": "Metadata about the image referred to in ``thumbnail_url``.",
                     "allOf": [{
-                        "$ref": "core/msgtype_infos/image_info.json"
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
                     }]
                 },
                 "info": {
diff --git a/event-schemas/schema/v1/m.room.message#m.location b/event-schemas/schema/v1/m.room.message#m.location
index 919c3325..9e59a0a9 100644
--- a/event-schemas/schema/v1/m.room.message#m.location
+++ b/event-schemas/schema/v1/m.room.message#m.location
@@ -4,7 +4,7 @@
     "title": "LocationMessage",
     "description": "This message represents a real-world location.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -30,7 +30,7 @@
                     "type": "object",
                     "title": "ImageInfo",
                     "allOf": [{
-                        "$ref": "core/msgtype_infos/image_info.json"
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
                     }]
                 }
             },
diff --git a/event-schemas/schema/v1/m.room.message#m.notice b/event-schemas/schema/v1/m.room.message#m.notice
index e6ce6f89..5fd217f8 100644
--- a/event-schemas/schema/v1/m.room.message#m.notice
+++ b/event-schemas/schema/v1/m.room.message#m.notice
@@ -4,7 +4,7 @@
     "title": "NoticeMessage",
     "description": "A m.notice message should be considered similar to a plain m.text message except that clients should visually distinguish it in some way. It is intended to be used by automated clients, such as bots, bridges, and other entities, rather than humans. Additionally, such automated agents which watch a room for messages and respond to them ought to ignore m.notice messages. This helps to prevent infinite-loop situations where two automated clients continuously exchange messages, as each responds to the other.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.text b/event-schemas/schema/v1/m.room.message#m.text
index ceadbcd0..e4b5ab11 100644
--- a/event-schemas/schema/v1/m.room.message#m.text
+++ b/event-schemas/schema/v1/m.room.message#m.text
@@ -4,7 +4,7 @@
     "title": "TextMessage",
     "description": "This message is the most basic message and is used to represent text.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.message#m.video b/event-schemas/schema/v1/m.room.message#m.video
index 667832ff..2de0f26a 100644
--- a/event-schemas/schema/v1/m.room.message#m.video
+++ b/event-schemas/schema/v1/m.room.message#m.video
@@ -4,7 +4,7 @@
     "title": "VideoMessage",
     "description": "This message represents a single video clip.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
@@ -55,7 +55,7 @@
                             "type": "object",
                             "title": "ImageInfo",
                             "allOf": [{
-                                "$ref": "core/msgtype_infos/image_info.json"
+                                "$ref": "core-event-schema/msgtype_infos/image_info.json"
                             }]
                         }
                     }
diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index b662e9e6..2dc63b8a 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -4,7 +4,7 @@
     "title": "MessageFeedback",
     "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index 5565147c..0e212a23 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -4,7 +4,7 @@
     "description": "A room has an opaque room ID which is not human-friendly to read. A room alias is human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not unique, as multiple rooms can have the same room name set. The room name can also be set when creating a room using ``/createRoom`` with the ``name`` key.",
     "type": "object",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.power_levels b/event-schemas/schema/v1/m.room.power_levels
index 95850fcb..a682a214 100644
--- a/event-schemas/schema/v1/m.room.power_levels
+++ b/event-schemas/schema/v1/m.room.power_levels
@@ -4,7 +4,7 @@
     "title": "Defines the power levels (privileges) of users in the room.",
     "description": "This event specifies the minimum level a user must have in order to perform a certain action. It also specifies the levels of each user in the room. If a ``user_id`` is in the ``users`` list, then that ``user_id`` has the associated power level. Otherwise they have the default level ``users_default``. If ``users_default`` is not supplied, it is assumed to be 0. The level required to send a certain event is governed by ``events``, ``state_default`` and ``events_default``. If an event type is specified in ``events``, then the user must have at least the level specified in order to send that event. If the event type is not supplied, it defaults to ``events_default`` for Message Events and ``state_default`` for State Events.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.redaction b/event-schemas/schema/v1/m.room.redaction
index b095b17d..5168e133 100644
--- a/event-schemas/schema/v1/m.room.redaction
+++ b/event-schemas/schema/v1/m.room.redaction
@@ -4,7 +4,7 @@
     "title": "Redaction",
     "description": "Events can be redacted by either room or server admins. Redacting an event means that all keys not required by the protocol are stripped off, allowing admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data.  There is also a concept of a moderator hiding a message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key.",
     "allOf": [{
-        "$ref": "core/room_event.json"
+        "$ref": "core-event-schema/room_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/m.room.topic b/event-schemas/schema/v1/m.room.topic
index 32319c38..f88f56ca 100644
--- a/event-schemas/schema/v1/m.room.topic
+++ b/event-schemas/schema/v1/m.room.topic
@@ -4,7 +4,7 @@
     "title": "Topic",
     "description": "A topic is a short message detailing what is currently being discussed in the room.  It can also be used as a way to display extra information about the room, which may not be suitable for the room name. The room topic can also be set when creating a room using ``/createRoom`` with the ``topic`` key.",
     "allOf": [{
-        "$ref": "core/state_event.json"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {
diff --git a/event-schemas/schema/v1/v1-event-schema b/event-schemas/schema/v1/v1-event-schema
new file mode 120000
index 00000000..945c9b46
--- /dev/null
+++ b/event-schemas/schema/v1/v1-event-schema
@@ -0,0 +1 @@
+.
\ No newline at end of file
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 4b8127ae..8e803d3b 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -281,7 +281,7 @@ class MatrixUnits(Units):
         return apis
 
     def load_common_event_fields(self):
-        path = "../event-schemas/schema/v1/core"
+        path = "../event-schemas/schema/v1/core-event-schema"
         event_types = {}
 
         for (root, dirs, files) in os.walk(path):

From 1f6b12b3e828696a588fed9a0028e59991d28b94 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 11:43:53 +0100
Subject: [PATCH 105/467] Modify and enforce the file format/structure used

Convert the file format to be of the form ##_##_something.rst where the
first ## is the top-level section number and the second ## is the
second-level section number, e.g. 07_01_push_cs_api.rst means
Section 7.1 - This is now enforced in gendoc.py along with the title line
style that should be used (= for top-level, - for 2nd level) which will
give helpful suggestions if you trip up. This feels much more intuitive
now looking in /specification
---
 scripts/gendoc.py                                  | 12 +++++++-----
 specification/{00_basis.rst => 00_00_intro.rst}    |  0
 ..._server_api.rst => 01_00_client_server_api.rst} |  0
 ...ications.rst => 01_01_typing_notifications.rst} |  8 ++++----
 .../{46_receipts.rst => 01_02_receipts.rst}        |  8 ++++----
 ...visibility.rst => 01_03_history_visibility.rst} |  2 +-
 specification/{20_events.rst => 02_00_events.rst}  |  0
 .../{22_voip_events.rst => 02_01_voip_events.rst}  |  0
 ...1_event_signing.rst => 02_02_event_signing.rst} | 14 +++++++-------
 ...e_api.rst => 03_00_application_service_api.rst} |  0
 ..._server_api.rst => 04_00_server_server_api.rst} |  0
 ...repository.rst => 05_00_content_repository.rst} |  0
 ...ryption.rst => 06_00_end_to_end_encryption.rst} |  0
 ...2_push_overview.rst => 07_00_push_overview.rst} |  0
 .../{43_push_cs_api.rst => 07_01_push_cs_api.rst}  |  0
 ..._push_gw_api.rst => 07_02_push_push_gw_api.rst} |  0
 ...n_core_apis.rst => 08_00_address_book_repo.rst} |  0
 ...tity_servers.rst => 09_00_identity_servers.rst} |  0
 .../{50_appendices.rst => 10_00_appendices.rst}    |  0
 19 files changed, 23 insertions(+), 21 deletions(-)
 rename specification/{00_basis.rst => 00_00_intro.rst} (100%)
 rename specification/{10_client_server_api.rst => 01_00_client_server_api.rst} (100%)
 rename specification/{45_typing_notifications.rst => 01_01_typing_notifications.rst} (96%)
 rename specification/{46_receipts.rst => 01_02_receipts.rst} (94%)
 rename specification/{47_history_visibility.rst => 01_03_history_visibility.rst} (97%)
 rename specification/{20_events.rst => 02_00_events.rst} (100%)
 rename specification/{22_voip_events.rst => 02_01_voip_events.rst} (100%)
 rename specification/{31_event_signing.rst => 02_02_event_signing.rst} (99%)
 rename specification/{25_application_service_api.rst => 03_00_application_service_api.rst} (100%)
 rename specification/{30_server_server_api.rst => 04_00_server_server_api.rst} (100%)
 rename specification/{40_content_repository.rst => 05_00_content_repository.rst} (100%)
 rename specification/{41_end_to_end_encryption.rst => 06_00_end_to_end_encryption.rst} (100%)
 rename specification/{42_push_overview.rst => 07_00_push_overview.rst} (100%)
 rename specification/{43_push_cs_api.rst => 07_01_push_cs_api.rst} (100%)
 rename specification/{44_push_push_gw_api.rst => 07_02_push_push_gw_api.rst} (100%)
 rename specification/{49_other_non_core_apis.rst => 08_00_address_book_repo.rst} (100%)
 rename specification/{60_identity_servers.rst => 09_00_identity_servers.rst} (100%)
 rename specification/{50_appendices.rst => 10_00_appendices.rst} (100%)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 1348679a..c97995e8 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -21,7 +21,7 @@ title_style_matchers = {
 }
 TOP_LEVEL = "="
 SECOND_LEVEL = "-"
-FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}_.*\.rst$")
+FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
 
 
 def check_valid_section(filename, section):
@@ -55,14 +55,16 @@ def check_valid_section(filename, section):
                 "style: expected '" + TOP_LEVEL + "' but got '" +
                 title_line[0] + "'"
             )
-    # anything marked as xx_x0_ is the start of a sub-section
-    elif re.match("^[0-9]+_0[0-9]{1}_", filename):
+    # anything marked as xx_xx_ is the start of a sub-section
+    elif re.match("^[0-9]+_[0-9]{2}_", filename):
         if not title_style_matchers[SECOND_LEVEL].match(title_line):
             raise Exception(
                 "The file " + filename + " is a 2nd-level section because it matches " +
-                "the filename format ##_#0_something.rst but has the wrong title " +
+                "the filename format ##_##_something.rst but has the wrong title " +
                 "style: expected '" + SECOND_LEVEL + "' but got '" +
-                title_line[0] + "'"
+                title_line[0] + "' - If this is meant to be a 3rd/4th/5th-level section " +
+                "then use the form '##_##b_something.rst' which will not apply this " +
+                "check."
             )
 
 def cat_spec_sections_to(out_file_name):
diff --git a/specification/00_basis.rst b/specification/00_00_intro.rst
similarity index 100%
rename from specification/00_basis.rst
rename to specification/00_00_intro.rst
diff --git a/specification/10_client_server_api.rst b/specification/01_00_client_server_api.rst
similarity index 100%
rename from specification/10_client_server_api.rst
rename to specification/01_00_client_server_api.rst
diff --git a/specification/45_typing_notifications.rst b/specification/01_01_typing_notifications.rst
similarity index 96%
rename from specification/45_typing_notifications.rst
rename to specification/01_01_typing_notifications.rst
index ff00165d..25b714ab 100644
--- a/specification/45_typing_notifications.rst
+++ b/specification/01_01_typing_notifications.rst
@@ -1,8 +1,8 @@
 Typing Notifications
-====================
+--------------------
 
 Client APIs
------------
+~~~~~~~~~~~
 
 To set "I am typing for the next N msec"::
 
@@ -21,7 +21,7 @@ To set "I am no longer typing"::
   Content: { "typing": false }
 
 Client Events
--------------
+~~~~~~~~~~~~~
 
 All room members will receive an event on the event stream::
 
@@ -39,7 +39,7 @@ users who are not currently typing, as that list gets big quickly. The client
 should mark as not typing, any user ID who is not in that list.
 
 Server APIs
------------
+~~~~~~~~~~~
 
 Servers will emit EDUs in the following form::
 
diff --git a/specification/46_receipts.rst b/specification/01_02_receipts.rst
similarity index 94%
rename from specification/46_receipts.rst
rename to specification/01_02_receipts.rst
index a0a90c16..e2f83eea 100644
--- a/specification/46_receipts.rst
+++ b/specification/01_02_receipts.rst
@@ -1,5 +1,5 @@
 Receipts
-========
+--------
 
 Receipts are used to publish which events in a room the user or their devices
 have interacted with. For example, which events the user has read. For
@@ -7,7 +7,7 @@ efficiency this is done as "up to" markers, i.e. marking a particular event
 as, say, ``read`` indicates the user has read all events *up to* that event.
 
 Client-Server API
------------------
+~~~~~~~~~~~~~~~~~
 
 Clients will receive receipts in the following format::
 
@@ -58,7 +58,7 @@ other users. The server will automatically set the ``ts`` field.
 
 
 Server-Server API
------------------
+~~~~~~~~~~~~~~~~~
 
 Receipts are sent across federation as EDUs with type ``m.receipt``. The
 format of the EDUs are::
@@ -73,5 +73,5 @@ format of the EDUs are::
         ...
     }
 
-These are always sent as deltas to previously sent reciepts.
+These are always sent as deltas to previously sent receipts.
 
diff --git a/specification/47_history_visibility.rst b/specification/01_03_history_visibility.rst
similarity index 97%
rename from specification/47_history_visibility.rst
rename to specification/01_03_history_visibility.rst
index b1630f1e..01c2e419 100644
--- a/specification/47_history_visibility.rst
+++ b/specification/01_03_history_visibility.rst
@@ -1,5 +1,5 @@
 Room History Visibility
-=======================
+-----------------------
 
 Whether a member of a room can see the events that happened in a room from
 before they joined the room is controlled by the ``history_visibility`` key
diff --git a/specification/20_events.rst b/specification/02_00_events.rst
similarity index 100%
rename from specification/20_events.rst
rename to specification/02_00_events.rst
diff --git a/specification/22_voip_events.rst b/specification/02_01_voip_events.rst
similarity index 100%
rename from specification/22_voip_events.rst
rename to specification/02_01_voip_events.rst
diff --git a/specification/31_event_signing.rst b/specification/02_02_event_signing.rst
similarity index 99%
rename from specification/31_event_signing.rst
rename to specification/02_02_event_signing.rst
index f3a22d1d..bde58f0c 100644
--- a/specification/31_event_signing.rst
+++ b/specification/02_02_event_signing.rst
@@ -1,8 +1,8 @@
 Signing Events
-==============
+--------------
 
 Canonical JSON
---------------
+~~~~~~~~~~~~~~
 
 Matrix events are represented using JSON objects. If we want to sign JSON
 events we need to encode the JSON as a binary string. Unfortunately the same
@@ -38,7 +38,7 @@ using this representation.
      ).encode("UTF-8")
 
 Grammar
-~~~~~~~
++++++++
 
 Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
 insignificant whitespace, fractions, exponents and redundant character escapes
@@ -69,14 +69,14 @@ insignificant whitespace, fractions, exponents and redundant character escapes
            / %x75.30.30.31 (%x30-39 / %x61-66)        ; u001X
 
 Signing JSON
-------------
+~~~~~~~~~~~~
 
 We can now sign a JSON object by encoding it as a sequence of bytes, computing
 the signature for that sequence and then adding the signature to the original
 JSON object.
 
 Signing Details
-~~~~~~~~~~~~~~~
++++++++++++++++
 
 JSON is signed by encoding the JSON object without ``signatures`` or keys grouped
 as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the
@@ -133,7 +133,7 @@ and additional signatures.
       return json_object
 
 Checking for a Signature
-~~~~~~~~~~~~~~~~~~~~~~~~
+++++++++++++++++++++++++
 
 To check if an entity has signed a JSON object a server does the following
 
@@ -151,7 +151,7 @@ To check if an entity has signed a JSON object a server does the following
    the check fails. Otherwise the check succeeds.
 
 Signing Events
---------------
+~~~~~~~~~~~~~~
 
 Signing events is a more complicated process since servers can choose to redact
 non-essential parts of an event. Before signing the event it is encoded as
diff --git a/specification/25_application_service_api.rst b/specification/03_00_application_service_api.rst
similarity index 100%
rename from specification/25_application_service_api.rst
rename to specification/03_00_application_service_api.rst
diff --git a/specification/30_server_server_api.rst b/specification/04_00_server_server_api.rst
similarity index 100%
rename from specification/30_server_server_api.rst
rename to specification/04_00_server_server_api.rst
diff --git a/specification/40_content_repository.rst b/specification/05_00_content_repository.rst
similarity index 100%
rename from specification/40_content_repository.rst
rename to specification/05_00_content_repository.rst
diff --git a/specification/41_end_to_end_encryption.rst b/specification/06_00_end_to_end_encryption.rst
similarity index 100%
rename from specification/41_end_to_end_encryption.rst
rename to specification/06_00_end_to_end_encryption.rst
diff --git a/specification/42_push_overview.rst b/specification/07_00_push_overview.rst
similarity index 100%
rename from specification/42_push_overview.rst
rename to specification/07_00_push_overview.rst
diff --git a/specification/43_push_cs_api.rst b/specification/07_01_push_cs_api.rst
similarity index 100%
rename from specification/43_push_cs_api.rst
rename to specification/07_01_push_cs_api.rst
diff --git a/specification/44_push_push_gw_api.rst b/specification/07_02_push_push_gw_api.rst
similarity index 100%
rename from specification/44_push_push_gw_api.rst
rename to specification/07_02_push_push_gw_api.rst
diff --git a/specification/49_other_non_core_apis.rst b/specification/08_00_address_book_repo.rst
similarity index 100%
rename from specification/49_other_non_core_apis.rst
rename to specification/08_00_address_book_repo.rst
diff --git a/specification/60_identity_servers.rst b/specification/09_00_identity_servers.rst
similarity index 100%
rename from specification/60_identity_servers.rst
rename to specification/09_00_identity_servers.rst
diff --git a/specification/50_appendices.rst b/specification/10_00_appendices.rst
similarity index 100%
rename from specification/50_appendices.rst
rename to specification/10_00_appendices.rst

From 63f08bace61b01d629bcdc3beaaf8870c7bfd993 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 14:40:48 +0100
Subject: [PATCH 106/467] Fix the examples in the swagger API documentation to
 be valid JSON

---
 api/client-server/v1/presence.yaml   | 6 +++---
 api/client-server/v1/sync.yaml       | 2 +-
 templating/matrix_templates/units.py | 2 ++
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v1/presence.yaml b/api/client-server/v1/presence.yaml
index 717e9c47..5684398b 100644
--- a/api/client-server/v1/presence.yaml
+++ b/api/client-server/v1/presence.yaml
@@ -101,7 +101,7 @@ paths:
                   The length of time in milliseconds since an action was performed
                   by this user.
               status_msg:
-                type: string
+                type: [string, "null"]
                 description: The state message for this user if one was set.
         404:
           description: |-
@@ -185,7 +185,7 @@ paths:
                     "last_active_ago": 395,
                     "presence": "offline",
                     "user_id": "@alice:matrix.org"
-                  }
+                  },
                   "type": "m.presence"
                 },
                 {
@@ -195,7 +195,7 @@ paths:
                     "last_active_ago": 16874,
                     "presence": "online",
                     "user_id": "@marisa:matrix.org"
-                  }
+                  },
                   "type": "m.presence"
                 }
               ]
diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 27c7073c..833c425a 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -343,7 +343,7 @@ paths:
                   "body": "Hello world!",
                   "msgtype": "m.text"
                 },
-                "room_id:" "!wfgy43Sg4a:matrix.org",
+                "room_id:": "!wfgy43Sg4a:matrix.org",
                 "user_id": "@bob:matrix.org",
                 "event_id": "$asfDuShaf7Gafaw:matrix.org",
                 "type": "m.room.message"
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 8e803d3b..3083a51e 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -87,6 +87,8 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                     desc += (
                         " Must be '%s'." % props[key_name]["enum"][0]
                     )
+            if isinstance(value_type, list):
+                value_type = " or ".join(value_type)
 
         fields["rows"].append({
             "key": key_name,

From f99a38ce726e5d04a9be182bd42f43daf73a3364 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 14:43:57 +0100
Subject: [PATCH 107/467] Update the hard-coded paths in templating units.py.

Replace the hard code paths with global variables.
---
 templating/matrix_templates/units.py | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 4b8127ae..701b68e2 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -8,6 +8,16 @@ import subprocess
 import urllib
 import yaml
 
+V1_CLIENT_API = "../api/client-server/v1"
+V1_EVENT_EXAMPLES = "../event-schemas/examples/v1"
+V1_EVENT_SCHEMA = "../event-schemas/schema/v1"
+CORE_EVENT_SCHEMA = "../event-schemas/schema/v1/core-event-schema"
+CHANGELOG = "../CHANGELOG.rst"
+
+ROOM_EVENT = "core-event-schema/room_event.json"
+STATE_EVENT = "core-event-schema/state_event.json"
+
+
 def get_json_schema_object_fields(obj, enforce_title=False):
     # Algorithm:
     # f.e. property => add field info (if field is object then recurse)
@@ -266,7 +276,7 @@ class MatrixUnits(Units):
         }
 
     def load_swagger_apis(self):
-        path = "../api/client-server/v1"
+        path = V1_CLIENT_API
         apis = {}
         for filename in os.listdir(path):
             if not filename.endswith(".yaml"):
@@ -281,7 +291,7 @@ class MatrixUnits(Units):
         return apis
 
     def load_common_event_fields(self):
-        path = "../event-schemas/schema/v1/core"
+        path = CORE_EVENT_SCHEMA
         event_types = {}
 
         for (root, dirs, files) in os.walk(path):
@@ -320,7 +330,7 @@ class MatrixUnits(Units):
         return event_types
 
     def load_event_examples(self):
-        path = "../event-schemas/examples/v1"
+        path = V1_EVENT_EXAMPLES
         examples = {}
         for filename in os.listdir(path):
             if not filename.startswith("m."):
@@ -332,7 +342,7 @@ class MatrixUnits(Units):
         return examples
 
     def load_event_schemas(self):
-        path = "../event-schemas/schema/v1"
+        path = V1_EVENT_SCHEMA
         schemata = {}
 
         for filename in os.listdir(path):
@@ -361,8 +371,8 @@ class MatrixUnits(Units):
 
                 # add typeof
                 base_defs = {
-                    "core#/definitions/room_event": "Message Event",
-                    "core#/definitions/state_event": "State Event"
+                    ROOM_EVENT: "Message Event",
+                    STATE_EVENT: "State Event"
                 }
                 if type(json_schema.get("allOf")) == list:
                     schema["typeof"] = base_defs.get(
@@ -413,7 +423,7 @@ class MatrixUnits(Units):
         return schemata
 
     def load_spec_meta(self):
-        path = "../CHANGELOG.rst"
+        path = CHANGELOG
         title_part = None
         version = None
         changelog_lines = []

From 427e4c8b1b23ab875b24acad5605848abf0d8ee9 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:46:37 +0100
Subject: [PATCH 108/467] Log port being listened on

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index e7757c06..dc7705a8 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-
+    fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From ba4998a62b28fcd13c9085ae58996d1c857b27c5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 17 Sep 2015 15:49:01 +0100
Subject: [PATCH 109/467] Apparently tabs are A Thing

---
 scripts/continuserv/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index dc7705a8..658ae0fb 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -56,7 +56,7 @@ func main() {
 	go doPopulate(ch, dir)
 
 	go watchFS(ch, w)
-    fmt.Printf("Listening on port %d\n", *port)
+	fmt.Printf("Listening on port %d\n", *port)
 	http.HandleFunc("/", serve)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 

From cf8abdafd43ad67802b20896d53ab977d6d4beee Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 18 Sep 2015 15:42:49 +0100
Subject: [PATCH 110/467] Fix PR comments

---
 scripts/gendoc.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index c97995e8..e2f51419 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -27,13 +27,13 @@ FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
 def check_valid_section(filename, section):
     if not re.match(FILE_FORMAT_MATCHER, filename):
         raise Exception(
-            "The filename of " + filename +" does not match the expected format " +
+            "The filename of " + filename + " does not match the expected format " +
             "of '##_##_words-go-here.rst'"
         )
 
     # we need TWO new lines else the next file's title gets merged
     # the last paragraph *WITHOUT RST PRODUCING A WARNING*
-    if not section[-2:] == '\n\n':
+    if not section[-2:] == "\n\n":
         raise Exception(
             "The file " + filename + " does not end with 2 new lines."
         )
@@ -72,7 +72,7 @@ def cat_spec_sections_to(out_file_name):
         for f in sorted(glob.glob("../specification/*.rst")):
             with open(f, "rb") as infile:
                 section = infile.read()
-                check_valid_section(f.split("/")[-1], section)
+                check_valid_section(os.path.basename(f), section)
                 outfile.write(section)
 
 

From 299a4356d49e9b694c1fb6340cda2669e8e30377 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 16:10:21 +0100
Subject: [PATCH 111/467] Add script to check that the example responses in the
 swagger matches the examples.

---
 api/check_examples.py | 75 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
 create mode 100755 api/check_examples.py

diff --git a/api/check_examples.py b/api/check_examples.py
new file mode 100755
index 00000000..cec8ac09
--- /dev/null
+++ b/api/check_examples.py
@@ -0,0 +1,75 @@
+#! /usr/bin/env python
+
+import sys
+
+def import_error(module, package, debian, error):
+    sys.stderr.write((
+        "Error importing %(module)s: %(error)r\n"
+        "To install %(module)s run:\n"
+        "  pip install %(package)s\n"
+        "or on Debian run:\n"
+        "  sudo apt-get install python-%(debian)s\n"
+    ) % locals())
+    if __name__=='__main__':
+        sys.exit(1)
+
+try:
+    import jsonschema
+except ImportError as e:
+    import_error("jsonschema", "jsonschema", "jsonschema", e)
+    raise
+
+try:
+    import yaml
+except ImportError as e:
+    import_error("yaml", "PyYAML", "yaml", e)
+    raise
+
+import json
+import os
+
+def check_response(filepath, request, code, response):
+    try:
+        example = json.loads(
+            response.get('examples', {}).get('application/json', "null")
+        )
+    except Exception as e:
+        raise ValueError("Error parsing JSON example response for %r %r" % (
+            request, code
+        ), e)
+    schema = response.get('schema')
+    fileurl = "file://" + os.path.abspath(filepath)
+    if example and schema:
+        try:
+            print ("Checking schema for: %r %r %r" % (filepath, request, code))
+            # Setting the 'id' tells jsonschema where the file is so that it
+            # can correctly resolve relative $ref references in the schema
+            schema['id'] = fileurl
+            jsonschema.validate(example, schema)
+        except Exception as e:
+            raise ValueError("Error validating JSON schema for %r %r" %(
+                request, code
+            ), e)
+
+
+def check_swagger_file(filepath):
+    with open(filepath) as f:
+        swagger = yaml.load(f)
+
+    for path, path_api in swagger['paths'].items():
+        for method, request_api in path_api.items():
+            request = "%s %s" % (method.upper(), path)
+            try:
+                responses = request_api['responses']
+            except KeyError:
+                raise ValueError("No responses for %r" % (request,))
+            for code, response in responses.items():
+                check_response(filepath, request, code, response)
+
+
+if __name__=='__main__':
+    for path in sys.argv[1:]:
+        try:
+            check_swagger_file(path)
+        except Exception as e:
+            raise ValueError("Error checking file %r" % (path,), e)

From 9896f98e2b63fd6204482bd9c7b8ba699487576d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 16:21:48 +0100
Subject: [PATCH 112/467] Search for yaml swagger files if check_examples.py is
 run without arguments.

---
 api/check_examples.py | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index cec8ac09..9b2f1fe9 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -1,6 +1,9 @@
 #! /usr/bin/env python
 
 import sys
+import json
+import os
+
 
 def import_error(module, package, debian, error):
     sys.stderr.write((
@@ -10,7 +13,7 @@ def import_error(module, package, debian, error):
         "or on Debian run:\n"
         "  sudo apt-get install python-%(debian)s\n"
     ) % locals())
-    if __name__=='__main__':
+    if __name__ == '__main__':
         sys.exit(1)
 
 try:
@@ -25,8 +28,6 @@ except ImportError as e:
     import_error("yaml", "PyYAML", "yaml", e)
     raise
 
-import json
-import os
 
 def check_response(filepath, request, code, response):
     try:
@@ -47,7 +48,7 @@ def check_response(filepath, request, code, response):
             schema['id'] = fileurl
             jsonschema.validate(example, schema)
         except Exception as e:
-            raise ValueError("Error validating JSON schema for %r %r" %(
+            raise ValueError("Error validating JSON schema for %r %r" % (
                 request, code
             ), e)
 
@@ -56,7 +57,7 @@ def check_swagger_file(filepath):
     with open(filepath) as f:
         swagger = yaml.load(f)
 
-    for path, path_api in swagger['paths'].items():
+    for path, path_api in swagger.get('paths', {}).items():
         for method, request_api in path_api.items():
             request = "%s %s" % (method.upper(), path)
             try:
@@ -67,8 +68,15 @@ def check_swagger_file(filepath):
                 check_response(filepath, request, code, response)
 
 
-if __name__=='__main__':
-    for path in sys.argv[1:]:
+if __name__ == '__main__':
+    paths = sys.argv[1:]
+    if not paths:
+        paths = []
+        for (root, dirs, files) in os.walk(os.curdir):
+            for filename in files:
+                if filename.endswith(".yaml"):
+                    paths.append(os.path.join(root, filename))
+    for path in paths:
         try:
             check_swagger_file(path)
         except Exception as e:

From f827765ba1999073bc9357695b27bcc3e7e0c30c Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 16:35:27 +0100
Subject: [PATCH 113/467] Make to code to skip checking swagger responses which
 don't have an application/json example clearer.

---
 api/check_examples.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index 9b2f1fe9..00e75263 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -30,10 +30,11 @@ except ImportError as e:
 
 
 def check_response(filepath, request, code, response):
+    example = None
     try:
-        example = json.loads(
-            response.get('examples', {}).get('application/json', "null")
-        )
+        example_json = response.get('examples', {}).get('application/json')
+        if example_json:
+            example = json.loads(example_json)
     except Exception as e:
         raise ValueError("Error parsing JSON example response for %r %r" % (
             request, code

From 6b5b8432b3761484f3e36910575f5fd844c09400 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 17:26:10 +0100
Subject: [PATCH 114/467] Turn on code highlighting for HTTP api responses and
 add a code highlighting stylesheet for the specification.

---
 scripts/codehighlight.css                           | 6 ++++++
 scripts/gendoc.py                                   | 2 +-
 templating/matrix_templates/templates/http-api.tmpl | 4 +++-
 3 files changed, 10 insertions(+), 2 deletions(-)
 create mode 100644 scripts/codehighlight.css

diff --git a/scripts/codehighlight.css b/scripts/codehighlight.css
new file mode 100644
index 00000000..5c9b0c36
--- /dev/null
+++ b/scripts/codehighlight.css
@@ -0,0 +1,6 @@
+pre.code .comment, code .comment { color: green }
+pre.code .keyword, code .keyword { color: darkred; font-weight: bold }
+pre.code .name.builtin, code .name.builtin { color: darkred; font-weight: bold }
+pre.code .literal.number, code .literal.number { color: blue }
+pre.code .name.tag, code .name.tag { color: darkgreen }
+pre.code .literal.string, code .literal.string { color: darkblue }
diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 3521efed..a821aea7 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -12,7 +12,7 @@ import sys
 os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
 stylesheets = {
-    "stylesheet_path": ["basic.css", "nature.css"]
+    "stylesheet_path": ["basic.css", "nature.css", "codehighlight.css"]
 }
 
 title_style_matchers = {
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index a0b25924..a03ffa7d 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -62,7 +62,9 @@ Response{{"s" if endpoint.example.responses|length > 1 else "" }}:
 
 {{res["description"]}}
 
-Example::
+Example
+
+.. code:: json
 
   {{res["example"] | indent_block(2)}}
 

From 6a2c4d27fc187d6efb755705be9eca7a07dec26f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 18 Sep 2015 17:58:44 +0100
Subject: [PATCH 115/467] Update the docs for room v1 api

---
 api/client-server/v1/rooms.yaml       | 10 +++++-----
 event-schemas/schema/v1/m.room.member |  2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index c038dba5..4902560b 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -168,7 +168,7 @@ paths:
               title: StateEvent
               type: object
               allOf:
-                - "$ref": "events/core/state_event.json"
+                - "$ref": "core-event-schema/state_event.json"
         403:
           description: You are not joined to the room.
 
@@ -344,7 +344,7 @@ paths:
                       type: object
                       title: RoomEvent
                       allOf:
-                        - "$ref": "events/core/room_event.json"
+                        - "$ref": "core-event-schema/room_event.json"
                 required: ["start", "end", "chunk"]
               state:
                 type: array
@@ -357,7 +357,7 @@ paths:
                   title: StateEvent
                   type: object
                   allOf:
-                    - "$ref": "events/core/state_event.json"
+                    - "$ref": "core-event-schema/state_event.json"
               visibility:
                 type: string
                 enum: ["private", "public"]
@@ -414,7 +414,7 @@ paths:
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"
-                  },
+                  }
                 ]
               }
           schema:
@@ -426,5 +426,5 @@ paths:
                   title: MemberEvent
                   type: object
                   allOf:
-                    - "$ref": "events/m.room.member"
+                    - "$ref": "v1-event-schema/m.room.member"
 
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 48f1b034..49b9f5b8 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -19,7 +19,7 @@
                     "description": "The avatar URL for this user, if any. This is added by the homeserver."
                 },
                 "displayname": {
-                    "type": "string",
+                    "type": ["string", "null"],
                     "description": "The display name for this user, if any. This is added by the homeserver."
                 }
             },

From 52640eb2053e155c7f33468c0e6c74512b062b0b Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 13:02:37 +0100
Subject: [PATCH 116/467] Add a python script for checking that the examples
 match the event schema.

Does the same checks as check.sh, but is a *lot* faster making it
suitable for using as a pre-commit hook.

I don't suggest replacing check.sh since it's good to check that the
schema works with multiple implementations of jsonschema.
---
 event-schemas/check_examples.py | 62 +++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
 create mode 100755 event-schemas/check_examples.py

diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py
new file mode 100755
index 00000000..5fc08f4a
--- /dev/null
+++ b/event-schemas/check_examples.py
@@ -0,0 +1,62 @@
+#! /usr/bin/env python
+
+import sys
+import json
+import os
+
+
+def import_error(module, package, debian, error):
+    sys.stderr.write((
+        "Error importing %(module)s: %(error)r\n"
+        "To install %(module)s run:\n"
+        "  pip install %(package)s\n"
+        "or on Debian run:\n"
+        "  sudo apt-get install python-%(debian)s\n"
+    ) % locals())
+    if __name__ == '__main__':
+        sys.exit(1)
+
+try:
+    import jsonschema
+except ImportError as e:
+    import_error("jsonschema", "jsonschema", "jsonschema", e)
+    raise
+
+try:
+    import yaml
+except ImportError as e:
+    import_error("yaml", "PyYAML", "yaml", e)
+    raise
+
+
+def check_example_file(examplepath, schemapath):
+    with open(examplepath) as f:
+        example = yaml.load(f)
+
+    with open(schemapath) as f:
+        schema = yaml.load(f)
+
+    fileurl = "file://" + os.path.abspath(schemapath)
+
+    print ("Checking schema for: %r %r" % (examplepath, schemapath))
+    # Setting the 'id' tells jsonschema where the file is so that it
+    # can correctly resolve relative $ref references in the schema
+    schema['id'] = fileurl
+    try:
+        jsonschema.validate(example, schema)
+    except:
+        raise ValueError("Error validating JSON schema for %r %r" % (
+            examplepath, schemapath
+        ), e)
+
+
+def check_example_dir(exampledir, schemadir):
+    for root, dirs, files in os.walk(exampledir):
+        for filename in files:
+            examplepath = os.path.join(root, filename)
+            schemapath = examplepath.replace(exampledir, schemadir)
+            check_example_file(examplepath, schemapath)
+
+
+if __name__ == '__main__':
+    check_example_dir("examples", "schema")

From f2c952ec5855bb120814a736bf482a19e1c43fd3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:13:18 +0100
Subject: [PATCH 117/467] Reshuffle spec into sections roughly right as per PR
 #52

---
 .../address-book-repo.rst                     |  0
 ..._repository.rst => 02_03_content_repo.rst} |  0
 ...on.rst => 02_04_end_to_end_encryption.rst} |  0
 ...ons.rst => 02_05_typing_notifications.rst} |  0
 ...{01_02_receipts.rst => 02_06_receipts.rst} |  0
 ...ility.rst => 02_07_history_visibility.rst} |  0
 ...h_overview.rst => 02_08_push_overview.rst} |  0
 ...push_cs_api.rst => 02_08b_push_cs_api.rst} |  0
 ...push_gw_api.rst => 02_08c_push_cs_api.rst} |  0
 .../03_00_application_service_api.rst         | 22 -------------------
 ...servers.rst => 05_00_identity_servers.rst} |  0
 ...00_appendices.rst => 06_00_appendices.rst} |  0
 12 files changed, 22 deletions(-)
 rename specification/08_00_address_book_repo.rst => drafts/address-book-repo.rst (100%)
 rename specification/{05_00_content_repository.rst => 02_03_content_repo.rst} (100%)
 rename specification/{06_00_end_to_end_encryption.rst => 02_04_end_to_end_encryption.rst} (100%)
 rename specification/{01_01_typing_notifications.rst => 02_05_typing_notifications.rst} (100%)
 rename specification/{01_02_receipts.rst => 02_06_receipts.rst} (100%)
 rename specification/{01_03_history_visibility.rst => 02_07_history_visibility.rst} (100%)
 rename specification/{07_00_push_overview.rst => 02_08_push_overview.rst} (100%)
 rename specification/{07_01_push_cs_api.rst => 02_08b_push_cs_api.rst} (100%)
 rename specification/{07_02_push_push_gw_api.rst => 02_08c_push_cs_api.rst} (100%)
 rename specification/{09_00_identity_servers.rst => 05_00_identity_servers.rst} (100%)
 rename specification/{10_00_appendices.rst => 06_00_appendices.rst} (100%)

diff --git a/specification/08_00_address_book_repo.rst b/drafts/address-book-repo.rst
similarity index 100%
rename from specification/08_00_address_book_repo.rst
rename to drafts/address-book-repo.rst
diff --git a/specification/05_00_content_repository.rst b/specification/02_03_content_repo.rst
similarity index 100%
rename from specification/05_00_content_repository.rst
rename to specification/02_03_content_repo.rst
diff --git a/specification/06_00_end_to_end_encryption.rst b/specification/02_04_end_to_end_encryption.rst
similarity index 100%
rename from specification/06_00_end_to_end_encryption.rst
rename to specification/02_04_end_to_end_encryption.rst
diff --git a/specification/01_01_typing_notifications.rst b/specification/02_05_typing_notifications.rst
similarity index 100%
rename from specification/01_01_typing_notifications.rst
rename to specification/02_05_typing_notifications.rst
diff --git a/specification/01_02_receipts.rst b/specification/02_06_receipts.rst
similarity index 100%
rename from specification/01_02_receipts.rst
rename to specification/02_06_receipts.rst
diff --git a/specification/01_03_history_visibility.rst b/specification/02_07_history_visibility.rst
similarity index 100%
rename from specification/01_03_history_visibility.rst
rename to specification/02_07_history_visibility.rst
diff --git a/specification/07_00_push_overview.rst b/specification/02_08_push_overview.rst
similarity index 100%
rename from specification/07_00_push_overview.rst
rename to specification/02_08_push_overview.rst
diff --git a/specification/07_01_push_cs_api.rst b/specification/02_08b_push_cs_api.rst
similarity index 100%
rename from specification/07_01_push_cs_api.rst
rename to specification/02_08b_push_cs_api.rst
diff --git a/specification/07_02_push_push_gw_api.rst b/specification/02_08c_push_cs_api.rst
similarity index 100%
rename from specification/07_02_push_push_gw_api.rst
rename to specification/02_08c_push_cs_api.rst
diff --git a/specification/03_00_application_service_api.rst b/specification/03_00_application_service_api.rst
index 9b58c861..2674ba44 100644
--- a/specification/03_00_application_service_api.rst
+++ b/specification/03_00_application_service_api.rst
@@ -400,25 +400,3 @@ in their content to provide a way for Matrix clients to link into the 'native'
 client from which the event originated.  For instance, this could contain the
 message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
 blog comment traffic in & out of matrix
-
-Active Application Services
-----------------------------
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-spec
-  API that provides hooks into the server so that you can intercept and
-  manipulate events, and/or insert virtual users & rooms into the server.
-
-Policy Servers
-==============
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-spec
-  We should mention them in the Architecture section at least: how they fit
-  into the picture.
-
-Enforcing policies
-------------------
-
diff --git a/specification/09_00_identity_servers.rst b/specification/05_00_identity_servers.rst
similarity index 100%
rename from specification/09_00_identity_servers.rst
rename to specification/05_00_identity_servers.rst
diff --git a/specification/10_00_appendices.rst b/specification/06_00_appendices.rst
similarity index 100%
rename from specification/10_00_appendices.rst
rename to specification/06_00_appendices.rst

From c77ef1a2ccca95149c4d9d9d9ee965068190fd23 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:17:31 +0100
Subject: [PATCH 118/467] Add stub files feature_profiles and modules

---
 specification/00_01_feature_profiles.rst                          | 0
 specification/{02_00_events.rst => 00_02a_events.rst}             | 0
 .../{02_02_event_signing.rst => 00_02b_event_signing.rst}         | 0
 specification/02_00_modules.rst                                   | 0
 .../{02_08_push_overview.rst => 02_08a_push_overview.rst}         | 0
 5 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 specification/00_01_feature_profiles.rst
 rename specification/{02_00_events.rst => 00_02a_events.rst} (100%)
 rename specification/{02_02_event_signing.rst => 00_02b_event_signing.rst} (100%)
 create mode 100644 specification/02_00_modules.rst
 rename specification/{02_08_push_overview.rst => 02_08a_push_overview.rst} (100%)

diff --git a/specification/00_01_feature_profiles.rst b/specification/00_01_feature_profiles.rst
new file mode 100644
index 00000000..e69de29b
diff --git a/specification/02_00_events.rst b/specification/00_02a_events.rst
similarity index 100%
rename from specification/02_00_events.rst
rename to specification/00_02a_events.rst
diff --git a/specification/02_02_event_signing.rst b/specification/00_02b_event_signing.rst
similarity index 100%
rename from specification/02_02_event_signing.rst
rename to specification/00_02b_event_signing.rst
diff --git a/specification/02_00_modules.rst b/specification/02_00_modules.rst
new file mode 100644
index 00000000..e69de29b
diff --git a/specification/02_08_push_overview.rst b/specification/02_08a_push_overview.rst
similarity index 100%
rename from specification/02_08_push_overview.rst
rename to specification/02_08a_push_overview.rst

From f520ac9d828890193772764e902c3ae5b4b77c1d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:25:52 +0100
Subject: [PATCH 119/467] Create a modules folder to group together similar
 parts of the spec

---
 .../{02_01_voip_events.rst => modules/01_00_voip_events.rst}      | 0
 .../02_00_typing_notifications.rst}                               | 0
 specification/{02_06_receipts.rst => modules/03_00_receipts.rst}  | 0
 .../{02_03_content_repo.rst => modules/04_00_content_repo.rst}    | 0
 .../05_00_end_to_end_encryption.rst}                              | 0
 .../06_00_history_visibility.rst}                                 | 0
 .../{02_08a_push_overview.rst => modules/07_00_push_overview.rst} | 0
 .../{02_08b_push_cs_api.rst => modules/07_01_push_cs_api.rst}     | 0
 .../07_02_push_push_gw_api.rst}                                   | 0
 9 files changed, 0 insertions(+), 0 deletions(-)
 rename specification/{02_01_voip_events.rst => modules/01_00_voip_events.rst} (100%)
 rename specification/{02_05_typing_notifications.rst => modules/02_00_typing_notifications.rst} (100%)
 rename specification/{02_06_receipts.rst => modules/03_00_receipts.rst} (100%)
 rename specification/{02_03_content_repo.rst => modules/04_00_content_repo.rst} (100%)
 rename specification/{02_04_end_to_end_encryption.rst => modules/05_00_end_to_end_encryption.rst} (100%)
 rename specification/{02_07_history_visibility.rst => modules/06_00_history_visibility.rst} (100%)
 rename specification/{02_08a_push_overview.rst => modules/07_00_push_overview.rst} (100%)
 rename specification/{02_08b_push_cs_api.rst => modules/07_01_push_cs_api.rst} (100%)
 rename specification/{02_08c_push_cs_api.rst => modules/07_02_push_push_gw_api.rst} (100%)

diff --git a/specification/02_01_voip_events.rst b/specification/modules/01_00_voip_events.rst
similarity index 100%
rename from specification/02_01_voip_events.rst
rename to specification/modules/01_00_voip_events.rst
diff --git a/specification/02_05_typing_notifications.rst b/specification/modules/02_00_typing_notifications.rst
similarity index 100%
rename from specification/02_05_typing_notifications.rst
rename to specification/modules/02_00_typing_notifications.rst
diff --git a/specification/02_06_receipts.rst b/specification/modules/03_00_receipts.rst
similarity index 100%
rename from specification/02_06_receipts.rst
rename to specification/modules/03_00_receipts.rst
diff --git a/specification/02_03_content_repo.rst b/specification/modules/04_00_content_repo.rst
similarity index 100%
rename from specification/02_03_content_repo.rst
rename to specification/modules/04_00_content_repo.rst
diff --git a/specification/02_04_end_to_end_encryption.rst b/specification/modules/05_00_end_to_end_encryption.rst
similarity index 100%
rename from specification/02_04_end_to_end_encryption.rst
rename to specification/modules/05_00_end_to_end_encryption.rst
diff --git a/specification/02_07_history_visibility.rst b/specification/modules/06_00_history_visibility.rst
similarity index 100%
rename from specification/02_07_history_visibility.rst
rename to specification/modules/06_00_history_visibility.rst
diff --git a/specification/02_08a_push_overview.rst b/specification/modules/07_00_push_overview.rst
similarity index 100%
rename from specification/02_08a_push_overview.rst
rename to specification/modules/07_00_push_overview.rst
diff --git a/specification/02_08b_push_cs_api.rst b/specification/modules/07_01_push_cs_api.rst
similarity index 100%
rename from specification/02_08b_push_cs_api.rst
rename to specification/modules/07_01_push_cs_api.rst
diff --git a/specification/02_08c_push_cs_api.rst b/specification/modules/07_02_push_push_gw_api.rst
similarity index 100%
rename from specification/02_08c_push_cs_api.rst
rename to specification/modules/07_02_push_push_gw_api.rst

From 703f913a47afc3b035a1730e9014e89807370b3e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:31:57 +0100
Subject: [PATCH 120/467] Add a 'targets.yaml' files in /specification

We're well beyond the point now where a simple `cat` of .rst files to "build"
the spec is practical. We may want to slice and dice the spec in different
ways to address various cross-cutting concerns. To this end, there is now a
'targets' file which contains the "build targets" for the spec, which contains
the sorting order for the .rst files. For now, we just have a single
target: 'main'.
---
 specification/targets.yaml | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 specification/targets.yaml

diff --git a/specification/targets.yaml b/specification/targets.yaml
new file mode 100644
index 00000000..845da99d
--- /dev/null
+++ b/specification/targets.yaml
@@ -0,0 +1,25 @@
+targets:
+  main:
+    files:
+      - 00_00_intro.rst
+      - 00_01_feature_profiles.rst
+      - 00_02a_events.rst
+      - 00_02b_event_signing.rst
+      - 01_00_client_server_api.rst
+      - 02_00_modules.rst
+      - "group:module"
+      - 03_00_application_service_api.rst
+      - 04_00_server_server_api.rst
+      - 05_00_identity_servers.rst
+      - 06_00_appendices.rst
+groups:
+  modules:
+    - modules/01_00_voip_events.rst
+    - modules/02_00_typing_notifications.rst
+    - modules/03_00_receipts.rst
+    - modules/04_00_content_repo.rst
+    - modules/05_00_end_to_end_encryption.rst
+    - modules/06_00_history_visibility.rst
+    - modules/07_00_push_overview.rst
+    - modules/07_01_push_cs_api.rst
+    - modules/07_02_push_push_gw_api.rst

From 568982ee2e5a3dbf2f76556475dfdd8fae96998d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 14:49:54 +0100
Subject: [PATCH 121/467] Represent nested deps in targets.yaml along with
 title styles

This will allow us to programatically position .rst snippets *anywhere*
which will for once and for all remove the horrid title level mismatch bugs.

We require this in order to allow people to re-shuffle the spec without
having to adjust the spec itself (e.g. 2 targets with different levels of
nesting).
---
 specification/targets.yaml | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/specification/targets.yaml b/specification/targets.yaml
index 845da99d..b61e2e20 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -20,6 +20,10 @@ groups:
     - modules/04_00_content_repo.rst
     - modules/05_00_end_to_end_encryption.rst
     - modules/06_00_history_visibility.rst
-    - modules/07_00_push_overview.rst
-    - modules/07_01_push_cs_api.rst
-    - modules/07_02_push_push_gw_api.rst
+    - 1: modules/07_00_push_overview.rst
+      2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
+title_styles:
+ - "="
+ - "-"
+ - "~"
+ - "+"

From 6ba9b29b3b4ffe9d19c30302e36813b1835eb117 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 15:04:03 +0100
Subject: [PATCH 122/467] Report all the errors in schemas/check_examples, not
 just the first error.

---
 event-schemas/check_examples.py | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py
index 5fc08f4a..8675396e 100755
--- a/event-schemas/check_examples.py
+++ b/event-schemas/check_examples.py
@@ -3,6 +3,7 @@
 import sys
 import json
 import os
+import traceback
 
 
 def import_error(module, package, debian, error):
@@ -44,19 +45,29 @@ def check_example_file(examplepath, schemapath):
     schema['id'] = fileurl
     try:
         jsonschema.validate(example, schema)
-    except:
+    except Exception as e:
         raise ValueError("Error validating JSON schema for %r %r" % (
             examplepath, schemapath
         ), e)
 
 
 def check_example_dir(exampledir, schemadir):
+    errors = []
     for root, dirs, files in os.walk(exampledir):
         for filename in files:
             examplepath = os.path.join(root, filename)
             schemapath = examplepath.replace(exampledir, schemadir)
-            check_example_file(examplepath, schemapath)
-
+            try:
+                check_example_file(examplepath, schemapath)
+            except Exception as e:
+                errors.append(sys.exc_info())
+    for (exc_type, exc_value, exc_trace) in errors:
+        traceback.print_exception(exc_type, exc_value, exc_trace)
+    if errors:
+        raise ValueError("Error validating examples")
 
 if __name__ == '__main__':
-    check_example_dir("examples", "schema")
+    try:
+        check_example_dir("examples", "schema")
+    except:
+        sys.exit(1)

From 8974b2b67bbc2acf0c482c646be6b07b230a1a15 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 15:05:10 +0100
Subject: [PATCH 123/467] Skip files that start with ".", e.g. vim swp files.

---
 event-schemas/check_examples.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py
index 8675396e..e54d3a1c 100755
--- a/event-schemas/check_examples.py
+++ b/event-schemas/check_examples.py
@@ -55,6 +55,9 @@ def check_example_dir(exampledir, schemadir):
     errors = []
     for root, dirs, files in os.walk(exampledir):
         for filename in files:
+            if filename.startswith("."):
+                # Skip over any vim .swp files.
+                continue
             examplepath = os.path.join(root, filename)
             schemapath = examplepath.replace(exampledir, schemadir)
             try:

From 2c31731262b3258307dd7e5eb8364d71817c4749 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 15:16:02 +0100
Subject: [PATCH 124/467] Add the jenkins command to source control so that we
 can update it without having to fiddle with the jenkins UI.

It also allow us to move files without breaking the CI since we
won't be hard coding the locations of scripts in the jenkins UI.
---
 jenkins.sh | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100755 jenkins.sh

diff --git a/jenkins.sh b/jenkins.sh
new file mode 100755
index 00000000..e1043644
--- /dev/null
+++ b/jenkins.sh
@@ -0,0 +1,9 @@
+#! /bin/bash
+
+set -ex
+
+(cd event-schemas/ && ./check_examples.py)
+(cd api && ./check_examples.py)
+(cd scripts && ./gendoc.py)
+(cd api && npm install && node validator.js -s "client-server/v1")
+(cd event-schemas/ && ./check.sh)

From 8590cc84b51ba81b14b0f73988879368c26e649f Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 15:33:28 +0100
Subject: [PATCH 125/467] Process and extract targets in gendoc.py

---
 scripts/gendoc.py          | 74 +++++++++++++++++++++++++++++++-------
 specification/targets.yaml | 10 +++---
 2 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 3521efed..69c39dcc 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -8,6 +8,7 @@ import re
 import shutil
 import subprocess
 import sys
+import yaml
 
 os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
@@ -24,7 +25,7 @@ SECOND_LEVEL = "-"
 FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
 
 
-def check_valid_section(filename, section):
+def check_valid_section_old(filename, section):
     if not re.match(FILE_FORMAT_MATCHER, filename):
         raise Exception(
             "The filename of " + filename + " does not match the expected format " +
@@ -67,13 +68,19 @@ def check_valid_section(filename, section):
                 "check."
             )
 
-def cat_spec_sections_to(out_file_name):
-    with open(out_file_name, "wb") as outfile:
-        for f in sorted(glob.glob("../specification/*.rst")):
-            with open(f, "rb") as infile:
-                section = infile.read()
-                check_valid_section(os.path.basename(f), section)
-                outfile.write(section)
+def check_valid_section(section):
+    pass
+
+
+def get_rst(file_info, target):
+    pass
+
+def build_spec(target, out_filename):
+    with open(out_filename, "wb") as outfile:
+        for file_info in target["files"]:
+            section = get_rst(file_info, target)
+            check_valid_section(section)
+            outfile.write(section)
 
 
 def rst2html(i, o):
@@ -88,6 +95,7 @@ def rst2html(i, o):
                 settings_overrides=stylesheets
             )
 
+
 def run_through_template(input):
     tmpfile = './tmp/output'
     try:
@@ -107,6 +115,43 @@ def run_through_template(input):
             sys.stderr.write(f.read() + "\n")
         raise
 
+
+def get_build_target(targets_listing, target_name):
+    build_target = {
+        "title_styles": [],
+        "files": []
+    }
+    with open(targets_listing, "r") as targ_file:
+        all_targets = yaml.load(targ_file.read())
+        build_target["title_styles"] = all_targets["title_styles"]
+        target = all_targets["targets"].get(target_name)
+        if not target:
+            raise Exception(
+                "No target by the name '" + target_name + "' exists in '" +
+                targets_listing + "'."
+            )
+        if not isinstance(target.get("files"), list):
+            raise Exception(
+                "Found target but 'files' key is not a list."
+            )
+        resolved_files = []
+        for f in target["files"]:
+            if isinstance(f, basestring) and f.startswith("group:"):
+                # copy across the group of files specified
+                group_name = f[len("group:"):]
+                group = all_targets.get("groups", {}).get(group_name)
+                if not isinstance(group, list):
+                    raise Exception(
+                        "Tried to find group '" + group_name + "' but either " +
+                        "it doesn't exist or it isn't a list of files."
+                    )
+                resolved_files.extend(group)
+            else:
+                resolved_files.append(f)
+        build_target["files"] = resolved_files
+    return build_target
+
+
 def prepare_env():
     try:
         os.makedirs("./gen")
@@ -116,13 +161,17 @@ def prepare_env():
         os.makedirs("./tmp")
     except OSError:
         pass
-    
+
+
 def cleanup_env():
     shutil.rmtree("./tmp")
 
-def main():
+
+def main(target_name):
     prepare_env()
-    cat_spec_sections_to("tmp/full_spec.rst")
+    target = get_build_target("../specification/targets.yaml", target_name)
+    print target
+    build_spec(target=target, out_filename="tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
     run_through_template("tmp/howto.rst")
@@ -131,6 +180,7 @@ def main():
     if "--nodelete" not in sys.argv:
         cleanup_env()
 
+
 if __name__ == '__main__':
     if len(sys.argv) > 1 and sys.argv[1:] != ["--nodelete"]:
         # we accept almost no args, so they don't know what they're doing!
@@ -145,4 +195,4 @@ if __name__ == '__main__':
         print "Requirements:"
         print " - This script requires Jinja2 and rst2html (docutils)."
         sys.exit(0)
-    main()
+    main("main")
diff --git a/specification/targets.yaml b/specification/targets.yaml
index b61e2e20..c4828ef1 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -1,18 +1,18 @@
 targets:
-  main:
-    files:
+  main:  # arbitrary name to identify this build target
+    files:  # the sort order of files to cat
       - 00_00_intro.rst
       - 00_01_feature_profiles.rst
       - 00_02a_events.rst
       - 00_02b_event_signing.rst
       - 01_00_client_server_api.rst
       - 02_00_modules.rst
-      - "group:module"
+      - "group:modules"  # reference a group of files
       - 03_00_application_service_api.rst
       - 04_00_server_server_api.rst
       - 05_00_identity_servers.rst
       - 06_00_appendices.rst
-groups:
+groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
     - modules/01_00_voip_events.rst
     - modules/02_00_typing_notifications.rst
@@ -20,6 +20,7 @@ groups:
     - modules/04_00_content_repo.rst
     - modules/05_00_end_to_end_encryption.rst
     - modules/06_00_history_visibility.rst
+    # Mark a nested file dependency
     - 1: modules/07_00_push_overview.rst
       2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
 title_styles:
@@ -27,3 +28,4 @@ title_styles:
  - "-"
  - "~"
  - "+"
+

From 65ce95249f99fb67e61377e5b35734522a6d7ba3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 16:54:30 +0100
Subject: [PATCH 126/467] cat the spec according to the build target. Remove
 old checks as they are now obsolete since we don't care about the filename

---
 scripts/gendoc.py | 121 ++++++++++++++++++++++++----------------------
 1 file changed, 62 insertions(+), 59 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 69c39dcc..22aaad11 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -16,70 +16,74 @@ stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css"]
 }
 
-title_style_matchers = {
-    "=": re.compile("^=+$"),
-    "-": re.compile("^-+$")
-}
-TOP_LEVEL = "="
-SECOND_LEVEL = "-"
-FILE_FORMAT_MATCHER = re.compile("^[0-9]+_[0-9]{2}[a-z]*_.*\.rst$")
-
-
-def check_valid_section_old(filename, section):
-    if not re.match(FILE_FORMAT_MATCHER, filename):
-        raise Exception(
-            "The filename of " + filename + " does not match the expected format " +
-            "of '##_##_words-go-here.rst'"
-        )
-
-    # we need TWO new lines else the next file's title gets merged
-    # the last paragraph *WITHOUT RST PRODUCING A WARNING*
-    if not section[-2:] == "\n\n":
-        raise Exception(
-            "The file " + filename + " does not end with 2 new lines."
-        )
-
-    # Enforce some rules to reduce the risk of having mismatched title
-    # styles.
-    title_line = section.split("\n")[1]
-    if title_line != (len(title_line) * title_line[0]):
-        raise Exception(
-            "The file " + filename + " doesn't have a title style line on line 2"
-        )
-
-    # anything marked as xx_00_ is the start of a new top-level section
-    if re.match("^[0-9]+_00_", filename):
-        if not title_style_matchers[TOP_LEVEL].match(title_line):
-            raise Exception(
-                "The file " + filename + " is a top-level section because it matches " +
-                "the filename format ##_00_something.rst but has the wrong title " +
-                "style: expected '" + TOP_LEVEL + "' but got '" +
-                title_line[0] + "'"
-            )
-    # anything marked as xx_xx_ is the start of a sub-section
-    elif re.match("^[0-9]+_[0-9]{2}_", filename):
-        if not title_style_matchers[SECOND_LEVEL].match(title_line):
-            raise Exception(
-                "The file " + filename + " is a 2nd-level section because it matches " +
-                "the filename format ##_##_something.rst but has the wrong title " +
-                "style: expected '" + SECOND_LEVEL + "' but got '" +
-                title_line[0] + "' - If this is meant to be a 3rd/4th/5th-level section " +
-                "then use the form '##_##b_something.rst' which will not apply this " +
-                "check."
-            )
-
-def check_valid_section(section):
-    pass
 
+def _list_get(l, index, default=None):
+    try:
+        return l[index]
+    except IndexError:
+        return default
+
+
+def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
+    rst_lines = []
+    title_chars = "".join(title_styles)
+    title_regex = re.compile("^[" + re.escape(title_chars) + "]+$")
+
+    curr_title_level = title_level
+    for i, line in enumerate(file_stream, 1):
+        if title_regex.match(line):
+            line_title_level = title_styles.index(line[0])
+            # Allowed to go 1 deeper or any number shallower
+            if curr_title_level - line_title_level < -1:
+                raise Exception(
+                    ("File '%s' line '%s' has a title " +
+                    "style '%s' which doesn't match one of the " +
+                    "allowed title styles of %s because the " +
+                    "title level before this line was '%s'") %
+                    (filename, (i + 1), line[0], title_styles,
+                    title_styles[curr_title_level])
+                )
+            curr_title_level = line_title_level
+            rst_lines.append(line)
+        else:
+            rst_lines.append(line)
+    return "".join(rst_lines)
+
+
+
+def get_rst(file_info, title_level, title_styles, spec_dir):
+    # string are file paths to RST blobs
+    if isinstance(file_info, basestring):
+        with open(spec_dir + file_info, "r") as f:
+            return load_with_adjusted_titles(file_info, f, title_level, title_styles)
+    # dicts look like {1: filepath, 2: filepath} where the key is the title level
+    elif isinstance(file_info, dict):
+        levels = sorted(file_info.keys())
+        rst = []
+        for l in levels:
+            rst.append(get_rst(file_info[l], l, title_styles, spec_dir))
+        return "".join(rst)
+    # lists are multiple file paths e.g. [filepath, filepath]
+    elif isinstance(file_info, list):
+        rst = []
+        for f in file_info:
+            rst.append(get_rst(f, title_level, title_styles, spec_dir))
+        return "".join(rst)
+    raise Exception(
+        "The following 'file' entry in this target isn't a string, list or dict. " +
+        "It really really should be. Entry: %s" % (file_info,)
+    )
 
-def get_rst(file_info, target):
-    pass
 
 def build_spec(target, out_filename):
     with open(out_filename, "wb") as outfile:
         for file_info in target["files"]:
-            section = get_rst(file_info, target)
-            check_valid_section(section)
+            section = get_rst(
+                file_info=file_info,
+                title_level=0,
+                title_styles=target["title_styles"],
+                spec_dir="../specification/"
+            )
             outfile.write(section)
 
 
@@ -170,7 +174,6 @@ def cleanup_env():
 def main(target_name):
     prepare_env()
     target = get_build_target("../specification/targets.yaml", target_name)
-    print target
     build_spec(target=target, out_filename="tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")

From cb41adee707a04591c93726f5002855febf2543f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:10:23 +0100
Subject: [PATCH 127/467] Fix the swagger host to be "localhost:8008" so that
 it can be used in a "Try it now" setup against localhost

---
 api/client-server/v1/rooms.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 4902560b..4494d357 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -2,7 +2,7 @@ swagger: '2.0'
 info:
   title: "Matrix Client-Server v1 Rooms API"
   version: "1.0.0"
-host: example.com:8008
+host: localhost:8008
 schemes:
   - https
   - http

From 615a9575cbad5401997e89c292cc741e0bf5ff23 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:12:29 +0100
Subject: [PATCH 128/467] SPEC-216: Clarify when the room getters will return
 403

---
 api/client-server/v1/rooms.yaml | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 4494d357..3bd23c4a 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -57,7 +57,9 @@ paths:
         404:
           description: The room has no state with the given type or key.
         403:
-          description: You are not joined to the room.
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 
   "/rooms/{roomId}/state":
     get:
@@ -170,7 +172,9 @@ paths:
               allOf:
                 - "$ref": "core-event-schema/state_event.json"
         403:
-          description: You are not joined to the room.
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 
   "/rooms/{roomId}/initialSync":
     get:
@@ -365,6 +369,10 @@ paths:
                   Whether this room is visible to the ``/publicRooms`` API
                   or not."
             required: ["room_id", "membership"]
+        403:
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 
   "/rooms/{roomId}/members":
     get:
@@ -427,4 +435,8 @@ paths:
                   type: object
                   allOf:
                     - "$ref": "v1-event-schema/m.room.member"
+        403:
+          description: >
+            You aren't a member of the room and weren't previously a
+            member of the room.
 

From ba6c7d267ca4b5265d2eb1ba19933fe6f790090a Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:17:40 +0100
Subject: [PATCH 129/467] SPEC-216: Document the behaviour of the room getters
 when the user has left the room

---
 api/client-server/v1/rooms.yaml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 3bd23c4a..7550c5fe 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -24,7 +24,8 @@ paths:
       description: |-
         Looks up the contents of a state event in a room. If the user is
         joined to the room then the state is taken from the current
-        state of the room.
+        state of the room. If the user has left the room then the state is
+        taken from the state of the room when they left.
       security:
         - accessToken: []
       parameters:
@@ -165,7 +166,9 @@ paths:
             title: RoomState
             description: |-
               If the user is a member of the room this will be the
-              current state of the room as a list of events.
+              current state of the room as a list of events. If the user
+              has left the room then this will be the state of the room
+              when they left as a list of events.
             items:
               title: StateEvent
               type: object
@@ -378,7 +381,7 @@ paths:
     get:
       summary: Get the m.room.member events for the room.
       description:
-        Get the list of members of the room.
+        Get the list of members for this room.
       parameters:
         - in: path
           type: string

From 067363c629bcf140e925980d20bcf8b264407cac Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 21 Sep 2015 17:21:08 +0100
Subject: [PATCH 130/467] Get the desired title levels right; print out the
 actual level used to stdout

---
 scripts/gendoc.py                          | 16 ++++++++-----
 specification/modules/00_modules_intro.rst |  0
 specification/targets.yaml                 | 26 +++++++++-------------
 3 files changed, 21 insertions(+), 21 deletions(-)
 create mode 100644 specification/modules/00_modules_intro.rst

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 2e303ea2..f7bde162 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -50,13 +50,13 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
     return "".join(rst_lines)
 
 
-
 def get_rst(file_info, title_level, title_styles, spec_dir):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
+        print "%s %s" % (">" * (1 + title_level), file_info)
         with open(spec_dir + file_info, "r") as f:
             return load_with_adjusted_titles(file_info, f, title_level, title_styles)
-    # dicts look like {1: filepath, 2: filepath} where the key is the title level
+    # dicts look like {0: filepath, 1: filepath} where the key is the title level
     elif isinstance(file_info, dict):
         levels = sorted(file_info.keys())
         rst = []
@@ -144,12 +144,15 @@ def get_build_target(targets_listing, target_name):
                 # copy across the group of files specified
                 group_name = f[len("group:"):]
                 group = all_targets.get("groups", {}).get(group_name)
-                if not isinstance(group, list):
+                if not group:
                     raise Exception(
-                        "Tried to find group '" + group_name + "' but either " +
-                        "it doesn't exist or it isn't a list of files."
+                        "Tried to find group '" + group_name + "' but it " +
+                        "doesn't exist."
                     )
-                resolved_files.extend(group)
+                if isinstance(group, list):
+                    resolved_files.extend(group)
+                else:
+                    resolved_files.append(group)
             else:
                 resolved_files.append(f)
         build_target["files"] = resolved_files
@@ -173,6 +176,7 @@ def cleanup_env():
 
 def main(target_name):
     prepare_env()
+    print "Building spec [target=%s]" % target_name
     target = get_build_target("../specification/targets.yaml", target_name)
     build_spec(target=target, out_filename="tmp/full_spec.rst")
     run_through_template("tmp/full_spec.rst")
diff --git a/specification/modules/00_modules_intro.rst b/specification/modules/00_modules_intro.rst
new file mode 100644
index 00000000..e69de29b
diff --git a/specification/targets.yaml b/specification/targets.yaml
index c4828ef1..e492ad3c 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -14,18 +14,14 @@ targets:
       - 06_00_appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-    - modules/01_00_voip_events.rst
-    - modules/02_00_typing_notifications.rst
-    - modules/03_00_receipts.rst
-    - modules/04_00_content_repo.rst
-    - modules/05_00_end_to_end_encryption.rst
-    - modules/06_00_history_visibility.rst
-    # Mark a nested file dependency
-    - 1: modules/07_00_push_overview.rst
-      2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
-title_styles:
- - "="
- - "-"
- - "~"
- - "+"
-
+      0: modules/00_modules_intro.rst
+      1:
+         - modules/01_00_voip_events.rst
+         - modules/02_00_typing_notifications.rst
+         - modules/03_00_receipts.rst
+         - modules/04_00_content_repo.rst
+         - modules/05_00_end_to_end_encryption.rst
+         - modules/06_00_history_visibility.rst
+         - 1: modules/07_00_push_overview.rst # Mark a nested file dependency
+           2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
+title_styles: ["=", "-", "~", "+"]

From f60190086a41579d35a7d42caaaee7e69fe27b00 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:30:10 +0100
Subject: [PATCH 131/467] Describe the behaviour of /rooms/{roomId}/member when
 the user has left the room

---
 api/client-server/v1/rooms.yaml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 7550c5fe..e02c75c6 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -391,7 +391,10 @@ paths:
           x-example: "!room:example.com"
       responses:
         200:
-          description: The members of the room.
+          description: |-
+            A list of members of the room. If you are joined to the room then
+            this will be the current members of the room. If you have left te
+            room then this will be the members of the room when you left.
           examples:
             application/json: |-
               {

From 98d91d0c2b3b098618bc91f70079a1a9c64bb546 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 21 Sep 2015 17:31:35 +0100
Subject: [PATCH 132/467] Make the example room id more "random" so that people
 are less likely to think that it is supposed to be human readable

---
 api/client-server/v1/rooms.yaml | 38 ++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index e02c75c6..9300d7d1 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -34,7 +34,7 @@ paths:
           name: roomId
           description: The room to look up the state in.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
         - in: path
           type: string
           name: eventType
@@ -75,7 +75,7 @@ paths:
           name: roomId
           description: The room to look up the state for.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
       responses:
         200:
           description: The current state of the room
@@ -89,7 +89,7 @@ paths:
                     },
                     "event_id": "$14259997323TLwtb:example.com",
                     "origin_server_ts": 1425999732392,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.join_rules",
                     "user_id": "@alice:example.com"
@@ -104,7 +104,7 @@ paths:
                     "event_id": "$1426600438280zExKY:example.com",
                     "membership": "join",
                     "origin_server_ts": 1426600438277,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@alice:example.com",
                     "type": "m.room.member",
                     "user_id": "@alice:example.com"
@@ -116,7 +116,7 @@ paths:
                     },
                     "event_id": "$14259997320KhbwJ:example.com",
                     "origin_server_ts": 1425999732089,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.create",
                     "user_id": "@alice:example.com"
@@ -131,7 +131,7 @@ paths:
                     "event_id": "$1431525430134MxlLX:example.com",
                     "origin_server_ts": 1431525430569,
                     "replaces_state": "$142652023736BSXcM:example.com",
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"
@@ -155,7 +155,7 @@ paths:
                     },
                     "event_id": "$14259997322mqfaq:example.com",
                     "origin_server_ts": 1425999732285,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.power_levels",
                     "user_id": "@alice:example.com"
@@ -192,7 +192,7 @@ paths:
           name: roomId
           description: The room to get the data.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
       responses:
         200:
           description: The current state of the room
@@ -210,7 +210,7 @@ paths:
                       },
                       "event_id": "$14328044851tzTJS:example.com",
                       "origin_server_ts": 1432804485886,
-                      "room_id": "!room:example.com",
+                      "room_id": "!636q39766251:example.com",
                       "type": "m.room.message",
                       "user_id": "@alice:example.com"
                     },
@@ -222,7 +222,7 @@ paths:
                       },
                       "event_id": "$14328044872spjFg:example.com",
                       "origin_server_ts": 1432804487480,
-                      "room_id": "!room:example.com",
+                      "room_id": "!636q39766251:example.com",
                       "type": "m.room.message",
                       "user_id": "@bob:example.com"
                     }
@@ -230,7 +230,7 @@ paths:
                   "end": "s3456_9_0",
                   "start": "t44-3453_9_0"
                 },
-                "room_id": "!room:example.com",
+                "room_id": "!636q39766251:example.com",
                 "state": [
                   {
                     "age": 7148266897,
@@ -239,7 +239,7 @@ paths:
                     },
                     "event_id": "$14259997323TLwtb:example.com",
                     "origin_server_ts": 1425999732392,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.join_rules",
                     "user_id": "@alice:example.com"
@@ -254,7 +254,7 @@ paths:
                     "event_id": "$1426600438280zExKY:example.com",
                     "membership": "join",
                     "origin_server_ts": 1426600438277,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@alice:example.com",
                     "type": "m.room.member",
                     "user_id": "@alice:example.com"
@@ -266,7 +266,7 @@ paths:
                     },
                     "event_id": "$14259997320KhbwJ:example.com",
                     "origin_server_ts": 1425999732089,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.create",
                     "user_id": "@alice:example.com"
@@ -281,7 +281,7 @@ paths:
                     "event_id": "$1431525430134MxlLX:example.com",
                     "origin_server_ts": 1431525430569,
                     "replaces_state": "$142652023736BSXcM:example.com",
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"
@@ -305,7 +305,7 @@ paths:
                     },
                     "event_id": "$14259997322mqfaq:example.com",
                     "origin_server_ts": 1425999732285,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "",
                     "type": "m.room.power_levels",
                     "user_id": "@alice:example.com"
@@ -388,7 +388,7 @@ paths:
           name: roomId
           description: The room to get the member events for.
           required: true
-          x-example: "!room:example.com"
+          x-example: "!636q39766251:example.com"
       responses:
         200:
           description: |-
@@ -409,7 +409,7 @@ paths:
                     "event_id": "$1426600438280zExKY:example.com",
                     "membership": "join",
                     "origin_server_ts": 1426600438277,
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@alice:example.com",
                     "type": "m.room.member",
                     "user_id": "@alice:example.com"
@@ -424,7 +424,7 @@ paths:
                     "event_id": "$1431525430134MxlLX:example.com",
                     "origin_server_ts": 1431525430569,
                     "replaces_state": "$142652023736BSXcM:example.com",
-                    "room_id": "!room:example.com",
+                    "room_id": "!636q39766251:example.com",
                     "state_key": "@bob:example.com",
                     "type": "m.room.member",
                     "user_id": "@bob:example.com"

From f71763b0d3b6d95ed7839f5c9735d21aa59d5d71 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 13:08:15 +0100
Subject: [PATCH 133/467] Implement relative title styles

Templates don't know at what level they will be inserted. Previously, we
hard-coded the title style which is not compatible with the build target
system. Define a set of styles which will be replaced by the gendoc script
when it encounters them:
 '<' : Make this title a sub-heading
 '/' : Make this title a heading at the same level
 '>' : Make this title a super-heading

The build target system is now basically complete and functioning.
---
 scripts/gendoc.py                             | 234 +++++++++++++++---
 specification/00_01_feature_profiles.rst      |   5 +
 specification/02_00_modules.rst               |   5 +
 .../03_00_application_service_api.rst         |   1 +
 specification/04_00_server_server_api.rst     |   4 +-
 specification/modules/00_modules_intro.rst    |   5 +
 specification/targets.yaml                    |  41 +--
 templating/matrix_templates/sections.py       |  35 +--
 .../matrix_templates/templates/msgtypes.tmpl  |   2 +-
 templating/matrix_templates/units.py          |   7 +
 10 files changed, 268 insertions(+), 71 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index f7bde162..308d148f 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -17,57 +17,133 @@ stylesheets = {
 }
 
 
-def _list_get(l, index, default=None):
-    try:
-        return l[index]
-    except IndexError:
-        return default
-
-
+"""
+Read a RST file and replace titles with a different title level if required.
+Args:
+    filename: The name of the file being read (for debugging)
+    file_stream: The open file stream to read from.
+    title_level: The integer which determines the offset to *start* from.
+    title_styles: An array of characters detailing the right title styles to use
+                  e.g. ["=", "-", "~", "+"]
+Returns:
+    string: The file contents with titles adjusted.
+Example:
+    Assume title_styles = ["=", "-", "~", "+"], title_level = 1, and the file
+    when read line-by-line encounters the titles "===", "---", "---", "===", "---".
+    This function will bump every title encountered down a sub-heading e.g.
+    "=" to "-" and "-" to "~" because title_level = 1, so the output would be
+    "---", "~~~", "~~~", "---", "~~~". There is no bumping "up" a title level.
+"""
 def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
     rst_lines = []
     title_chars = "".join(title_styles)
-    title_regex = re.compile("^[" + re.escape(title_chars) + "]+$")
+    title_regex = re.compile("^[" + re.escape(title_chars) + "]{3,}$")
 
-    curr_title_level = title_level
+    prev_line_title_level = 0 # We expect the file to start with '=' titles
+    file_offset = None
+    prev_non_title_line = None
     for i, line in enumerate(file_stream, 1):
-        if title_regex.match(line):
-            line_title_level = title_styles.index(line[0])
-            # Allowed to go 1 deeper or any number shallower
-            if curr_title_level - line_title_level < -1:
-                raise Exception(
-                    ("File '%s' line '%s' has a title " +
-                    "style '%s' which doesn't match one of the " +
-                    "allowed title styles of %s because the " +
-                    "title level before this line was '%s'") %
-                    (filename, (i + 1), line[0], title_styles,
-                    title_styles[curr_title_level])
-                )
-            curr_title_level = line_title_level
+        # ignore anything which isn't a title (e.g. '===============')
+        if not title_regex.match(line):
             rst_lines.append(line)
-        else:
+            prev_non_title_line = line
+            continue
+        # The title underline must match at a minimum the length of the title
+        if len(prev_non_title_line) > len(line):
             rst_lines.append(line)
+            prev_non_title_line = line
+            continue
+
+        line_title_style = line[0]
+        line_title_level = title_styles.index(line_title_style)
+
+        # Not all files will start with "===" and we should be flexible enough
+        # to allow that. The first title we encounter sets the "file offset"
+        # which is added to the title_level desired.
+        if file_offset is None:
+            file_offset = line_title_level
+            if file_offset != 0:
+                print ("     WARNING: %s starts with a title style of '%s' but '%s' " +
+                    "is preferable.") % (filename, line_title_style, title_styles[0])
+
+        # Sanity checks: Make sure that this file is obeying the title levels
+        # specified and bail if it isn't.
+        # The file is allowed to go 1 deeper or any number shallower
+        if prev_line_title_level - line_title_level < -1:
+            raise Exception(
+                ("File '%s' line '%s' has a title " +
+                "style '%s' which doesn't match one of the " +
+                "allowed title styles of %s because the " +
+                "title level before this line was '%s'") %
+                (filename, (i + 1), line_title_style, title_styles,
+                title_styles[prev_line_title_level])
+            )
+        prev_line_title_level = line_title_level
+
+        adjusted_level = (
+            title_level + line_title_level - file_offset
+        )
+
+        # Sanity check: Make sure we can bump down the title and we aren't at the
+        # lowest level already
+        if adjusted_level >= len(title_styles):
+            raise Exception(
+                ("Files '%s' line '%s' has a sub-title level too low and it " +
+                "cannot be adjusted to fit. You can add another level to the " +
+                "'title_styles' key in targets.yaml to fix this.") %
+                (filename, (i + 1))
+            )
+
+        if adjusted_level == line_title_level:
+            # no changes required
+            rst_lines.append(line)
+            continue
+
+        # Adjusting line levels
+        # print (
+        #     "File: %s Adjusting %s to %s because file_offset=%s title_offset=%s" %
+        #     (filename, line_title_style,
+        #         title_styles[adjusted_level],
+        #         file_offset, title_level)
+        # )
+        rst_lines.append(line.replace(
+            line_title_style,
+            title_styles[adjusted_level]
+        ))
+            
     return "".join(rst_lines)
 
 
-def get_rst(file_info, title_level, title_styles, spec_dir):
+def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
         print "%s %s" % (">" * (1 + title_level), file_info)
         with open(spec_dir + file_info, "r") as f:
-            return load_with_adjusted_titles(file_info, f, title_level, title_styles)
+            rst = None
+            if adjust_titles:
+                rst = load_with_adjusted_titles(
+                    file_info, f, title_level, title_styles
+                )
+            else:
+                rst = f.read()
+            if rst[-2:] != "\n\n":
+                raise Exception(
+                    ("File %s should end with TWO new-line characters to ensure " +
+                    "file concatenation works correctly.") % (file_info,)
+                )
+            return rst
     # dicts look like {0: filepath, 1: filepath} where the key is the title level
     elif isinstance(file_info, dict):
         levels = sorted(file_info.keys())
         rst = []
         for l in levels:
-            rst.append(get_rst(file_info[l], l, title_styles, spec_dir))
+            rst.append(get_rst(file_info[l], l, title_styles, spec_dir, adjust_titles))
         return "".join(rst)
     # lists are multiple file paths e.g. [filepath, filepath]
     elif isinstance(file_info, list):
         rst = []
         for f in file_info:
-            rst.append(get_rst(f, title_level, title_styles, spec_dir))
+            rst.append(get_rst(f, title_level, title_styles, spec_dir, adjust_titles))
         return "".join(rst)
     raise Exception(
         "The following 'file' entry in this target isn't a string, list or dict. " +
@@ -82,11 +158,74 @@ def build_spec(target, out_filename):
                 file_info=file_info,
                 title_level=0,
                 title_styles=target["title_styles"],
-                spec_dir="../specification/"
+                spec_dir="../specification/",
+                adjust_titles=True
             )
             outfile.write(section)
 
 
+"""
+Replaces relative title styles with actual title styles.
+
+The templating system has no idea what the right title style is when it produces
+RST because it depends on the build target. As a result, it uses relative title
+styles defined in targets.yaml to say "down a level, up a level, same level".
+
+This function replaces these relative titles with actual title styles from the
+array in targets.yaml.
+"""
+def fix_relative_titles(target, filename, out_filename):
+    title_styles = target["title_styles"] # ["=", "-", "~", "+"]
+    relative_title_chars = [ # ["<", "/", ">"]
+        target["relative_title_styles"]["subtitle"],
+        target["relative_title_styles"]["sametitle"],
+        target["relative_title_styles"]["supertitle"]
+    ]
+    relative_title_matcher = re.compile(
+        "^[" + re.escape("".join(relative_title_chars)) + "]{3,}$"
+    )
+    title_matcher = re.compile(
+        "^[" + re.escape("".join(title_styles)) + "]{3,}$"
+    )
+    current_title_style = None
+    with open(filename, "r") as infile:
+        with open(out_filename, "w") as outfile:
+            for line in infile.readlines():
+                if not relative_title_matcher.match(line):
+                    if title_matcher.match(line):
+                        current_title_style = line[0]
+                    outfile.write(line)
+                    continue
+                line_char = line[0]
+                replacement_char = None
+                current_title_level = title_styles.index(current_title_style)
+                if line_char == target["relative_title_styles"]["subtitle"]:
+                    if (current_title_level + 1) == len(title_styles):
+                        raise Exception(
+                            "Encountered sub-title line style but we can't go " +
+                            "any lower."
+                        )
+                    replacement_char = title_styles[current_title_level + 1]
+                elif line_char == target["relative_title_styles"]["sametitle"]:
+                    replacement_char = title_styles[current_title_level]
+                elif line_char == target["relative_title_styles"]["supertitle"]:
+                    if (current_title_level - 1) < 0:
+                        raise Exception(
+                            "Encountered super-title line style but we can't go " +
+                            "any higher."
+                        )
+                    replacement_char = title_styles[current_title_level - 1]
+                else:
+                    raise Exception(
+                        "Unknown relative line char %s" % (line_char,)
+                    )
+
+                outfile.write(
+                    line.replace(line_char, replacement_char)
+                )
+
+
+
 def rst2html(i, o):
     with open(i, "r") as in_file:
         with open(o, "w") as out_file:
@@ -123,11 +262,13 @@ def run_through_template(input):
 def get_build_target(targets_listing, target_name):
     build_target = {
         "title_styles": [],
+        "relative_title_styles": {},
         "files": []
     }
     with open(targets_listing, "r") as targ_file:
         all_targets = yaml.load(targ_file.read())
         build_target["title_styles"] = all_targets["title_styles"]
+        build_target["relative_title_styles"] = all_targets["relative_title_styles"]
         target = all_targets["targets"].get(target_name)
         if not target:
             raise Exception(
@@ -138,17 +279,30 @@ def get_build_target(targets_listing, target_name):
             raise Exception(
                 "Found target but 'files' key is not a list."
             )
+
+        def get_group(group_id):
+            group_name = group_id[len("group:"):]
+            group = all_targets.get("groups", {}).get(group_name)
+            if not group:
+                raise Exception(
+                    "Tried to find group '" + group_name + "' but it " +
+                    "doesn't exist."
+                )
+            return group
+
         resolved_files = []
         for f in target["files"]:
+            group = None
             if isinstance(f, basestring) and f.startswith("group:"):
-                # copy across the group of files specified
-                group_name = f[len("group:"):]
-                group = all_targets.get("groups", {}).get(group_name)
-                if not group:
-                    raise Exception(
-                        "Tried to find group '" + group_name + "' but it " +
-                        "doesn't exist."
-                    )
+                group = get_group(f)
+            elif isinstance(f, dict):
+                for (k, v) in f.iteritems():
+                    if isinstance(v, basestring) and v.startswith("group:"):
+                        f[k] = get_group(v)
+                resolved_files.append(f)
+                continue
+
+            if group:
                 if isinstance(group, list):
                     resolved_files.extend(group)
                 else:
@@ -178,8 +332,12 @@ def main(target_name):
     prepare_env()
     print "Building spec [target=%s]" % target_name
     target = get_build_target("../specification/targets.yaml", target_name)
-    build_spec(target=target, out_filename="tmp/full_spec.rst")
-    run_through_template("tmp/full_spec.rst")
+    build_spec(target=target, out_filename="tmp/templated_spec.rst")
+    run_through_template("tmp/templated_spec.rst")
+    fix_relative_titles(
+        target=target, filename="tmp/templated_spec.rst",
+        out_filename="tmp/full_spec.rst"
+    )
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
     run_through_template("tmp/howto.rst")
     rst2html("tmp/full_spec.rst", "gen/specification.html")
diff --git a/specification/00_01_feature_profiles.rst b/specification/00_01_feature_profiles.rst
index e69de29b..155e51c5 100644
--- a/specification/00_01_feature_profiles.rst
+++ b/specification/00_01_feature_profiles.rst
@@ -0,0 +1,5 @@
+Feature Profiles
+================
+
+Feature profiles blurb goes here.
+
diff --git a/specification/02_00_modules.rst b/specification/02_00_modules.rst
index e69de29b..ab35fe9f 100644
--- a/specification/02_00_modules.rst
+++ b/specification/02_00_modules.rst
@@ -0,0 +1,5 @@
+Modules
+=======
+
+Modules intro here.
+
diff --git a/specification/03_00_application_service_api.rst b/specification/03_00_application_service_api.rst
index 2674ba44..e982390b 100644
--- a/specification/03_00_application_service_api.rst
+++ b/specification/03_00_application_service_api.rst
@@ -400,3 +400,4 @@ in their content to provide a way for Matrix clients to link into the 'native'
 client from which the event originated.  For instance, this could contain the
 message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
 blog comment traffic in & out of matrix
+
diff --git a/specification/04_00_server_server_api.rst b/specification/04_00_server_server_api.rst
index f5cadabf..8d1f8898 100644
--- a/specification/04_00_server_server_api.rst
+++ b/specification/04_00_server_server_api.rst
@@ -92,7 +92,7 @@ server by querying other servers.
 .. _Perspectives Project: http://perspectives-project.org/
 
 Publishing Keys
-_______________
+^^^^^^^^^^^^^^^
 
 Home servers publish the allowed TLS fingerprints and signing keys in a JSON
 object at ``/_matrix/key/v2/server/{key_id}``. The response contains a list of
@@ -178,7 +178,7 @@ events sent by that server can still be checked.
     }
 
 Querying Keys Through Another Server
-____________________________________
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Servers may offer a query API ``_matrix/key/v2/query/`` for getting the keys
 for another server. This API can be used to GET at list of JSON objects for a
diff --git a/specification/modules/00_modules_intro.rst b/specification/modules/00_modules_intro.rst
index e69de29b..fdfcbbdc 100644
--- a/specification/modules/00_modules_intro.rst
+++ b/specification/modules/00_modules_intro.rst
@@ -0,0 +1,5 @@
+Modules
+=======
+
+Modules blurb goes here.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index e492ad3c..7bd3d882 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -2,26 +2,37 @@ targets:
   main:  # arbitrary name to identify this build target
     files:  # the sort order of files to cat
       - 00_00_intro.rst
-      - 00_01_feature_profiles.rst
-      - 00_02a_events.rst
-      - 00_02b_event_signing.rst
+      - { 1: 00_01_feature_profiles.rst }
+      - { 1: 00_02a_events.rst }
+      - { 1: 00_02b_event_signing.rst }
       - 01_00_client_server_api.rst
       - 02_00_modules.rst
-      - "group:modules"  # reference a group of files
+      - { 1: "group:modules" }  # reference a group of files
       - 03_00_application_service_api.rst
       - 04_00_server_server_api.rst
       - 05_00_identity_servers.rst
       - 06_00_appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-      0: modules/00_modules_intro.rst
-      1:
-         - modules/01_00_voip_events.rst
-         - modules/02_00_typing_notifications.rst
-         - modules/03_00_receipts.rst
-         - modules/04_00_content_repo.rst
-         - modules/05_00_end_to_end_encryption.rst
-         - modules/06_00_history_visibility.rst
-         - 1: modules/07_00_push_overview.rst # Mark a nested file dependency
-           2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst]
-title_styles: ["=", "-", "~", "+"]
+    - modules/00_modules_intro.rst
+    - modules/01_00_voip_events.rst
+    - modules/02_00_typing_notifications.rst
+    - modules/03_00_receipts.rst
+    - modules/04_00_content_repo.rst
+    - modules/05_00_end_to_end_encryption.rst
+    - modules/06_00_history_visibility.rst
+    - modules/07_00_push_overview.rst
+    - { 2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst] }
+
+title_styles: ["=", "-", "~", "+", "^"]
+
+# The templating system doesn't know the right title style to use when generating
+# RST. These symbols are 'relative' to say "make a sub-title" (-1), "make a title
+# at the same level (0)", or "make a title one above (+1)". The gendoc script
+# will inspect this file and replace these relative styles with actual title
+# styles. The templating system will also inspect this file to know which symbols
+# to inject.
+relative_title_styles:
+  subtitle: "<"
+  sametitle: "/"
+  supertitle: ">"
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 729157bb..6072222a 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -23,10 +23,13 @@ class MatrixSections(Sections):
         spec_meta = self.units.get("spec_meta")
         return spec_meta["changelog"]
 
-    def _render_events(self, filterFn, sortFn, title_kind="~"):
+    def _render_events(self, filterFn, sortFn):
         template = self.env.get_template("events.tmpl")
         examples = self.units.get("event_examples")
         schemas = self.units.get("event_schemas")
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
         sections = []
         for event_name in sortFn(schemas):
             if not filterFn(event_name):
@@ -34,14 +37,16 @@ class MatrixSections(Sections):
             sections.append(template.render(
                 example=examples[event_name], 
                 event=schemas[event_name],
-                title_kind=title_kind
+                title_kind=subtitle_title_char
             ))
         return "\n\n".join(sections)
 
-    def _render_http_api_group(self, group, sortFnOrPathList=None, 
-                               title_kind="-"):
+    def _render_http_api_group(self, group, sortFnOrPathList=None):
         template = self.env.get_template("http-api.tmpl")
         http_api = self.units.get("swagger_apis")[group]["__meta"]
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
         sections = []
         endpoints = []
         if sortFnOrPathList:
@@ -67,15 +72,14 @@ class MatrixSections(Sections):
         for endpoint in endpoints:
             sections.append(template.render(
                 endpoint=endpoint,
-                title_kind=title_kind
+                title_kind=subtitle_title_char
             ))
         return "\n\n".join(sections)
 
     def render_profile_http_api(self):
         return self._render_http_api_group(
             "profile", 
-            sortFnOrPathList=["displayname", "avatar_url"],
-            title_kind="~"
+            sortFnOrPathList=["displayname", "avatar_url"]
         )
 
     def render_sync_http_api(self):
@@ -86,20 +90,17 @@ class MatrixSections(Sections):
     def render_presence_http_api(self):
         return self._render_http_api_group(
             "presence",
-            sortFnOrPathList=["status"],
-            title_kind="~"
+            sortFnOrPathList=["status"]
         )
 
     def render_membership_http_api(self):
         return self._render_http_api_group(
-            "membership",
-            title_kind="~"
+            "membership"
         )
 
     def render_login_http_api(self):
         return self._render_http_api_group(
-            "login",
-            title_kind="~"
+            "login"
         )
 
     def render_room_events(self):
@@ -114,6 +115,9 @@ class MatrixSections(Sections):
         template = self.env.get_template("msgtypes.tmpl")
         examples = self.units.get("event_examples")
         schemas = self.units.get("event_schemas")
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
         sections = []
         msgtype_order = [
             "m.room.message#m.text", "m.room.message#m.emote",
@@ -129,7 +133,8 @@ class MatrixSections(Sections):
                 continue
             sections.append(template.render(
                 example=examples[event_name], 
-                event=schemas[event_name]
+                event=schemas[event_name],
+                title_kind=subtitle_title_char
             ))
         return "\n\n".join(sections)
 
@@ -150,7 +155,7 @@ class MatrixSections(Sections):
     def render_presence_events(self):
         def filterFn(eventType):
             return eventType.startswith("m.presence")
-        return self._render_events(filterFn, sorted, title_kind="+")
+        return self._render_events(filterFn, sorted)
 
     def _render_ce_type(self, type):
         template = self.env.get_template("common-event-fields.tmpl")
diff --git a/templating/matrix_templates/templates/msgtypes.tmpl b/templating/matrix_templates/templates/msgtypes.tmpl
index 29e86160..f7862451 100644
--- a/templating/matrix_templates/templates/msgtypes.tmpl
+++ b/templating/matrix_templates/templates/msgtypes.tmpl
@@ -1,5 +1,5 @@
 ``{{event.msgtype}}``
-{{(4 + event.msgtype | length) * '+'}}
+{{(4 + event.msgtype | length) * title_kind}}
 {{event.desc | wrap(80)}}
 {% for table in event.content_fields -%}
 {{"``"+table.title+"``" if table.title else "" }}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 1a9d981a..0096bbfa 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -13,6 +13,7 @@ V1_EVENT_EXAMPLES = "../event-schemas/examples/v1"
 V1_EVENT_SCHEMA = "../event-schemas/schema/v1"
 CORE_EVENT_SCHEMA = "../event-schemas/schema/v1/core-event-schema"
 CHANGELOG = "../CHANGELOG.rst"
+TARGETS = "../specification/targets.yaml"
 
 ROOM_EVENT = "core-event-schema/room_event.json"
 STATE_EVENT = "core-event-schema/state_event.json"
@@ -466,6 +467,12 @@ class MatrixUnits(Units):
             "changelog": "".join(changelog_lines)
         }
 
+
+    def load_spec_targets(self):
+        with open(TARGETS, "r") as f:
+            return yaml.load(f.read())
+
+
     def load_git_version(self):
         null = open(os.devnull, 'w')
         cwd = os.path.dirname(os.path.abspath(__file__))

From e8cdfcbba28223055a182e06c1ee1a4f04cc3738 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 14:01:55 +0100
Subject: [PATCH 134/467] Rename spec files to not have numbers everywhere;
 update targets.yaml

---
 ..._event_signing.rst => 0-event_signing.rst} |  0
 .../{00_02a_events.rst => 0-events.rst}       |  0
 ...re_profiles.rst => 0-feature_profiles.rst} |  0
 .../{00_00_intro.rst => 0-intro.rst}          |  0
 ...server_api.rst => 1-client_server_api.rst} |  0
 .../{02_00_modules.rst => 2-modules.rst}      |  0
 ..._api.rst => 3-application_service_api.rst} |  0
 ...server_api.rst => 4-server_server_api.rst} |  0
 ...ity_servers.rst => 5-identity_servers.rst} |  0
 ...{06_00_appendices.rst => 6-appendices.rst} |  0
 ...4_00_content_repo.rst => content_repo.rst} |  0
 ...cryption.rst => end_to_end_encryption.rst} |  0
 ..._visibility.rst => history_visibility.rst} |  0
 .../{00_modules_intro.rst => intro.rst}       |  0
 ...{07_01_push_cs_api.rst => push_cs_api.rst} |  0
 ...00_push_overview.rst => push_overview.rst} |  0
 ...h_push_gw_api.rst => push_push_gw_api.rst} |  0
 .../{03_00_receipts.rst => receipts.rst}      |  0
 ...fications.rst => typing_notifications.rst} |  0
 ...{01_00_voip_events.rst => voip_events.rst} |  0
 specification/targets.yaml                    | 38 +++++++++----------
 21 files changed, 19 insertions(+), 19 deletions(-)
 rename specification/{00_02b_event_signing.rst => 0-event_signing.rst} (100%)
 rename specification/{00_02a_events.rst => 0-events.rst} (100%)
 rename specification/{00_01_feature_profiles.rst => 0-feature_profiles.rst} (100%)
 rename specification/{00_00_intro.rst => 0-intro.rst} (100%)
 rename specification/{01_00_client_server_api.rst => 1-client_server_api.rst} (100%)
 rename specification/{02_00_modules.rst => 2-modules.rst} (100%)
 rename specification/{03_00_application_service_api.rst => 3-application_service_api.rst} (100%)
 rename specification/{04_00_server_server_api.rst => 4-server_server_api.rst} (100%)
 rename specification/{05_00_identity_servers.rst => 5-identity_servers.rst} (100%)
 rename specification/{06_00_appendices.rst => 6-appendices.rst} (100%)
 rename specification/modules/{04_00_content_repo.rst => content_repo.rst} (100%)
 rename specification/modules/{05_00_end_to_end_encryption.rst => end_to_end_encryption.rst} (100%)
 rename specification/modules/{06_00_history_visibility.rst => history_visibility.rst} (100%)
 rename specification/modules/{00_modules_intro.rst => intro.rst} (100%)
 rename specification/modules/{07_01_push_cs_api.rst => push_cs_api.rst} (100%)
 rename specification/modules/{07_00_push_overview.rst => push_overview.rst} (100%)
 rename specification/modules/{07_02_push_push_gw_api.rst => push_push_gw_api.rst} (100%)
 rename specification/modules/{03_00_receipts.rst => receipts.rst} (100%)
 rename specification/modules/{02_00_typing_notifications.rst => typing_notifications.rst} (100%)
 rename specification/modules/{01_00_voip_events.rst => voip_events.rst} (100%)

diff --git a/specification/00_02b_event_signing.rst b/specification/0-event_signing.rst
similarity index 100%
rename from specification/00_02b_event_signing.rst
rename to specification/0-event_signing.rst
diff --git a/specification/00_02a_events.rst b/specification/0-events.rst
similarity index 100%
rename from specification/00_02a_events.rst
rename to specification/0-events.rst
diff --git a/specification/00_01_feature_profiles.rst b/specification/0-feature_profiles.rst
similarity index 100%
rename from specification/00_01_feature_profiles.rst
rename to specification/0-feature_profiles.rst
diff --git a/specification/00_00_intro.rst b/specification/0-intro.rst
similarity index 100%
rename from specification/00_00_intro.rst
rename to specification/0-intro.rst
diff --git a/specification/01_00_client_server_api.rst b/specification/1-client_server_api.rst
similarity index 100%
rename from specification/01_00_client_server_api.rst
rename to specification/1-client_server_api.rst
diff --git a/specification/02_00_modules.rst b/specification/2-modules.rst
similarity index 100%
rename from specification/02_00_modules.rst
rename to specification/2-modules.rst
diff --git a/specification/03_00_application_service_api.rst b/specification/3-application_service_api.rst
similarity index 100%
rename from specification/03_00_application_service_api.rst
rename to specification/3-application_service_api.rst
diff --git a/specification/04_00_server_server_api.rst b/specification/4-server_server_api.rst
similarity index 100%
rename from specification/04_00_server_server_api.rst
rename to specification/4-server_server_api.rst
diff --git a/specification/05_00_identity_servers.rst b/specification/5-identity_servers.rst
similarity index 100%
rename from specification/05_00_identity_servers.rst
rename to specification/5-identity_servers.rst
diff --git a/specification/06_00_appendices.rst b/specification/6-appendices.rst
similarity index 100%
rename from specification/06_00_appendices.rst
rename to specification/6-appendices.rst
diff --git a/specification/modules/04_00_content_repo.rst b/specification/modules/content_repo.rst
similarity index 100%
rename from specification/modules/04_00_content_repo.rst
rename to specification/modules/content_repo.rst
diff --git a/specification/modules/05_00_end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst
similarity index 100%
rename from specification/modules/05_00_end_to_end_encryption.rst
rename to specification/modules/end_to_end_encryption.rst
diff --git a/specification/modules/06_00_history_visibility.rst b/specification/modules/history_visibility.rst
similarity index 100%
rename from specification/modules/06_00_history_visibility.rst
rename to specification/modules/history_visibility.rst
diff --git a/specification/modules/00_modules_intro.rst b/specification/modules/intro.rst
similarity index 100%
rename from specification/modules/00_modules_intro.rst
rename to specification/modules/intro.rst
diff --git a/specification/modules/07_01_push_cs_api.rst b/specification/modules/push_cs_api.rst
similarity index 100%
rename from specification/modules/07_01_push_cs_api.rst
rename to specification/modules/push_cs_api.rst
diff --git a/specification/modules/07_00_push_overview.rst b/specification/modules/push_overview.rst
similarity index 100%
rename from specification/modules/07_00_push_overview.rst
rename to specification/modules/push_overview.rst
diff --git a/specification/modules/07_02_push_push_gw_api.rst b/specification/modules/push_push_gw_api.rst
similarity index 100%
rename from specification/modules/07_02_push_push_gw_api.rst
rename to specification/modules/push_push_gw_api.rst
diff --git a/specification/modules/03_00_receipts.rst b/specification/modules/receipts.rst
similarity index 100%
rename from specification/modules/03_00_receipts.rst
rename to specification/modules/receipts.rst
diff --git a/specification/modules/02_00_typing_notifications.rst b/specification/modules/typing_notifications.rst
similarity index 100%
rename from specification/modules/02_00_typing_notifications.rst
rename to specification/modules/typing_notifications.rst
diff --git a/specification/modules/01_00_voip_events.rst b/specification/modules/voip_events.rst
similarity index 100%
rename from specification/modules/01_00_voip_events.rst
rename to specification/modules/voip_events.rst
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 7bd3d882..df61e7a1 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -1,28 +1,28 @@
 targets:
   main:  # arbitrary name to identify this build target
     files:  # the sort order of files to cat
-      - 00_00_intro.rst
-      - { 1: 00_01_feature_profiles.rst }
-      - { 1: 00_02a_events.rst }
-      - { 1: 00_02b_event_signing.rst }
-      - 01_00_client_server_api.rst
-      - 02_00_modules.rst
+      - 0-intro.rst
+      - { 1: 0-feature_profiles.rst }
+      - { 1: 0-events.rst }
+      - { 1: 0-event_signing.rst }
+      - 1-client_server_api.rst
+      - 2-modules.rst
       - { 1: "group:modules" }  # reference a group of files
-      - 03_00_application_service_api.rst
-      - 04_00_server_server_api.rst
-      - 05_00_identity_servers.rst
-      - 06_00_appendices.rst
+      - 3-application_service_api.rst
+      - 4-server_server_api.rst
+      - 5-identity_servers.rst
+      - 6-appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-    - modules/00_modules_intro.rst
-    - modules/01_00_voip_events.rst
-    - modules/02_00_typing_notifications.rst
-    - modules/03_00_receipts.rst
-    - modules/04_00_content_repo.rst
-    - modules/05_00_end_to_end_encryption.rst
-    - modules/06_00_history_visibility.rst
-    - modules/07_00_push_overview.rst
-    - { 2: [modules/07_01_push_cs_api.rst , modules/07_02_push_push_gw_api.rst] }
+    - modules/intro.rst
+    - modules/voip_events.rst
+    - modules/typing_notifications.rst
+    - modules/receipts.rst
+    - modules/content_repo.rst
+    - modules/end_to_end_encryption.rst
+    - modules/history_visibility.rst
+    - modules/push_overview.rst
+    - { 2: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
 
 title_styles: ["=", "-", "~", "+", "^"]
 

From 0c0ac3e814c763e271d7a0c9b5c9b5e824b2f2f3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 14:07:23 +0100
Subject: [PATCH 135/467] Fix common event fields template to use subtitle char

---
 specification/modules/intro.rst                            | 5 -----
 specification/targets.yaml                                 | 1 -
 templating/matrix_templates/sections.py                    | 7 ++++++-
 .../matrix_templates/templates/common-event-fields.tmpl    | 2 +-
 4 files changed, 7 insertions(+), 8 deletions(-)
 delete mode 100644 specification/modules/intro.rst

diff --git a/specification/modules/intro.rst b/specification/modules/intro.rst
deleted file mode 100644
index fdfcbbdc..00000000
--- a/specification/modules/intro.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Modules
-=======
-
-Modules blurb goes here.
-
diff --git a/specification/targets.yaml b/specification/targets.yaml
index df61e7a1..96527dae 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -14,7 +14,6 @@ targets:
       - 6-appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
-    - modules/intro.rst
     - modules/voip_events.rst
     - modules/typing_notifications.rst
     - modules/receipts.rst
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 6072222a..78f011b9 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -160,7 +160,12 @@ class MatrixSections(Sections):
     def _render_ce_type(self, type):
         template = self.env.get_template("common-event-fields.tmpl")
         ce_types = self.units.get("common_event_fields")
-        return template.render(common_event=ce_types[type])
+        subtitle_title_char = self.units.get("spec_targets")[
+            "relative_title_styles"
+        ]["subtitle"]
+        return template.render(
+            common_event=ce_types[type], title_kind=subtitle_title_char
+        )
 
     def render_common_event_fields(self):
         return self._render_ce_type("event")
diff --git a/templating/matrix_templates/templates/common-event-fields.tmpl b/templating/matrix_templates/templates/common-event-fields.tmpl
index 2a3f7ec5..3f16be3d 100644
--- a/templating/matrix_templates/templates/common-event-fields.tmpl
+++ b/templating/matrix_templates/templates/common-event-fields.tmpl
@@ -1,5 +1,5 @@
 {{common_event.title}} Fields
-{{(7 + common_event.title | length) * '-'}}
+{{(7 + common_event.title | length) * title_kind}}
 
 {{common_event.desc | wrap(80)}}
 

From 056b5eba22e94b613c50b6255b23152553bced2e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:01:22 +0100
Subject: [PATCH 136/467] Partially handle representing top-level array
 responses

If an HTTP API returned a top-level array response, the templating system
would fail to create a table for it. This is now partially fixed by pulling
out the type of the elements (no recursion is done to populate nested tables)
---
 templating/matrix_templates/units.py | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 84c96ce3..975f7c80 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -247,7 +247,8 @@ class MatrixUnits(Units):
                             "rows": [{
                                 "key": good_response["schema"].get("name", ""),
                                 "type": res_type,
-                                "desc": res.get("description", "")
+                                "desc": res.get("description", ""),
+                                "req_str": ""
                             }]
                         })
                     elif res_type and Units.prop(good_response, "schema/properties"):
@@ -257,6 +258,24 @@ class MatrixUnits(Units):
                         for table in res_tables:
                             if "no-table" not in table:
                                 endpoint["res_tables"].append(table)
+                    elif res_type and Units.prop(good_response, "schema/items"):
+                        # response is an array:
+                        # FIXME: Doesn't recurse at all.
+                        schema = good_response["schema"]
+                        array_type = Units.prop(schema, "items/type")
+                        if Units.prop(schema, "items/allOf"):
+                            array_type = (
+                                Units.prop(schema, "items/title")
+                            )
+                        endpoint["res_tables"].append({
+                            "title": schema.get("title", ""),
+                            "rows": [{
+                                "key": "N/A",
+                                "type": ("[%s]" % array_type),
+                                "desc": schema.get("description", ""),
+                                "req_str": ""
+                            }]
+                        })
 
                 endpoints.append(endpoint)
 

From 388aeefac000d89e9b775e1fb7905b8e8f178ae2 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:07:19 +0100
Subject: [PATCH 137/467] Remove obsolete key

---
 templating/matrix_templates/sections.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index 15dca9b1..cacaa06a 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -105,8 +105,7 @@ class MatrixSections(Sections):
 
     def render_rooms_http_api(self):
         return self._render_http_api_group(
-            "rooms",
-            title_kind="+"
+            "rooms"
         )
 
     def render_room_events(self):

From b21859836d1b6ca0df6f4072c393211706eae9a8 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:11:16 +0100
Subject: [PATCH 138/467] Print stdout of build.py in verbose mode

---
 scripts/gendoc.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 308d148f..6c95e33c 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -243,9 +243,9 @@ def run_through_template(input):
     tmpfile = './tmp/output'
     try:
         with open(tmpfile, 'w') as out:
-            subprocess.check_output(
+            print subprocess.check_output(
                 [
-                    'python', 'build.py',
+                    'python', 'build.py', "-v",
                     "-i", "matrix_templates",
                     "-o", "../scripts/tmp",
                     "../scripts/"+input

From 16693a644aead93c67ad1f10d3afb1d7fc0169a5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 16:59:49 +0100
Subject: [PATCH 139/467] Various review fixes

---
 scripts/gendoc.py                    | 105 ++++++++++++++++-----------
 specification/0-feature_profiles.rst |   2 -
 specification/2-modules.rst          |   2 -
 3 files changed, 62 insertions(+), 47 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 6c95e33c..26ee5a48 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -118,7 +118,7 @@ def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
         print "%s %s" % (">" * (1 + title_level), file_info)
-        with open(spec_dir + file_info, "r") as f:
+        with open(os.path.join(spec_dir, file_info), "r") as f:
             rst = None
             if adjust_titles:
                 rst = load_with_adjusted_titles(
@@ -175,8 +175,8 @@ This function replaces these relative titles with actual title styles from the
 array in targets.yaml.
 """
 def fix_relative_titles(target, filename, out_filename):
-    title_styles = target["title_styles"] # ["=", "-", "~", "+"]
-    relative_title_chars = [ # ["<", "/", ">"]
+    title_styles = target["title_styles"]
+    relative_title_chars = [
         target["relative_title_styles"]["subtitle"],
         target["relative_title_styles"]["sametitle"],
         target["relative_title_styles"]["supertitle"]
@@ -259,6 +259,17 @@ def run_through_template(input):
         raise
 
 
+"""
+Extract and resolve groups for the given target in the given targets listing.
+Args:
+    targets_listing (str): The path to a YAML file containing a list of targets
+    target_name (str): The name of the target to extract from the listings.
+Returns:
+    dict: Containing "filees" (a list of file paths), "relative_title_styles"
+          (a dict of relative style keyword to title character) and "title_styles"
+          (a list of characters which represent the global title style to follow,
+           with the top section title first, the second section second, and so on.)
+"""
 def get_build_target(targets_listing, target_name):
     build_target = {
         "title_styles": [],
@@ -267,49 +278,57 @@ def get_build_target(targets_listing, target_name):
     }
     with open(targets_listing, "r") as targ_file:
         all_targets = yaml.load(targ_file.read())
-        build_target["title_styles"] = all_targets["title_styles"]
-        build_target["relative_title_styles"] = all_targets["relative_title_styles"]
-        target = all_targets["targets"].get(target_name)
-        if not target:
-            raise Exception(
-                "No target by the name '" + target_name + "' exists in '" +
-                targets_listing + "'."
-            )
-        if not isinstance(target.get("files"), list):
+
+    build_target["title_styles"] = all_targets["title_styles"]
+    build_target["relative_title_styles"] = all_targets["relative_title_styles"]
+    target = all_targets["targets"].get(target_name)
+    if not target:
+        raise Exception(
+            "No target by the name '" + target_name + "' exists in '" +
+            targets_listing + "'."
+        )
+    if not isinstance(target.get("files"), list):
+        raise Exception(
+            "Found target but 'files' key is not a list."
+        )
+
+    def get_group(group_id):
+        group_name = group_id[len("group:"):]
+        group = all_targets.get("groups", {}).get(group_name)
+        if not group:
             raise Exception(
-                "Found target but 'files' key is not a list."
+                "Tried to find group '" + group_name + "' but it " +
+                "doesn't exist."
             )
-
-        def get_group(group_id):
-            group_name = group_id[len("group:"):]
-            group = all_targets.get("groups", {}).get(group_name)
-            if not group:
-                raise Exception(
-                    "Tried to find group '" + group_name + "' but it " +
-                    "doesn't exist."
-                )
-            return group
-
-        resolved_files = []
-        for f in target["files"]:
-            group = None
-            if isinstance(f, basestring) and f.startswith("group:"):
-                group = get_group(f)
-            elif isinstance(f, dict):
-                for (k, v) in f.iteritems():
-                    if isinstance(v, basestring) and v.startswith("group:"):
-                        f[k] = get_group(v)
-                resolved_files.append(f)
-                continue
-
-            if group:
-                if isinstance(group, list):
-                    resolved_files.extend(group)
-                else:
-                    resolved_files.append(group)
+        return group
+
+    resolved_files = []
+    for file_entry in target["files"]:
+        # file_entry is a group id
+        if isinstance(file_entry, basestring) and file_entry.startswith("group:"):
+            group = get_group(file_entry)
+            # The group may be resolved to a list of file entries, in which case
+            # we want to extend the array to insert each of them rather than
+            # insert the entire list as a single element (which is what append does)
+            if isinstance(group, list):
+                resolved_files.extend(group)
             else:
-                resolved_files.append(f)
-        build_target["files"] = resolved_files
+                resolved_files.append(group)
+        # file_entry is a dict which has more file entries as values
+        elif isinstance(file_entry, dict):
+            resolved_entry = {}
+            for (k, v) in file_entry.iteritems():
+                if isinstance(v, basestring) and v.startswith("group:"):
+                    resolved_entry[k] = get_group(v)
+                else:
+                    # map across without editing (e.g. normal file path)
+                    resolved_entry[k] = v
+            resolved_files.append(resolved_entry)
+            continue
+        # file_entry is just a plain ol' file path
+        else:
+            resolved_files.append(file_entry)
+    build_target["files"] = resolved_files
     return build_target
 
 
diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 155e51c5..234e14db 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -1,5 +1,3 @@
 Feature Profiles
 ================
 
-Feature profiles blurb goes here.
-
diff --git a/specification/2-modules.rst b/specification/2-modules.rst
index ab35fe9f..0aad77e1 100644
--- a/specification/2-modules.rst
+++ b/specification/2-modules.rst
@@ -1,5 +1,3 @@
 Modules
 =======
 
-Modules intro here.
-

From b49cb57fe596eb942759f67a054906386854867a Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 22 Sep 2015 17:32:31 +0100
Subject: [PATCH 140/467] Move events sections to CS API

---
 specification/targets.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/targets.yaml b/specification/targets.yaml
index 96527dae..92ad9bb4 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -3,9 +3,9 @@ targets:
     files:  # the sort order of files to cat
       - 0-intro.rst
       - { 1: 0-feature_profiles.rst }
+      - 1-client_server_api.rst
       - { 1: 0-events.rst }
       - { 1: 0-event_signing.rst }
-      - 1-client_server_api.rst
       - 2-modules.rst
       - { 1: "group:modules" }  # reference a group of files
       - 3-application_service_api.rst

From 5b134119bd53200f00da3b52557ff88b739f8ad5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 09:59:54 +0100
Subject: [PATCH 141/467] Add presence module; fix relative title bug

If a relative title appeared after an HTTP API table, it
would insert the wrong level because it thought that part
of the table was a title.
---
 specification/1-client_server_api.rst         |  6 --
 specification/modules/presence.rst            | 60 +++++++++++++++++++
 specification/targets.yaml                    |  1 +
 .../matrix_templates/templates/http-api.tmpl  |  2 +-
 4 files changed, 62 insertions(+), 7 deletions(-)
 create mode 100644 specification/modules/presence.rst

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index ad39dafc..93e3cb90 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1054,12 +1054,6 @@ medium
 address
   The textual address of the 3pid, eg. the email address
 
-Presence
---------
-.. TODO-spec
-  - Define how users receive presence invites, and how they accept/decline them
-
-{{presence_http_api}}
 
 Profiles
 --------
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
new file mode 100644
index 00000000..6f644879
--- /dev/null
+++ b/specification/modules/presence.rst
@@ -0,0 +1,60 @@
+Presence Events
+===============
+
+{{presence_events}}
+ 
+Each user has the concept of presence information. This encodes the
+"availability" of that user, suitable for display on other user's clients.
+This is transmitted as an ``m.presence`` event and is one of the few events
+which are sent *outside the context of a room*. The basic piece of presence
+information is represented by the ``presence`` key, which is an enum of one
+of the following:
+
+      - ``online`` : The default state when the user is connected to an event
+        stream.
+      - ``unavailable`` : The user is not reachable at this time.
+      - ``offline`` : The user is not connected to an event stream.
+      - ``free_for_chat`` : The user is generally willing to receive messages
+        moreso than default.
+      - ``hidden`` : Behaves as offline, but allows the user to see the client
+        state anyway and generally interact with client features. (Not yet
+        implemented in synapse).
+
+In addition, the server maintains a timestamp of the last time it saw a
+pro-active event from the user; either sending a message to a room, or
+changing presence state from a lower to a higher level of availability
+(thus: changing state from ``unavailable`` to ``online`` counts as a
+proactive event, whereas in the other direction it will not). This timestamp
+is presented via a key called ``last_active_ago``, which gives the relative
+number of milliseconds since the message is generated/emitted that the user
+was last seen active.
+
+Presence HTTP API
+-----------------
+.. TODO-spec
+  - Define how users receive presence invites, and how they accept/decline them
+
+{{presence_http_api}}
+    
+
+Events on Change of Profile Information
+---------------------------------------
+Because the profile displayname and avatar information are likely to be used in
+many places of a client's display, changes to these fields cause an automatic
+propagation event to occur, informing likely-interested parties of the new
+values. This change is conveyed using two separate mechanisms:
+
+ - a ``m.room.member`` event is sent to every room the user is a member of,
+   to update the ``displayname`` and ``avatar_url``.
+ - a ``m.presence`` presence status update is sent, again containing the new values of the
+   ``displayname`` and ``avatar_url`` keys, in addition to the required
+   ``presence`` key containing the current presence state of the user.
+
+Both of these should be done automatically by the home server when a user
+successfully changes their displayname or avatar URL fields.
+
+Additionally, when home servers emit room membership events for their own
+users, they should include the displayname and avatar URL fields in these
+events so that clients already have these details to hand, and do not have to
+perform extra roundtrips to query it.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 92ad9bb4..454f23b9 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -17,6 +17,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/voip_events.rst
     - modules/typing_notifications.rst
     - modules/receipts.rst
+    - modules/presence.rst
     - modules/content_repo.rst
     - modules/end_to_end_encryption.rst
     - modules/history_visibility.rst
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index a03ffa7d..eb3f3e64 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -18,7 +18,7 @@ Request format:
 ================== ================= ===========================================
 {% for loc in endpoint.req_param_by_loc -%}
 *{{loc}} parameters*
---------------------------------------------------------------------------------
+-------------------------------------------------------------------------------- 
 {% for param in endpoint.req_param_by_loc[loc] -%}
 {{param.key}}{{param.type|indent(19-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(37)}}
 {% endfor -%}

From 51153462971d3c6ffd4fae4672504e857532c5c2 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 10:48:49 +0100
Subject: [PATCH 142/467] Add instant_messaging module; modify batesian section
 rules

Previously, all `m.room.*` events were wodged into `{{room_events}}` which
isn't great when you want to pull specific ones out. Batesian had a 1:1
mapping of `render_foo()` to a section `{{foo}}`, and having to constantly
add functions for new types is a PITA. Batesian now supports returning a
`dict` instead of a section `string` where the keys are the `{{foo}}` and
the value is what will be inserted. Also add conflicting section key checks
to avoid multiple definitions of the same `{{foo}}`. Define dicts for
event schemata and swagger HTTP APIs.

Using this new feature, split out the instant messaging stuff from the events
section, and replace `{{room_events}}` with a list of specific events e.g.
`{{m_room_member_event}}`.
---
 specification/0-events.rst                  | 82 ++++-----------------
 specification/modules/instant_messaging.rst | 27 +++++++
 specification/modules/presence.rst          | 11 ++-
 specification/targets.yaml                  |  1 +
 templating/batesian/sections.py             | 38 ++++++++--
 templating/matrix_templates/sections.py     | 58 +++++++--------
 6 files changed, 111 insertions(+), 106 deletions(-)
 create mode 100644 specification/modules/instant_messaging.rst

diff --git a/specification/0-events.rst b/specification/0-events.rst
index ce36b040..a50d01fa 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -20,71 +20,19 @@ Room Events
 This specification outlines several standard event types, all of which are
 prefixed with ``m.``
 
-{{room_events}}
-
-m.room.message msgtypes
-~~~~~~~~~~~~~~~~~~~~~~~
-
-.. TODO-spec
-   How a client should handle unknown message types.
-
-
-Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type
-of message being sent. Each type has their own required and optional keys, as
-outlined below.
-
-{{msgtype_events}}
-
-Presence Events
-~~~~~~~~~~~~~~~
-
-{{presence_events}}
- 
-Each user has the concept of presence information. This encodes the
-"availability" of that user, suitable for display on other user's clients.
-This is transmitted as an ``m.presence`` event and is one of the few events
-which are sent *outside the context of a room*. The basic piece of presence
-information is represented by the ``presence`` key, which is an enum of one
-of the following:
-
-      - ``online`` : The default state when the user is connected to an event
-        stream.
-      - ``unavailable`` : The user is not reachable at this time.
-      - ``offline`` : The user is not connected to an event stream.
-      - ``free_for_chat`` : The user is generally willing to receive messages
-        moreso than default.
-      - ``hidden`` : Behaves as offline, but allows the user to see the client
-        state anyway and generally interact with client features. (Not yet
-        implemented in synapse).
-
-In addition, the server maintains a timestamp of the last time it saw a
-pro-active event from the user; either sending a message to a room, or
-changing presence state from a lower to a higher level of availability
-(thus: changing state from ``unavailable`` to ``online`` counts as a
-proactive event, whereas in the other direction it will not). This timestamp
-is presented via a key called ``last_active_ago``, which gives the relative
-number of milliseconds since the message is generated/emitted that the user
-was last seen active.
-    
-
-Events on Change of Profile Information
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Because the profile displayname and avatar information are likely to be used in
-many places of a client's display, changes to these fields cause an automatic
-propagation event to occur, informing likely-interested parties of the new
-values. This change is conveyed using two separate mechanisms:
-
- - a ``m.room.member`` event is sent to every room the user is a member of,
-   to update the ``displayname`` and ``avatar_url``.
- - a ``m.presence`` presence status update is sent, again containing the new values of the
-   ``displayname`` and ``avatar_url`` keys, in addition to the required
-   ``presence`` key containing the current presence state of the user.
-
-Both of these should be done automatically by the home server when a user
-successfully changes their displayname or avatar URL fields.
-
-Additionally, when home servers emit room membership events for their own
-users, they should include the displayname and avatar URL fields in these
-events so that clients already have these details to hand, and do not have to
-perform extra roundtrips to query it.
+{{m_room_aliases_event}}
+
+{{m_room_canonical_aliases_event}}
+
+{{m_room_create_event}}
+
+{{m_room_history_visibility_event}}
+
+{{m_room_join_rules_event}}
+
+{{m_room_member_event}}
+
+{{m_room_power_levels_event}}
+
+{{m_room_redaction_event}}
 
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
new file mode 100644
index 00000000..7f582ca4
--- /dev/null
+++ b/specification/modules/instant_messaging.rst
@@ -0,0 +1,27 @@
+Instant Messaging
+=================
+
+Events
+------
+
+{{m_room_message_event}}
+
+{{m_room_message_feedback_event}}
+
+{{m_room_name_event}}
+
+{{m_room_topic_event}}
+
+m.room.message msgtypes
+-----------------------
+
+.. TODO-spec
+   How a client should handle unknown message types.
+
+
+Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type
+of message being sent. Each type has their own required and optional keys, as
+outlined below.
+
+{{msgtype_events}}
+
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index 6f644879..ddd2adff 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -1,7 +1,5 @@
-Presence Events
-===============
-
-{{presence_events}}
+Presence
+========
  
 Each user has the concept of presence information. This encodes the
 "availability" of that user, suitable for display on other user's clients.
@@ -29,6 +27,11 @@ is presented via a key called ``last_active_ago``, which gives the relative
 number of milliseconds since the message is generated/emitted that the user
 was last seen active.
 
+Events
+------
+
+{{presence_events}}
+
 Presence HTTP API
 -----------------
 .. TODO-spec
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 454f23b9..e33bc785 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -14,6 +14,7 @@ targets:
       - 6-appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
+    - modules/instant_messaging.rst
     - modules/voip_events.rst
     - modules/typing_notifications.rst
     - modules/receipts.rst
diff --git a/templating/batesian/sections.py b/templating/batesian/sections.py
index 11c34fb3..31849389 100644
--- a/templating/batesian/sections.py
+++ b/templating/batesian/sections.py
@@ -27,12 +27,38 @@ class Sections(object):
             section_key = func_name[len("render_"):]
             self.log("Generating section '%s'" % section_key)
             section = func()
-            if not isinstance(section, basestring):
+            if isinstance(section, basestring):
+                if section_key in section_dict:
+                    raise Exception(
+                        ("%s : Section %s already exists. It must have been " +
+                        "generated dynamically. Check which render_ methods " +
+                        "return a dict.") %
+                        (func_name, section_key)
+                    )
+                section_dict[section_key] = section
+                self.log(
+                    "  Generated. Snippet => %s" % section[:60].replace("\n","")
+                )
+            elif isinstance(section, dict):
+                self.log("  Generated multiple sections:")
+                for (k, v) in section.iteritems():
+                    if not isinstance(k, basestring) or not isinstance(v, basestring):
+                        raise Exception(
+                            ("Method %s returned multiple sections as a dict but " +
+                            "expected the dict elements to be strings but they aren't.") %
+                            (func_name, )
+                        )
+                    if k in section_dict:
+                        raise Exception(
+                            "%s tried to produce section %s which already exists." %
+                            (func_name, k)
+                        )
+                    section_dict[k] = v
+                    self.log(
+                        "  %s  =>  %s" % (k, v[:60].replace("\n",""))
+                    )
+            else:
                 raise Exception(
-                    "Section function '%s' didn't return a string!" % func_name
+                    "Section function '%s' didn't return a string/dict!" % func_name
                 )
-            section_dict[section_key] = section
-            self.log(
-                "  Generated. Snippet => %s" % section[:60].replace("\n","")
-            )
         return section_dict
\ No newline at end of file
diff --git a/templating/matrix_templates/sections.py b/templating/matrix_templates/sections.py
index cacaa06a..e75a75af 100644
--- a/templating/matrix_templates/sections.py
+++ b/templating/matrix_templates/sections.py
@@ -76,37 +76,36 @@ class MatrixSections(Sections):
             ))
         return "\n\n".join(sections)
 
-    def render_profile_http_api(self):
-        return self._render_http_api_group(
-            "profile", 
-            sortFnOrPathList=["displayname", "avatar_url"]
-        )
-
-    def render_sync_http_api(self):
-        return self._render_http_api_group(
-            "sync"
-        )
-
-    def render_presence_http_api(self):
-        return self._render_http_api_group(
-            "presence",
-            sortFnOrPathList=["status"]
-        )
-
-    def render_membership_http_api(self):
-        return self._render_http_api_group(
-            "membership"
-        )
 
-    def render_login_http_api(self):
-        return self._render_http_api_group(
-            "login"
-        )
+    # Special function: Returning a dict will specify multiple sections where
+    # the key is the section name and the value is the value of the section
+    def render_group_http_apis(self):
+        # map all swagger_apis to the form $GROUP_http_api
+        swagger_groups = self.units.get("swagger_apis").keys()
+        renders = {}
+        for group in swagger_groups:
+            sortFnOrPathList = None
+            if group == "presence":
+                sortFnOrPathList = ["status"]
+            elif group == "profile":
+                sortFnOrPathList=["displayname", "avatar_url"]
+            renders[group + "_http_api"] = self._render_http_api_group(
+                group, sortFnOrPathList
+            )
+        return renders
 
-    def render_rooms_http_api(self):
-        return self._render_http_api_group(
-            "rooms"
-        )
+    # Special function: Returning a dict will specify multiple sections where
+    # the key is the section name and the value is the value of the section
+    def render_group_events(self):
+        # map all event schemata to the form $EVENTTYPE_event with s/./_/g
+        # e.g. m_room_topic_event
+        schemas = self.units.get("event_schemas")
+        renders = {}
+        for event_type in schemas:
+            renders[event_type.replace(".", "_") + "_event"] = self._render_events(
+                lambda x: x == event_type, sorted
+            )
+        return renders
 
     def render_room_events(self):
         def filterFn(eventType):
@@ -180,3 +179,4 @@ class MatrixSections(Sections):
 
     def render_common_state_event_fields(self):
         return self._render_ce_type("state_event")
+

From 29bae1579017c05d8ff5f5cf415459cebf0bac12 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 11:30:07 +0100
Subject: [PATCH 143/467] Fix typo

---
 specification/0-events.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/0-events.rst b/specification/0-events.rst
index a50d01fa..16948462 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -22,7 +22,7 @@ prefixed with ``m.``
 
 {{m_room_aliases_event}}
 
-{{m_room_canonical_aliases_event}}
+{{m_room_canonical_alias_event}}
 
 {{m_room_create_event}}
 

From 1da64db302f07019e410e607775c9efa0702d100 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 14:29:37 +0100
Subject: [PATCH 144/467] Use relative depths for groups instead of absolute
 ones

This means the group can be agnostic to how deeply nested it is,
improving reusability of groups.
---
 scripts/gendoc.py          | 32 ++++++++++++++++++++++++--------
 specification/targets.yaml |  3 ++-
 2 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 26ee5a48..364b8e65 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -1,6 +1,7 @@
 #! /usr/bin/env python
 
 from docutils.core import publish_file
+import copy
 import fileinput
 import glob
 import os
@@ -292,21 +293,32 @@ def get_build_target(targets_listing, target_name):
             "Found target but 'files' key is not a list."
         )
 
-    def get_group(group_id):
+    def get_group(group_id, depth):
         group_name = group_id[len("group:"):]
         group = all_targets.get("groups", {}).get(group_name)
         if not group:
             raise Exception(
-                "Tried to find group '" + group_name + "' but it " +
-                "doesn't exist."
+                "Tried to find group '%s' but it doesn't exist." % group_name
             )
+        if not isinstance(group, list):
+            raise Exception(
+                "Expected group '%s' to be a list but it isn't." % group_name
+            )
+        # deep copy so changes to depths don't contaminate multiple uses of this group
+        group = copy.deepcopy(group)
+        # swap relative depths for absolute ones
+        for i, entry in enumerate(group):
+            if isinstance(entry, dict):
+                group[i] = {
+                    (rel_depth + depth): v for (rel_depth, v) in entry.items()
+                }
         return group
 
     resolved_files = []
     for file_entry in target["files"]:
         # file_entry is a group id
         if isinstance(file_entry, basestring) and file_entry.startswith("group:"):
-            group = get_group(file_entry)
+            group = get_group(file_entry, 0)
             # The group may be resolved to a list of file entries, in which case
             # we want to extend the array to insert each of them rather than
             # insert the entire list as a single element (which is what append does)
@@ -317,12 +329,16 @@ def get_build_target(targets_listing, target_name):
         # file_entry is a dict which has more file entries as values
         elif isinstance(file_entry, dict):
             resolved_entry = {}
-            for (k, v) in file_entry.iteritems():
-                if isinstance(v, basestring) and v.startswith("group:"):
-                    resolved_entry[k] = get_group(v)
+            for (depth, entry) in file_entry.iteritems():
+                if not isinstance(entry, basestring):
+                    raise Exception(
+                        "Double-nested depths are not supported. Entry: %s" % (file_entry,)
+                    )
+                if entry.startswith("group:"):
+                    resolved_entry[depth] = get_group(entry, depth)
                 else:
                     # map across without editing (e.g. normal file path)
-                    resolved_entry[k] = v
+                    resolved_entry[depth] = entry
             resolved_files.append(resolved_entry)
             continue
         # file_entry is just a plain ol' file path
diff --git a/specification/targets.yaml b/specification/targets.yaml
index e33bc785..62585c69 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -23,7 +23,8 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/end_to_end_encryption.rst
     - modules/history_visibility.rst
     - modules/push_overview.rst
-    - { 2: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
+      # relative depth
+    - { 1: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
 
 title_styles: ["=", "-", "~", "+", "^"]
 

From 064a2c91721a285cf03dcd0505e3c01a8af39256 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 14:59:57 +0100
Subject: [PATCH 145/467] Use argparse and log functions for gendoc.py

gendoc.py has become more complex such that we actually want to pass things
to it like `--verbose`, `--nodelete`, `--target`, so use `argparse` to do this
like we have `build.py`. Pass through `-v` flags to `build.py`.
---
 scripts/gendoc.py | 91 +++++++++++++++++++++++++++++------------------
 1 file changed, 56 insertions(+), 35 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 364b8e65..d2b0d75e 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -1,5 +1,6 @@
 #! /usr/bin/env python
 
+from argparse import ArgumentParser
 from docutils.core import publish_file
 import copy
 import fileinput
@@ -17,6 +18,7 @@ stylesheets = {
     "stylesheet_path": ["basic.css", "nature.css", "codehighlight.css"]
 }
 
+VERBOSE = False
 
 """
 Read a RST file and replace titles with a different title level if required.
@@ -64,8 +66,8 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
         if file_offset is None:
             file_offset = line_title_level
             if file_offset != 0:
-                print ("     WARNING: %s starts with a title style of '%s' but '%s' " +
-                    "is preferable.") % (filename, line_title_style, title_styles[0])
+                logv(("     WARNING: %s starts with a title style of '%s' but '%s' " +
+                    "is preferable.") % (filename, line_title_style, title_styles[0]))
 
         # Sanity checks: Make sure that this file is obeying the title levels
         # specified and bail if it isn't.
@@ -101,12 +103,11 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
             continue
 
         # Adjusting line levels
-        # print (
-        #     "File: %s Adjusting %s to %s because file_offset=%s title_offset=%s" %
-        #     (filename, line_title_style,
-        #         title_styles[adjusted_level],
-        #         file_offset, title_level)
-        # )
+        logv(
+            "File: %s Adjusting %s to %s because file_offset=%s title_offset=%s" %
+            (filename, line_title_style, title_styles[adjusted_level],
+                file_offset, title_level)
+        )
         rst_lines.append(line.replace(
             line_title_style,
             title_styles[adjusted_level]
@@ -118,7 +119,7 @@ def load_with_adjusted_titles(filename, file_stream, title_level, title_styles):
 def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
     # string are file paths to RST blobs
     if isinstance(file_info, basestring):
-        print "%s %s" % (">" * (1 + title_level), file_info)
+        log("%s %s" % (">" * (1 + title_level), file_info))
         with open(os.path.join(spec_dir, file_info), "r") as f:
             rst = None
             if adjust_titles:
@@ -244,15 +245,21 @@ def run_through_template(input):
     tmpfile = './tmp/output'
     try:
         with open(tmpfile, 'w') as out:
-            print subprocess.check_output(
-                [
-                    'python', 'build.py', "-v",
-                    "-i", "matrix_templates",
-                    "-o", "../scripts/tmp",
-                    "../scripts/"+input
-                ],
-                stderr=out,
-                cwd="../templating",
+            args = [
+                'python', 'build.py',
+                "-i", "matrix_templates",
+                "-o", "../scripts/tmp",
+                "../scripts/"+input
+            ]
+            if VERBOSE:
+                args.insert(2, "-v")
+            log(" ==== build.py output ==== ")
+            log(
+                subprocess.check_output(
+                    args,
+                    stderr=out,
+                    cwd="../templating"
+                )
             )
     except subprocess.CalledProcessError as e:
         with open(tmpfile, 'r') as f:
@@ -347,6 +354,13 @@ def get_build_target(targets_listing, target_name):
     build_target["files"] = resolved_files
     return build_target
 
+def log(line):
+    print "gendoc: %s" % line
+
+def logv(line):
+    if VERBOSE:
+        print "gendoc:V: %s" % line
+
 
 def prepare_env():
     try:
@@ -363,9 +377,9 @@ def cleanup_env():
     shutil.rmtree("./tmp")
 
 
-def main(target_name):
+def main(target_name, keep_intermediates):
     prepare_env()
-    print "Building spec [target=%s]" % target_name
+    log("Building spec [target=%s]" % target_name)
     target = get_build_target("../specification/targets.yaml", target_name)
     build_spec(target=target, out_filename="tmp/templated_spec.rst")
     run_through_template("tmp/templated_spec.rst")
@@ -377,22 +391,29 @@ def main(target_name):
     run_through_template("tmp/howto.rst")
     rst2html("tmp/full_spec.rst", "gen/specification.html")
     rst2html("tmp/howto.rst", "gen/howtos.html")
-    if "--nodelete" not in sys.argv:
+    if not keep_intermediates:
         cleanup_env()
 
 
 if __name__ == '__main__':
-    if len(sys.argv) > 1 and sys.argv[1:] != ["--nodelete"]:
-        # we accept almost no args, so they don't know what they're doing!
-        print "gendoc.py - Generate the Matrix specification as HTML."
-        print "Usage:"
-        print "  python gendoc.py [--nodelete]"
-        print ""
-        print "The specification can then be found in the gen/ folder."
-        print ("If --nodelete was specified, intermediate files will be "
-               "present in the tmp/ folder.")
-        print ""
-        print "Requirements:"
-        print " - This script requires Jinja2 and rst2html (docutils)."
-        sys.exit(0)
-    main("main")
+    parser = ArgumentParser(
+        "gendoc.py - Generate the Matrix specification as HTML to the gen/ folder."
+    )
+    parser.add_argument(
+        "--nodelete", "-n", action="store_true",
+        help="Do not delete intermediate files. They will be found in tmp/"
+    )
+    parser.add_argument(
+        "--target", "-t", default="main",
+        help="Specify the build target to build from specification/targets.yaml"
+    )
+    parser.add_argument(
+        "--verbose", "-v", action="store_true",
+        help="Turn on verbose mode."
+    )
+    args = parser.parse_args()
+    if not args.target:
+        parser.print_help()
+        sys.exit(1)
+    VERBOSE = args.verbose
+    main(args.target, args.nodelete)

From f1adad5fb3b2c6f03d1b635880726a661ba7bb03 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 15:10:55 +0100
Subject: [PATCH 146/467] Add more logging with file prefixes

This makes the handoff between gendoc and batesian clearer in the logs.
---
 scripts/gendoc.py               | 11 +++++------
 templating/batesian/sections.py |  2 +-
 templating/batesian/units.py    |  2 +-
 templating/build.py             | 14 ++++++++------
 4 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index d2b0d75e..611d0d10 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -253,13 +253,12 @@ def run_through_template(input):
             ]
             if VERBOSE:
                 args.insert(2, "-v")
+            log("EXEC: %s" % " ".join(args))
             log(" ==== build.py output ==== ")
-            log(
-                subprocess.check_output(
-                    args,
-                    stderr=out,
-                    cwd="../templating"
-                )
+            print subprocess.check_output(
+                args,
+                stderr=out,
+                cwd="../templating"
             )
     except subprocess.CalledProcessError as e:
         with open(tmpfile, 'r') as f:
diff --git a/templating/batesian/sections.py b/templating/batesian/sections.py
index 31849389..c18c9a5f 100644
--- a/templating/batesian/sections.py
+++ b/templating/batesian/sections.py
@@ -16,7 +16,7 @@ class Sections(object):
 
     def log(self, text):
         if self.debug:
-            print text
+            print "batesian:sections: %s" % text
 
     def get_sections(self):
         render_list = inspect.getmembers(self, predicate=inspect.ismethod)
diff --git a/templating/batesian/units.py b/templating/batesian/units.py
index c20a2d1f..993f253d 100644
--- a/templating/batesian/units.py
+++ b/templating/batesian/units.py
@@ -22,7 +22,7 @@ class Units(object):
 
     def log(self, text):
         if self.debug:
-            print text
+            print "batesian:units: %s" % text
 
     def get_units(self, debug=False):
         unit_list = inspect.getmembers(self, predicate=inspect.ismethod)
diff --git a/templating/build.py b/templating/build.py
index ac3d2491..013248f4 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -52,8 +52,8 @@ def create_from_template(template, sections):
 def check_unaccessed(name, store):
     unaccessed_keys = store.get_unaccessed_set()
     if len(unaccessed_keys) > 0:
-        print "Found %s unused %s keys." % (len(unaccessed_keys), name)
-        print unaccessed_keys
+        log("Found %s unused %s keys." % (len(unaccessed_keys), name))
+        log(unaccessed_keys)
 
 def main(input_module, file_stream=None, out_dir=None, verbose=False):
     if out_dir and not os.path.exists(out_dir):
@@ -121,17 +121,19 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
         return
 
     # check the input files and substitute in sections where required
-    print "Parsing input template: %s" % file_stream.name
+    log("Parsing input template: %s" % file_stream.name)
     temp = Template(file_stream.read())
-    print "Creating output for: %s" % file_stream.name
+    log("Creating output for: %s" % file_stream.name)
     output = create_from_template(temp, sections)
     with open(
             os.path.join(out_dir, os.path.basename(file_stream.name)), "w"
             ) as f:
         f.write(output)
-    print "Output file for: %s" % file_stream.name
+    log("Output file for: %s" % file_stream.name)
     check_unaccessed("units", units)
 
+def log(line):
+    print "batesian: %s" % line
 
 if __name__ == '__main__':
     parser = ArgumentParser(
@@ -175,7 +177,7 @@ if __name__ == '__main__':
         sys.exit(0)
 
     if not args.file:
-        print "No file supplied."
+        log("No file supplied.")
         parser.print_help()
         sys.exit(1)
 

From 6afdfc0771265ded3b7e0752c7ee438b09f076cb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 23 Sep 2015 15:36:13 +0100
Subject: [PATCH 147/467] Add more logging and make logging context clearer

This is now actually useful if you want to debug why your swagger YAML
isn't producing a table you think it should be.
---
 scripts/gendoc.py                    |  8 ++++----
 templating/batesian/units.py         |  6 +++++-
 templating/matrix_templates/units.py | 28 +++++++++++++++++++++++++---
 3 files changed, 34 insertions(+), 8 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 611d0d10..1655d6f0 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -241,7 +241,7 @@ def rst2html(i, o):
             )
 
 
-def run_through_template(input):
+def run_through_template(input, set_verbose):
     tmpfile = './tmp/output'
     try:
         with open(tmpfile, 'w') as out:
@@ -251,7 +251,7 @@ def run_through_template(input):
                 "-o", "../scripts/tmp",
                 "../scripts/"+input
             ]
-            if VERBOSE:
+            if set_verbose:
                 args.insert(2, "-v")
             log("EXEC: %s" % " ".join(args))
             log(" ==== build.py output ==== ")
@@ -381,13 +381,13 @@ def main(target_name, keep_intermediates):
     log("Building spec [target=%s]" % target_name)
     target = get_build_target("../specification/targets.yaml", target_name)
     build_spec(target=target, out_filename="tmp/templated_spec.rst")
-    run_through_template("tmp/templated_spec.rst")
+    run_through_template("tmp/templated_spec.rst", VERBOSE)
     fix_relative_titles(
         target=target, filename="tmp/templated_spec.rst",
         out_filename="tmp/full_spec.rst"
     )
     shutil.copy("../supporting-docs/howtos/client-server.rst", "tmp/howto.rst")
-    run_through_template("tmp/howto.rst")
+    run_through_template("tmp/howto.rst", False)  # too spammy to mark -v on this
     rst2html("tmp/full_spec.rst", "gen/specification.html")
     rst2html("tmp/howto.rst", "gen/howtos.html")
     if not keep_intermediates:
diff --git a/templating/batesian/units.py b/templating/batesian/units.py
index 993f253d..144cf245 100644
--- a/templating/batesian/units.py
+++ b/templating/batesian/units.py
@@ -22,7 +22,11 @@ class Units(object):
 
     def log(self, text):
         if self.debug:
-            print "batesian:units: %s" % text
+            func_name = ""
+            trace = inspect.stack()
+            if len(trace) > 1 and len(trace[1]) > 2:
+                func_name = trace[1][3] + ":"
+            print "batesian:units:%s %s" % (func_name, text)
 
     def get_units(self, debug=False):
         unit_list = inspect.getmembers(self, predicate=inspect.ismethod)
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index a5c6a815..50fa784e 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -1,4 +1,12 @@
-"""Contains all the units for the spec."""
+"""
+Contains all the units for the spec.
+
+This file loads swagger and JSON schema files and parses out the useful bits
+and returns them as Units for use in Batesian.
+
+For the actual conversion of data -> RST (including templates), see the sections
+file instead.
+"""
 from batesian.units import Units
 import inspect
 import json
@@ -134,7 +142,7 @@ class MatrixUnits(Units):
                         "good_response": ""
                     }
                 }
-                self.log(".o.O.o. Endpoint: %s %s" % (method, path))
+                self.log(" ------- Endpoint: %s %s ------- " % (method, path))
                 for param in single_api.get("parameters", []):
                     # description
                     desc = param.get("description", "")
@@ -183,6 +191,9 @@ class MatrixUnits(Units):
                             "desc": json_body[key]["description"]
                         })
                 # endfor[param]
+                for row in endpoint["req_params"]:
+                    self.log("Request parameter: %s" % row)
+
                 # group params by location to ease templating
                 endpoint["req_param_by_loc"] = {
                     #   path: [...], query: [...], body: [...]
@@ -240,6 +251,7 @@ class MatrixUnits(Units):
 
                 # add response params if this API has any.
                 if good_response:
+                    self.log("Found a 200 response for this API")
                     res_type = Units.prop(good_response, "schema/type")
                     if res_type and res_type not in ["object", "array"]:
                         # response is a raw string or something like that
@@ -278,6 +290,16 @@ class MatrixUnits(Units):
                             }]
                         })
 
+                for response_table in endpoint["res_tables"]:
+                    self.log("Response: %s" % response_table["title"])
+                    for r in response_table["rows"]:
+                        self.log("Row: %s" % r)
+                if len(endpoint["res_tables"]) == 0:
+                    self.log(
+                        "This API appears to have no response table. Are you " +
+                        "sure this API returns no parameters?"
+                    )
+
                 endpoints.append(endpoint)
 
                 aliases = single_api.get("x-alias", None)
@@ -475,7 +497,7 @@ class MatrixUnits(Units):
             if re.match("^v[0-9\.]+$", word):
                 version = word[1:]  # strip the 'v'
 
-        self.log("Version: %s Title part: %s Changelog lines: %s" % (
+        self.log("Version: %s Title part: %s Changelog line count: %s" % (
             version, title_part, len(changelog_lines)
         ))
         if not version or len(changelog_lines) == 0:

From 385b6c4759515b8b79151984e28932968ba9dc0c Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 13:37:32 +0100
Subject: [PATCH 148/467] Only validate a file if it ends with ".yaml".

Otherwise we try to validate vim .swp files.
---
 api/validator.js | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/api/validator.js b/api/validator.js
index 6eef652f..3b89a5a3 100644
--- a/api/validator.js
+++ b/api/validator.js
@@ -44,7 +44,8 @@ if (isDir) {
             process.exit(1);
         }
         files.forEach(function(f) {
-            if (f.indexOf(".yaml") > 0) {
+            var suffix = ".yaml";
+            if (f.indexOf(suffix, f.length - suffix.length) > 0) {
                 parser.parse(path.join(opts.schema, f), function(err, api, metadata) {
                     if (!err) {
                         console.log("%s is valid.", f);

From 2f039a114201887f7b2da8b4f8a334bcd9f8f515 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 15:39:17 +0100
Subject: [PATCH 149/467] Add swagger docs for the v2 filter POST API

---
 api/client-server/v2_alpha/filter.yaml | 188 +++++++++++++++++++++++++
 1 file changed, 188 insertions(+)
 create mode 100644 api/client-server/v2_alpha/filter.yaml

diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
new file mode 100644
index 00000000..99dc0ebc
--- /dev/null
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -0,0 +1,188 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v2 filter API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+basePath: /_matrix/client/api/v2_alpha
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+definitions:
+  EventFilter:
+    title:
+    type: object
+    properties:
+      types:
+        type: array
+        description: |-
+          A list of event types to include.
+          If this list is absent then all event types are included.
+        items:
+          type: string
+      not_types:
+        type: array
+        description: |-
+          A list of event types to exclude.
+          If this list is absent then no event types are excluded.
+        items:
+          type: string
+      senders:
+        type: array
+        description: |-
+          A list of senders IDs to include.
+          If this list is absent then all senders are included.
+        items:
+          type: string
+      not_senders:
+        type: array
+        description: |-
+          A list of sender IDs to exclude
+          If this list is absent then no senders are excluded.
+        items:
+          type: string
+      rooms:
+        type: array
+        description: |-
+          A list of room IDs to include.
+          If this list is absent then all rooms are included.
+        items:
+          type: string
+      not_rooms:
+        type: array
+        description: |-
+          A list of room IDs to exclude
+          If this list is absent then no rooms are excluded.
+        items:
+          type: string
+  SyncFilter:
+    room:
+      type: object
+      properties:
+        state:
+          description:
+            The state events to include for rooms.
+          allOf:
+          - $ref: "#/definitions/EventFilter"
+        events:
+          description:
+            The message and state update events to include for rooms.
+          allOf:
+          - $ref: "#/definitions/EventFilter"
+        ephemeral:
+          description: |-
+            The events that aren't recorded in the permenant history, e.g.
+            typing and receipts, to include for rooms.
+          allOf:
+          - $ref: "#/definitions/EventFilter"
+    public_user_data:
+      description: |-
+        The public user data, e.g. profile and presence, to include.
+      allOf:
+          - $ref: "#/definitions/EventFilter"
+    private_user_data:
+      description: |-
+        Events that are private to a user but shared amoungst their devices,
+        e.g. notification settings, to include.
+      allOf:
+          - $ref: "#/definitions/EventFilter"
+    event_format:
+      description: |-
+        The format to use for events. "client" will return the events in a
+        format suitable for clients. "federation" will return the raw event
+        as receieved over federation. The default is "client".
+      type: string
+    event_fields:
+      type: array
+      description: |-
+        List of event fields to include. If this list is absent then all fields
+        are included. The entries may include "." charaters to indicate
+        sub-fields. So ["content.body"] will include the "body" field of the
+        "content" object. A server may include more fields than were requested.
+      items:
+        string
+paths:
+    "/user/{userId}/filter":
+      post:
+        summary: Upload a new filter.
+      description: |-
+        Uploads a new filter definition to the homeserver.
+        Returns a filter ID that may be used in /sync requests to
+        retrict which events are returned to the client.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: userId
+          required: true
+          description:
+            The id of the user uploading the filter. The access token must be
+            authorized to make requests for this user id.
+          x-example: "@alice:example.com"
+        - in: body
+          name: filter
+          required: true
+          description: The filter to upload.
+          schema:
+            type: object
+            allOf:
+              - $ref: "#/definitions/SyncFilter"
+            example:
+              type: object
+              example: |-
+                {
+                  "room": {
+                    "state": {
+                      "types": ["m.room.*"],
+                      "not_rooms": ["!726s6s6q:example.com"],
+                    },
+                    "events": {
+                      "types": ["m.room.message"],
+                      "not_rooms": ["!726s6s6q:example.com"],
+                      "not_senders": ["@spam:example.com"]
+                    },
+                    "emphemeral": {
+                      "types": ["m.receipt", "m.typing"],
+                      "not_rooms": ["!726s6s6q:example.com"],
+                      "not_senders": ["@spam:example.com"]
+                    }
+                  },
+                  "public_user_data": {
+                    "types": ["m.presence"]
+                  },
+                  "private_user_data": {
+                    "types": []
+                  },
+                  "server_data": {
+                    "types": []
+                  },
+                  "event_format": "client",
+                  "event_fields": ["type", "content", "sender"]
+                }
+      responses:
+        200:
+          description: The filter was created.
+          examples:
+            application/json: |-
+              {
+                "filter_id": "66696p746572"
+              }
+          schema:
+            type: object
+            properties:
+              filter_id:
+                type: string
+                description:
+                  The ID of the filter that was created.
+
+
+

From 883105eae69d51e8672f63b9aa48946b6ae5347b Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 16:25:03 +0100
Subject: [PATCH 150/467] Document the v2 filter GET API

---
 .../v2_alpha/definitions/definitions          |   1 +
 .../v2_alpha/definitions/event_filter.json    |  53 ++++++
 .../v2_alpha/definitions/sync_filter.json     |  48 ++++++
 api/client-server/v2_alpha/filter.yaml        | 158 +++++++-----------
 4 files changed, 159 insertions(+), 101 deletions(-)
 create mode 120000 api/client-server/v2_alpha/definitions/definitions
 create mode 100644 api/client-server/v2_alpha/definitions/event_filter.json
 create mode 100644 api/client-server/v2_alpha/definitions/sync_filter.json

diff --git a/api/client-server/v2_alpha/definitions/definitions b/api/client-server/v2_alpha/definitions/definitions
new file mode 120000
index 00000000..945c9b46
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/definitions
@@ -0,0 +1 @@
+.
\ No newline at end of file
diff --git a/api/client-server/v2_alpha/definitions/event_filter.json b/api/client-server/v2_alpha/definitions/event_filter.json
new file mode 100644
index 00000000..b8ab1352
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/event_filter.json
@@ -0,0 +1,53 @@
+{
+  "type": "object",
+  "properties": {
+    "types": {
+      "type": "array",
+      "description":
+        "A list of event types to include. If this list is absent then all event types are included.",
+      "items": {
+        "type": "string"
+      }
+    },
+    "not_types": {
+      "type": "array",
+      "description":
+        "A list of event types to exclude. If this list is absent then no event types are excluded.",
+      "items": {
+        "type": "string"
+      }
+    },
+    "senders": {
+      "type": "array",
+      "description":
+        "A list of senders IDs to include. If this list is absent then all senders are included.",
+      "items": {
+        "type": "string"
+      }
+    },
+    "not_senders": {
+      "type": "array",
+      "description":
+        "A list of sender IDs to exclude. If this list is absent then no senders are excluded.",
+      "items": {
+        "type": "string"
+      }
+    },
+    "rooms": {
+      "type": "array",
+      "description":
+        "A list of room IDs to include. If this list is absent then all rooms are included.",
+      "items": {
+        "type": "string"
+      }
+    },
+    "not_rooms": {
+      "type": "array",
+      "description":
+        "A list of room IDs to exclude. If this list is absent then no rooms are excluded.",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/definitions/sync_filter.json b/api/client-server/v2_alpha/definitions/sync_filter.json
new file mode 100644
index 00000000..c9c47646
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/sync_filter.json
@@ -0,0 +1,48 @@
+{
+  "type": "object",
+  "properties": {
+    "room": {
+      "type": "object",
+      "properties": {
+        "state": {
+          "description":
+            "The state events to include for rooms.",
+          "allOf": [{"$ref": "definitions/event_filter.json"}]
+        },
+        "events": {
+          "description":
+            "The message and state update events to include for rooms.",
+          "allOf": [{"$ref": "definitions/event_filter.json"}]
+        },
+        "ephemeral": {
+          "description":
+            "The events that aren't recorded in the room history, e.g. typing and receipts, to include for rooms.",
+          "allOf": [{"$ref": "definitions/event_filter.json"}]
+        }
+      }
+    },
+    "public_user_data": {
+      "description":
+        "The public user data, e.g. profile and presence, to include.",
+      "allOf": [{"$ref": "definitions/event_filter.json"}]
+    },
+    "private_user_data": {
+      "description":
+        "Events that are private to a user but shared amoungst their devices, e.g. notification settings, to include.",
+      "allOf": [{"$ref": "definitions/event_filter.json"}]
+    },
+    "event_format": {
+      "description":
+        "The format to use for events. 'client' will return the events in a format suitable for clients. 'federation' will return the raw event as receieved over federation. The default is 'client'.",
+      "type": "string"
+    },
+    "event_fields": {
+      "type": "array",
+      "description":
+        "List of event fields to include. If this list is absent then all fields are included. The entries may include '.' charaters to indicate sub-fields. So ['content.body'] will include the 'body' field of the 'content' object. A server may include more fields than were requested.",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 99dc0ebc..115ef230 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -16,103 +16,10 @@ securityDefinitions:
     description: The user_id or application service access_token
     name: access_token
     in: query
-definitions:
-  EventFilter:
-    title:
-    type: object
-    properties:
-      types:
-        type: array
-        description: |-
-          A list of event types to include.
-          If this list is absent then all event types are included.
-        items:
-          type: string
-      not_types:
-        type: array
-        description: |-
-          A list of event types to exclude.
-          If this list is absent then no event types are excluded.
-        items:
-          type: string
-      senders:
-        type: array
-        description: |-
-          A list of senders IDs to include.
-          If this list is absent then all senders are included.
-        items:
-          type: string
-      not_senders:
-        type: array
-        description: |-
-          A list of sender IDs to exclude
-          If this list is absent then no senders are excluded.
-        items:
-          type: string
-      rooms:
-        type: array
-        description: |-
-          A list of room IDs to include.
-          If this list is absent then all rooms are included.
-        items:
-          type: string
-      not_rooms:
-        type: array
-        description: |-
-          A list of room IDs to exclude
-          If this list is absent then no rooms are excluded.
-        items:
-          type: string
-  SyncFilter:
-    room:
-      type: object
-      properties:
-        state:
-          description:
-            The state events to include for rooms.
-          allOf:
-          - $ref: "#/definitions/EventFilter"
-        events:
-          description:
-            The message and state update events to include for rooms.
-          allOf:
-          - $ref: "#/definitions/EventFilter"
-        ephemeral:
-          description: |-
-            The events that aren't recorded in the permenant history, e.g.
-            typing and receipts, to include for rooms.
-          allOf:
-          - $ref: "#/definitions/EventFilter"
-    public_user_data:
-      description: |-
-        The public user data, e.g. profile and presence, to include.
-      allOf:
-          - $ref: "#/definitions/EventFilter"
-    private_user_data:
-      description: |-
-        Events that are private to a user but shared amoungst their devices,
-        e.g. notification settings, to include.
-      allOf:
-          - $ref: "#/definitions/EventFilter"
-    event_format:
-      description: |-
-        The format to use for events. "client" will return the events in a
-        format suitable for clients. "federation" will return the raw event
-        as receieved over federation. The default is "client".
-      type: string
-    event_fields:
-      type: array
-      description: |-
-        List of event fields to include. If this list is absent then all fields
-        are included. The entries may include "." charaters to indicate
-        sub-fields. So ["content.body"] will include the "body" field of the
-        "content" object. A server may include more fields than were requested.
-      items:
-        string
 paths:
-    "/user/{userId}/filter":
-      post:
-        summary: Upload a new filter.
+  "/user/{userId}/filter":
+    post:
+      summary: Upload a new filter.
       description: |-
         Uploads a new filter definition to the homeserver.
         Returns a filter ID that may be used in /sync requests to
@@ -135,7 +42,7 @@ paths:
           schema:
             type: object
             allOf:
-              - $ref: "#/definitions/SyncFilter"
+              - $ref: "definitions/sync_filter.json"
             example:
               type: object
               example: |-
@@ -181,8 +88,57 @@ paths:
             properties:
               filter_id:
                 type: string
-                description:
+                description: |-
                   The ID of the filter that was created.
-
-
-
+  "/user/{userId}/filter/{filterId}":
+    get:
+      summary: Download a filter
+      parameters:
+        - in: path
+          name: userId
+          type: string
+          description: |-
+            The user ID to download a filter for.
+          x-example: "@alice:example.com"
+        - in: path
+          name: filterId
+          description: |-
+            The filter ID to download.
+          x-example: "66696p746572"
+      responses:
+        200:
+          examples:
+            application/json: |-
+              {
+                "room": {
+                  "state": {
+                    "types": ["m.room.*"],
+                    "not_rooms": ["!726s6s6q:example.com"]
+                  },
+                  "events": {
+                    "types": ["m.room.message"],
+                    "not_rooms": ["!726s6s6q:example.com"],
+                    "not_senders": ["@spam:example.com"]
+                  },
+                  "emphemeral": {
+                    "types": ["m.receipt", "m.typing"],
+                    "not_rooms": ["!726s6s6q:example.com"],
+                    "not_senders": ["@spam:example.com"]
+                  }
+                },
+                "public_user_data": {
+                  "types": ["m.presence"]
+                },
+                "private_user_data": {
+                  "types": []
+                },
+                "server_data": {
+                  "types": []
+                },
+                "event_format": "client",
+                "event_fields": ["type", "content", "sender"]
+              }
+          schema:
+            type: object
+            allOf:
+              - $ref: "definitions/sync_filter.json"

From 69298b96121182e44a47a29eab3c69de5e91f37a Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 16:29:51 +0100
Subject: [PATCH 151/467] Check "v2_alpha" in jenkins

---
 api/client-server/v2_alpha/filter.yaml | 5 +++++
 jenkins.sh                             | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 115ef230..f53ef2ba 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -100,13 +100,18 @@ paths:
           description: |-
             The user ID to download a filter for.
           x-example: "@alice:example.com"
+          required: true
         - in: path
           name: filterId
+          type: string
           description: |-
             The filter ID to download.
           x-example: "66696p746572"
+          required: true
       responses:
         200:
+          description: |-
+            "The filter defintion"
           examples:
             application/json: |-
               {
diff --git a/jenkins.sh b/jenkins.sh
index e1043644..0936de9d 100755
--- a/jenkins.sh
+++ b/jenkins.sh
@@ -5,5 +5,5 @@ set -ex
 (cd event-schemas/ && ./check_examples.py)
 (cd api && ./check_examples.py)
 (cd scripts && ./gendoc.py)
-(cd api && npm install && node validator.js -s "client-server/v1")
+(cd api && npm install && node validator.js -s "client-server/v1" && node validator.js -s "client-server/v2_alpha")
 (cd event-schemas/ && ./check.sh)

From a31a446661262872ba668c2c3192c5cf0a180387 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 16:48:00 +0100
Subject: [PATCH 152/467] Fix the POST example for the v2 filter API

---
 api/client-server/v2_alpha/filter.yaml | 58 +++++++++++++-------------
 1 file changed, 28 insertions(+), 30 deletions(-)

diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index f53ef2ba..5296f11a 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -43,38 +43,36 @@ paths:
             type: object
             allOf:
               - $ref: "definitions/sync_filter.json"
-            example:
-              type: object
-              example: |-
-                {
-                  "room": {
-                    "state": {
-                      "types": ["m.room.*"],
-                      "not_rooms": ["!726s6s6q:example.com"],
-                    },
-                    "events": {
-                      "types": ["m.room.message"],
-                      "not_rooms": ["!726s6s6q:example.com"],
-                      "not_senders": ["@spam:example.com"]
-                    },
-                    "emphemeral": {
-                      "types": ["m.receipt", "m.typing"],
-                      "not_rooms": ["!726s6s6q:example.com"],
-                      "not_senders": ["@spam:example.com"]
-                    }
-                  },
-                  "public_user_data": {
-                    "types": ["m.presence"]
-                  },
-                  "private_user_data": {
-                    "types": []
+            example: |-
+              {
+                "room": {
+                  "state": {
+                    "types": ["m.room.*"],
+                    "not_rooms": ["!726s6s6q:example.com"]
                   },
-                  "server_data": {
-                    "types": []
+                  "events": {
+                    "types": ["m.room.message"],
+                    "not_rooms": ["!726s6s6q:example.com"],
+                    "not_senders": ["@spam:example.com"]
                   },
-                  "event_format": "client",
-                  "event_fields": ["type", "content", "sender"]
-                }
+                  "emphemeral": {
+                    "types": ["m.receipt", "m.typing"],
+                    "not_rooms": ["!726s6s6q:example.com"],
+                    "not_senders": ["@spam:example.com"]
+                  }
+                },
+                "public_user_data": {
+                  "types": ["m.presence"]
+                },
+                "private_user_data": {
+                  "types": []
+                },
+                "server_data": {
+                  "types": []
+                },
+                "event_format": "client",
+                "event_fields": ["type", "content", "sender"]
+              }
       responses:
         200:
           description: The filter was created.

From f34722485f5984d5bbf3bc26f50bd54949730295 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 24 Sep 2015 16:50:27 +0100
Subject: [PATCH 153/467] Check the request example JSON matches the schema

---
 api/check_examples.py | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index 00e75263..a0cd0658 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -29,6 +29,33 @@ except ImportError as e:
     raise
 
 
+def check_parameter(filepath, request, parameter):
+    schema = parameter.get("schema")
+    example = None
+    try:
+        example_json = schema.get('example')
+        if example_json:
+            example = json.loads(example_json)
+    except Exception as e:
+        raise ValueError("Error parsing JSON example request for %r" % (
+            request
+        ), e)
+    fileurl = "file://" + os.path.abspath(filepath)
+    if example and schema:
+        try:
+            print ("Checking request schema for: %r %r" % (
+                filepath, request
+            ))
+            # Setting the 'id' tells jsonschema where the file is so that it
+            # can correctly resolve relative $ref references in the schema
+            schema['id'] = fileurl
+            jsonschema.validate(example, schema)
+        except Exception as e:
+            raise ValueError("Error validating JSON schema for %r %r" % (
+                request, code
+            ), e)
+
+
 def check_response(filepath, request, code, response):
     example = None
     try:
@@ -43,7 +70,9 @@ def check_response(filepath, request, code, response):
     fileurl = "file://" + os.path.abspath(filepath)
     if example and schema:
         try:
-            print ("Checking schema for: %r %r %r" % (filepath, request, code))
+            print ("Checking response schema for: %r %r %r" % (
+                filepath, request, code
+            ))
             # Setting the 'id' tells jsonschema where the file is so that it
             # can correctly resolve relative $ref references in the schema
             schema['id'] = fileurl
@@ -59,8 +88,13 @@ def check_swagger_file(filepath):
         swagger = yaml.load(f)
 
     for path, path_api in swagger.get('paths', {}).items():
+
         for method, request_api in path_api.items():
             request = "%s %s" % (method.upper(), path)
+            for parameter in request_api.get('parameters', ()):
+                if parameter['in'] == 'body':
+                    check_parameter(filepath, request, parameter)
+
             try:
                 responses = request_api['responses']
             except KeyError:

From 078dd0165ffe045229b0efee92ec8f425cfd28df Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 25 Sep 2015 11:58:47 +0100
Subject: [PATCH 154/467] Update the room creation API spec to include new
 keys: 'preset' and 'initial_state'

---
 specification/1-client_server_api.rst | 33 ++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 93e3cb90..e3dbde5a 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -750,10 +750,41 @@ options which can be set when creating a room:
     This will tell the server to invite everyone in the list to the newly
     created room.
 
+``preset``
+  Type:
+    String
+  Optional:
+    Yes
+  Value:
+    ``private_chat`` or ``public_chat``
+  Description:
+    Convenience parameter for setting various default state events based on a
+    preset.
+
+    Two presets are defined:
+
+    - ``private_chat``: Sets the ``join_rules`` to ``invite`` and
+      ``history_visibility`` to ``shared``
+    - ``public_chat``: Sets the ``join_rules`` to ``public`` and
+      ``history_visibility`` to ``shared``
+
+``initial_state``
+  Type:
+    List
+  Optional:
+    Yes
+  Value:
+    A list of state events to set in the new room.
+  Description:
+    Allows the user to override the default state events set in the new room.
+
+    The expected format of the state events are an object with ``type``,
+    ``state_key`` and ``content`` keys set.
+
 Example::
 
   {
-    "visibility": "public",
+    "preset": "public_chat",
     "room_alias_name": "thepub",
     "name": "The Grand Duke Pub",
     "topic": "All about happy hour"

From 6c89e6ea6717d9cd07afed7906add62ab82c0431 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:03:46 +0100
Subject: [PATCH 155/467] Wrap refresh_token in `s

---
 api/client-server/v1/login.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 0852db6b..a4fd8b95 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -66,7 +66,7 @@ paths:
                 description: |-
                   An access token for the account.
                   This access token can then be used to authorize other requests.
-                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+                  The access token may expire at some point, and if so, it SHOULD come with a ``refresh_token``.
                   There is no specific error message to indicate that a request has failed because
                   an access token has expired; instead, if a client has reason to believe its
                   access token is valid, and it receives an auth error, they should attempt to
@@ -132,7 +132,7 @@ paths:
                 description: |-
                   An access token for the account.
                   This access token can then be used to authorize other requests.
-                  The access token may expire at some point, and if so, it SHOULD come with a refresh_token.
+                  The access token may expire at some point, and if so, it SHOULD come with a ``refresh_token``.
               refresh_token:
                 type: string
                 description: (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the TODO Linkify /tokenrefresh API endpoint.

From fc87f4cdb03cca548ee7ec499949c1d52d5104a3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:10:15 +0100
Subject: [PATCH 156/467] Remove unused keys

---
 api/client-server/v1/login.yaml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index a4fd8b95..4cb564b7 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -29,7 +29,6 @@ paths:
       parameters:
         - in: body
           name: body
-          required: true
           schema:
             type: object
             example: |-
@@ -100,7 +99,6 @@ paths:
       parameters:
         - in: body
           name: body
-          required: true
           schema:
             type: object
             example: |-

From 5c4398c181bbf9da4aada6d4f8a3bf5b0d49c877 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:10:49 +0100
Subject: [PATCH 157/467] Remove superfluous comma

---
 api/client-server/v1/login.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 4cb564b7..e415e798 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -114,7 +114,7 @@ paths:
         200:
           description: |-
             The refresh token was accepted, and a new access token has been issued.
-            The passed refresh token is no longer valid, and cannot be used.
+            The passed refresh token is no longer valid and cannot be used.
             A new refresh token may have been returned.
           examples:
             application/json: |-

From 6c1491b3bafb3a806d8fdf4ed59eacdf4df79d06 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:17:11 +0100
Subject: [PATCH 158/467] Respond to some review comments

---
 api/client-server/v1/login.yaml | 3 ++-
 drafts/macaroons_caveats.rst    | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index e415e798..3d415c29 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -115,7 +115,8 @@ paths:
           description: |-
             The refresh token was accepted, and a new access token has been issued.
             The passed refresh token is no longer valid and cannot be used.
-            A new refresh token may have been returned.
+            A new refresh token will have been returned unless some policy does
+            not allow the user to continue to renew their session.
           examples:
             application/json: |-
               {
diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 791d217a..c4b6b6a4 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -1,7 +1,9 @@
 Macaroon Caveats
 ================
 
-Macaroons (http://theory.stanford.edu/~ataly/Papers/macaroons.pdf) are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
+`Macaroons`_ are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
+
+.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf)
 
 Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
 

From f5d436bd808ac9a1f20d816dccceac63a9202546 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:18:09 +0100
Subject: [PATCH 159/467] Remove extraneous )

---
 drafts/macaroons_caveats.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index c4b6b6a4..93622c3d 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -3,7 +3,7 @@ Macaroon Caveats
 
 `Macaroons`_ are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
 
-.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf)
+.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf
 
 Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
 

From cd6f15f62740075c2e8d1914fe77f3596046b166 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <dawagner@gmail.com>
Date: Fri, 25 Sep 2015 13:34:24 +0100
Subject: [PATCH 160/467] Remove obsolete comment

---
 api/client-server/v1/membership.yaml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 04922656..41f2febc 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -69,7 +69,6 @@ paths:
   "/rooms/{roomId}/invite":
     post:
       summary: Invite a user to participate in a particular room.
-      # It's a crying shame that I don't know how to force line breaks.
       description: |-
         This API invites a user to participate in a particular room.
         They do not start participating in the room until they actually join the

From d7d9f406a67419306d992bf906f01232aad59e47 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 25 Sep 2015 14:21:50 +0100
Subject: [PATCH 161/467] Bundle some state into invites

---
 api/client-server/v1/sync.yaml        |  5 +++++
 event-schemas/schema/v1/m.room.member | 19 +++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 833c425a..c7f78187 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -267,6 +267,11 @@ paths:
                       type: string
                       description: "The user's membership state in this room."
                       enum: ["invite", "join", "leave", "ban"]
+                    invite:
+                        type: object
+                        description: "The invite event if `membership` is `invite`"
+                        allOf:
+                          - "$ref": "room.room.member"
                     messages:
                       type: object
                       title: PaginationChunk
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 49b9f5b8..32302b85 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -32,6 +32,25 @@
         "type": {
             "type": "string",
             "enum": ["m.room.member"]
+        },
+        "invite_room_state": {
+            "type": "array",
+            "description": "A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``",
+            "items": {
+                "type": "object",
+                "title": "StateEvent",
+                "properties": {
+                    "type": {
+                        "type": "string"
+                    },
+                    "state_key": {
+                        "type": "string"
+                    },
+                    "content": {
+                        "type": "object"
+                    }
+                }
+            }
         }
     }
 }

From 37ccddb3087283067617a94d83dc968489bae20f Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 25 Sep 2015 14:25:07 +0100
Subject: [PATCH 162/467] Typo

---
 api/client-server/v1/sync.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index c7f78187..d82ee9e4 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -271,7 +271,7 @@ paths:
                         type: object
                         description: "The invite event if `membership` is `invite`"
                         allOf:
-                          - "$ref": "room.room.member"
+                          - "$ref": "m.room.member"
                     messages:
                       type: object
                       title: PaginationChunk

From 18dc7784df09c3ad6c3f4d844b87d046538bd5a5 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 25 Sep 2015 14:34:06 +0100
Subject: [PATCH 163/467] Mention precedence

---
 specification/1-client_server_api.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index e3dbde5a..c5453931 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -781,6 +781,9 @@ options which can be set when creating a room:
     The expected format of the state events are an object with ``type``,
     ``state_key`` and ``content`` keys set.
 
+    Takes precedence over events set by ``presets``, but gets overriden by
+    ``name`` and ``topic`` keys.
+
 Example::
 
   {

From c6375ed3d1743eb931e37d12b082c91a2b20d7b4 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 25 Sep 2015 15:09:15 +0100
Subject: [PATCH 164/467] Flesh out feature profiles section

Add table detailing the profiles. Add anchors to link through to each module
following a well-defined format (rather than the name of the module section).
Allow UTF-8 in the spec.
---
 specification/0-feature_profiles.rst          | 35 +++++++++++++++++++
 specification/modules/content_repo.rst        |  2 ++
 .../modules/end_to_end_encryption.rst         |  2 ++
 specification/modules/instant_messaging.rst   |  2 ++
 specification/modules/presence.rst            |  2 ++
 specification/modules/push_overview.rst       |  2 ++
 specification/modules/receipts.rst            |  2 ++
 .../modules/typing_notifications.rst          |  2 ++
 specification/modules/voip_events.rst         |  3 ++
 templating/build.py                           |  4 +--
 10 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 234e14db..dcf7f642 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -1,3 +1,38 @@
 Feature Profiles
 ================
 
+Matrix supports many different kinds of clients: from embedded IoT devices to
+desktop clients. Not all clients can provide the same feature sets as other
+clients e.g. due to lack of physical hardware such as not having a screen.
+Clients can fall into one of several profiles and each profile contains a set
+of features that the client MUST support. This section details a set of
+"feature profiles". Clients are expected to implement a profile in its entirety
+in order for it to be classified as that profile.
+
+Summary
+-------
+
+============================ ===== =========== ======== ========= ===== =====
+  Module / Profile            Web   Embed-Web   Mobile   Desktop   CLI   IoT
+============================ ===== =========== ======== ========= ===== =====
+ `End-to-End Encryption`_     YES                YES       YES     YES
+ `Instant Messaging`_         YES    YES         YES       YES     YES   YES
+ `Presence`_                  YES                YES       YES     YES
+ `Push Notifications`_                           YES
+ `Receipts`_                  YES                YES       YES     YES
+ `Typing Notifications`_      YES                YES       YES     YES
+ `VoIP`_                      YES                YES       YES
+ `Content Repository`_        YES                YES       YES     YES
+============================ ===== =========== ======== ========= ===== =====
+
+*Please see each module for more details on what clients need to implement.*
+
+.. _End-to-End Encryption: `module:e2e`_
+.. _Instant Messaging: `module:im`_
+.. _Presence: `module:presence`_
+.. _Push Notifications: `module:push`_
+.. _Receipts: `module:receipts`_
+.. _Typing Notifications: `module:typing`_
+.. _VoIP: `module:voip`_
+.. _Content Repository: `module:content`_
+
diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index 2c45ced7..c2cf3505 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -1,6 +1,8 @@
 Content repository
 ==================
 
+.. _module:content:
+
 HTTP API
 --------
 
diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst
index 02388152..e3a52613 100644
--- a/specification/modules/end_to_end_encryption.rst
+++ b/specification/modules/end_to_end_encryption.rst
@@ -1,6 +1,8 @@
 End-to-End Encryption
 =====================
 
+.. _module:e2e:
+
 .. TODO-doc
   - Why is this needed.
   - Overview of process
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 7f582ca4..43a06aa1 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -1,6 +1,8 @@
 Instant Messaging
 =================
 
+.. _module:im:
+
 Events
 ------
 
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index ddd2adff..cb71107f 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -1,5 +1,7 @@
 Presence
 ========
+
+.. _module:presence:
  
 Each user has the concept of presence information. This encodes the
 "availability" of that user, suitable for display on other user's clients.
diff --git a/specification/modules/push_overview.rst b/specification/modules/push_overview.rst
index 972a8eea..46028283 100644
--- a/specification/modules/push_overview.rst
+++ b/specification/modules/push_overview.rst
@@ -1,6 +1,8 @@
 Push Notifications
 ==================
 
+.. _module:push:
+
 Overview
 --------
 
diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst
index e2f83eea..9787682f 100644
--- a/specification/modules/receipts.rst
+++ b/specification/modules/receipts.rst
@@ -1,6 +1,8 @@
 Receipts
 --------
 
+.. _module:receipts:
+
 Receipts are used to publish which events in a room the user or their devices
 have interacted with. For example, which events the user has read. For
 efficiency this is done as "up to" markers, i.e. marking a particular event
diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst
index 25b714ab..614f6af7 100644
--- a/specification/modules/typing_notifications.rst
+++ b/specification/modules/typing_notifications.rst
@@ -1,6 +1,8 @@
 Typing Notifications
 --------------------
 
+.. _module:typing:
+
 Client APIs
 ~~~~~~~~~~~
 
diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst
index a5468237..60d61078 100644
--- a/specification/modules/voip_events.rst
+++ b/specification/modules/voip_events.rst
@@ -1,5 +1,8 @@
 Voice over IP
 -------------
+
+.. _module:voip:
+
 Matrix can also be used to set up VoIP calls. This is part of the core
 specification, although is at a relatively early stage. Voice (and video) over
 Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
diff --git a/templating/build.py b/templating/build.py
index 013248f4..6f465607 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -122,13 +122,13 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
 
     # check the input files and substitute in sections where required
     log("Parsing input template: %s" % file_stream.name)
-    temp = Template(file_stream.read())
+    temp = Template(file_stream.read().decode("utf-8"))
     log("Creating output for: %s" % file_stream.name)
     output = create_from_template(temp, sections)
     with open(
             os.path.join(out_dir, os.path.basename(file_stream.name)), "w"
             ) as f:
-        f.write(output)
+        f.write(output.encode("utf-8"))
     log("Output file for: %s" % file_stream.name)
     check_unaccessed("units", units)
 

From 9fac152d32cea2dff81ed5094bb47dcb793ac640 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 25 Sep 2015 15:26:58 +0100
Subject: [PATCH 165/467] Explain what the clients are and the column reference

---
 specification/0-feature_profiles.rst | 43 ++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index dcf7f642..4961c216 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -36,3 +36,46 @@ Summary
 .. _VoIP: `module:voip`_
 .. _Content Repository: `module:content`_
 
+Clients
+-------
+
+Stand-alone web client (``Web``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a web page which heavily uses Matrix for communication. Single-page web
+apps would be classified as a stand-alone web client, as would multi-page web
+apps which use Matrix on nearly every page.
+
+Embedded web client (``EmbedWeb``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a Matrix client which is embedded in another website, e.g. using
+iframes. These embedded clients are typically for a single purpose
+related to the website in question, and are not intended to be fully-fledged
+communication apps.
+
+Mobile client (``Mobile``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a Matrix client specifically designed for consumption on mobile devices.
+This is typically a mobile app but need not be so provided the feature set can
+be reached (e.g. if a mobile site could display push notifications it could be
+classified as a mobile client).
+
+Desktop client (``Desktop``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a native application which can run in its own environment outside a
+browser.
+
+Command Line Interface client (``CLI``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a client which is used via a text-based terminal.
+
+Internet of Things client (``IoT``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a client which is typically running on an embedded device such as a
+kettle, fridge or car.
+

From 510553ee009049616288a262aec3a01705e14429 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 25 Sep 2015 15:29:33 +0100
Subject: [PATCH 166/467] Remove smurf suffixes. Add anchor for feature
 profiles.

---
 specification/0-feature_profiles.rst | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 4961c216..ab977162 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -1,6 +1,8 @@
 Feature Profiles
 ================
 
+.. sect:feature-profiles:
+
 Matrix supports many different kinds of clients: from embedded IoT devices to
 desktop clients. Not all clients can provide the same feature sets as other
 clients e.g. due to lack of physical hardware such as not having a screen.
@@ -39,42 +41,42 @@ Summary
 Clients
 -------
 
-Stand-alone web client (``Web``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Stand-alone web (``Web``)
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 This is a web page which heavily uses Matrix for communication. Single-page web
 apps would be classified as a stand-alone web client, as would multi-page web
 apps which use Matrix on nearly every page.
 
-Embedded web client (``EmbedWeb``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Embedded web (``EmbedWeb``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 This is a Matrix client which is embedded in another website, e.g. using
 iframes. These embedded clients are typically for a single purpose
 related to the website in question, and are not intended to be fully-fledged
 communication apps.
 
-Mobile client (``Mobile``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Mobile (``Mobile``)
+~~~~~~~~~~~~~~~~~~~
 
 This is a Matrix client specifically designed for consumption on mobile devices.
 This is typically a mobile app but need not be so provided the feature set can
 be reached (e.g. if a mobile site could display push notifications it could be
 classified as a mobile client).
 
-Desktop client (``Desktop``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Desktop (``Desktop``)
+~~~~~~~~~~~~~~~~~~~~~
 
 This is a native application which can run in its own environment outside a
 browser.
 
-Command Line Interface client (``CLI``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Command Line Interface (``CLI``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 This is a client which is used via a text-based terminal.
 
-Internet of Things client (``IoT``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Internet of Things (``IoT``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 This is a client which is typically running on an embedded device such as a
 kettle, fridge or car.

From 9c882ea5e4df6dd68fa77e08235ded9e1f8a6e83 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 25 Sep 2015 16:24:31 +0100
Subject: [PATCH 167/467] Detect unused {{ vars }} and whine loudly.

---
 templating/build.py | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/templating/build.py b/templating/build.py
index 013248f4..ee3d79de 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -38,7 +38,7 @@ Processing
 """
 from batesian import AccessKeyStore
 
-from jinja2 import Environment, FileSystemLoader, StrictUndefined, Template
+from jinja2 import Environment, FileSystemLoader, StrictUndefined, Template, meta
 from argparse import ArgumentParser, FileType
 import importlib
 import json
@@ -122,7 +122,19 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
 
     # check the input files and substitute in sections where required
     log("Parsing input template: %s" % file_stream.name)
-    temp = Template(file_stream.read())
+    temp_str = file_stream.read()
+    # do sanity checking on the template to make sure they aren't reffing things
+    # which will never be replaced with a section.
+    ast = env.parse(temp_str)
+    template_vars = meta.find_undeclared_variables(ast)
+    unused_vars = [var for var in template_vars if var not in sections]
+    if len(unused_vars) > 0:
+        raise Exception(
+            "You have {{ variables }} which are not found in sections: %s" %
+            (unused_vars,)
+        )
+    # process the template
+    temp = Template(temp_str)
     log("Creating output for: %s" % file_stream.name)
     output = create_from_template(temp, sections)
     with open(

From e1f73f523318951d2303e2d538b26fb3478abeda Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 25 Sep 2015 18:09:17 +0100
Subject: [PATCH 168/467] Add a limit to filters

---
 api/client-server/v2_alpha/definitions/event_filter.json | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/api/client-server/v2_alpha/definitions/event_filter.json b/api/client-server/v2_alpha/definitions/event_filter.json
index b8ab1352..76d8be64 100644
--- a/api/client-server/v2_alpha/definitions/event_filter.json
+++ b/api/client-server/v2_alpha/definitions/event_filter.json
@@ -1,6 +1,11 @@
 {
   "type": "object",
   "properties": {
+    "limit": {
+      "type": "integer",
+      "description":
+        "The maximum number of events to return."
+    },
     "types": {
       "type": "array",
       "description":

From a0068e1adaa028d29bc00d5ad62dd5a5b6371b1b Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 25 Sep 2015 18:10:28 +0100
Subject: [PATCH 169/467] Draft documention for the v2 sync api

---
 .../v2_alpha/definitions/event_batch.json     |  12 ++
 .../v2_alpha/definitions/timeline_batch.json  |  14 ++
 api/client-server/v2_alpha/sync.yaml          | 167 ++++++++++++++++++
 3 files changed, 193 insertions(+)
 create mode 100644 api/client-server/v2_alpha/definitions/event_batch.json
 create mode 100644 api/client-server/v2_alpha/definitions/timeline_batch.json
 create mode 100644 api/client-server/v2_alpha/sync.yaml

diff --git a/api/client-server/v2_alpha/definitions/event_batch.json b/api/client-server/v2_alpha/definitions/event_batch.json
new file mode 100644
index 00000000..c7cdbfb3
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/event_batch.json
@@ -0,0 +1,12 @@
+{
+  "type": "object",
+  "properties": {
+    "events": {
+      "type": "array",
+      "description": "List of indicies into an events array",
+      "items": {
+        "type": "integer"
+      }
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/definitions/timeline_batch.json b/api/client-server/v2_alpha/definitions/timeline_batch.json
new file mode 100644
index 00000000..dc62724b
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/timeline_batch.json
@@ -0,0 +1,14 @@
+{
+  "type": "object",
+  "allOf": [{"$ref":"definitions/event_batch.json"}],
+  "properties": {
+    "limited": {
+      "type": "boolean",
+      "description": "Whether there are more events on the server"
+    },
+    "prev_batch": {
+      "type": "string",
+      "description": "If the batch was limited then this is a token that can be supplied to the server to retrieve more events"
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
new file mode 100644
index 00000000..c857d272
--- /dev/null
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -0,0 +1,167 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v2 sync API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+basePath: /_matrix/client/api/v2_alpha
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/sync":
+    get:
+      summary: Synchronise the client's state and receive new messages.
+      description: |-
+        Synchronise the client's state with the latest state on the server.
+        Client's use this API when they first log in to get an initial snapshot
+        of the state on the server, and then continue to call this API to get
+        incremental deltas to the state, and to receive new messages.
+      security:
+        - accessToken: []
+      parameters:
+        - in: query
+          name: filter
+          type: string
+          description: |-
+            The ID of a filter created using the filter API.
+        - in: query
+          name: since
+          type: string
+          description: |-
+            A point in time to continue a sync from.
+        - in: query
+          name: timeout
+          type: integer
+          description: |-
+            The maximum time to poll in milliseconds before returning this
+            request.
+      responses:
+        200:
+          description:
+            The initial snapshot or delta for the client to use to update their
+            state.
+          schema:
+            type: object
+            properties:
+              next_batch:
+                type: string
+                description: |-
+                  The batch token to supply in the ``since`` param of the next
+                  ``/sync`` request.
+              events:
+                type: array
+                description: |-
+                  A list of event objects that are referred to by index in the
+                  rest of the ``sync`` response.
+                items:
+                  type: object
+              rooms:
+                type: array
+                description: |-
+                  A list of rooms that the client needs to update.
+                items:
+                  type: object
+                  properties:
+                    room_id:
+                      type: string
+                      description: |-
+                        The ID of the room.
+                    state:
+                      description: |-
+                        The state updates for the room.
+                      allOf:
+                        - $ref: "definitions/event_batch.json"
+                    timeline:
+                      description: |-
+                        The timeline of messages and state changes in the room.
+                      allOf:
+                        - $ref: "definitions/timeline_batch.json"
+                    ephemeral:
+                      description: |-
+                        The ephemeral events in the room that aren't recorded
+                        in the timeline or state of the room. E.g. typing.
+                      allOf:
+                        - $ref: "definitions/event_batch.json"
+              public_user_data:
+                description: |-
+                  The updates to publically visible user data.
+                allOf:
+                  - $ref: "definitions/event_batch.json"
+              private_user_data:
+                description: |-
+                  Updates to the data which is private to the user but shared
+                  amoungst their devices.
+                allOf:
+                  - $ref: "definitions/event_batch.json"
+          examples:
+            application/json: |-
+              {
+                "next_batch": "s72595_4483_1934",
+                "events": [
+                  {
+                    "sender": "@bob:example.com",
+                    "type": "com.example.weird.setting",
+                    "content": {"setting1": true, "setting2": false}
+                  },
+                  {
+                    "sender": "@alice:example.com",
+                    "type": "m.profile.display_name",
+                    "content": {"display_name": "Alice"}
+                  },
+                  {
+                    "sender": "@alice:example.com",
+                    "type": "m.presence",
+                    "content": {"presence": "online"}
+                  },
+                  {
+                    "room_id": "!726s6s6q:example.com",
+                    "type": "m.typing",
+                    "content": {"user_ids": ["@alice:example.com"]}
+                  },
+                  {
+                    "sender": "@alice:example.com",
+                    "room_id": "!726s6s6q:example.com",
+                    "type": "m.room.member",
+                    "state_key": "@alice:example.com",
+                    "content": {"membership": "join"},
+                    "origin_server_ts": 1417731086795
+                  },
+                  {
+                    "sender": "@bob:example.com",
+                    "room_id": "!726s6s6q:example.com",
+                    "type": "m.room.member",
+                    "state_key": "@bob:example.com",
+                    "content": {"membership": "join"},
+                    "origin_server_ts": 1417731086795
+                  },
+                  {
+                    "sender": "@alice:example.com",
+                    "room_id": "!726s6s6q:example.com",
+                    "type": "m.room.message",
+                    "unsigned": {"age": "124524", "txn_id": "1234"},
+                    "content": {"body": "I am a fish", "msgtype": "m.text"},
+                    "origin_server_ts": 1417731086797
+                  }
+                ],
+                "private_user_data": {"events": [0]},
+                "public_user_data": {"events": [1, 2]},
+                "rooms": [{
+                  "room_id": "!726s6s6q:example.com",
+                  "ephemeral": {"events": [3]},
+                  "state": {"events": [4, 5]},
+                  "timeline": {
+                    "events": [5,6],
+                    "limited": true,
+                    "prev_batch": "t34-23535_0_0"
+                  }
+                }]
+              }

From de07586ab786d8cc8431dfc1b23030e9a77d6b57 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 25 Sep 2015 19:34:55 +0100
Subject: [PATCH 170/467] Rename 'events' to 'timeline' in the sync filter

---
 api/client-server/v2_alpha/definitions/sync_filter.json | 2 +-
 api/client-server/v2_alpha/filter.yaml                  | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v2_alpha/definitions/sync_filter.json b/api/client-server/v2_alpha/definitions/sync_filter.json
index c9c47646..bfc8b51d 100644
--- a/api/client-server/v2_alpha/definitions/sync_filter.json
+++ b/api/client-server/v2_alpha/definitions/sync_filter.json
@@ -9,7 +9,7 @@
             "The state events to include for rooms.",
           "allOf": [{"$ref": "definitions/event_filter.json"}]
         },
-        "events": {
+        "timeline": {
           "description":
             "The message and state update events to include for rooms.",
           "allOf": [{"$ref": "definitions/event_filter.json"}]
diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 5296f11a..3e564f4b 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -50,7 +50,7 @@ paths:
                     "types": ["m.room.*"],
                     "not_rooms": ["!726s6s6q:example.com"]
                   },
-                  "events": {
+                  "timeline": {
                     "types": ["m.room.message"],
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]
@@ -118,7 +118,7 @@ paths:
                     "types": ["m.room.*"],
                     "not_rooms": ["!726s6s6q:example.com"]
                   },
-                  "events": {
+                  "timeline": {
                     "types": ["m.room.message"],
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]

From c3b3b2df63ed8964d72517d0092b298988009d3f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 25 Sep 2015 19:37:41 +0100
Subject: [PATCH 171/467] Add "set_presence" in the sync parameters

---
 api/client-server/v2_alpha/sync.yaml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index c857d272..5018080e 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -38,6 +38,11 @@ paths:
           type: string
           description: |-
             A point in time to continue a sync from.
+        - in: query
+          name: set_presence
+          type: string
+          description: |-
+            Set the presence status of this client.
         - in: query
           name: timeout
           type: integer

From 8c22b715cabdb283de39a0f79bdef71847366323 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 09:29:07 +0100
Subject: [PATCH 172/467] Add title

---
 api/client-server/v1/sync.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index d82ee9e4..dbfa39b8 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -269,6 +269,7 @@ paths:
                       enum: ["invite", "join", "leave", "ban"]
                     invite:
                         type: object
+                        title: "InviteEvent"
                         description: "The invite event if `membership` is `invite`"
                         allOf:
                           - "$ref": "m.room.member"

From 317c2f20d31345a656c553dc239563cf9f2de78d Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 09:29:35 +0100
Subject: [PATCH 173/467] Draft login token spec

---
 specification/1-client_server_api.rst | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 93e3cb90..7f967337 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -197,6 +197,7 @@ This specification defines the following login types:
  - ``m.login.recaptcha``
  - ``m.login.oauth2``
  - ``m.login.email.identity``
+ - ``m.login.token``
  - ``m.login.dummy``
 
 Password-based
@@ -228,6 +229,29 @@ To respond to this type, reply with an auth dict as follows::
     "response": "<captcha response>"
   }
 
+Token-based
+~~~~~~~~~~~
+:Type:
+  ``m.login.token``
+:Description:
+  The client submits a username and token that was generated by the server.
+
+To respond to this type, reply with an auth dict as follows::
+
+  {
+    "type": "m.login.token",
+    "user": "<user_id or user localpart>",
+    "token": "<token>",
+    "nonce": "<client generated nonce>"
+  }
+
+The ``nonce`` should be a random string generated by the client for the
+request. The same ``nonce`` should be used if retrying the request.
+
+The ``token`` may be discovered from e.g. an email or dynamically generated QR
+code.
+
+
 OAuth2-based
 ~~~~~~~~~~~~
 :Type:

From 4c9f524cabbbc74a837743aaa043e977b67ae7d2 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 09:32:03 +0100
Subject: [PATCH 174/467] Neaten things up

---
 specification/1-client_server_api.rst | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 7f967337..e3817046 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -234,7 +234,7 @@ Token-based
 :Type:
   ``m.login.token``
 :Description:
-  The client submits a username and token that was generated by the server.
+  The client submits a username and token.
 
 To respond to this type, reply with an auth dict as follows::
 
@@ -248,9 +248,8 @@ To respond to this type, reply with an auth dict as follows::
 The ``nonce`` should be a random string generated by the client for the
 request. The same ``nonce`` should be used if retrying the request.
 
-The ``token`` may be discovered from e.g. an email or dynamically generated QR
-code.
-
+There are many ways a client may receive a ``token``, including via an email or
+from an existing logged in device.
 
 OAuth2-based
 ~~~~~~~~~~~~

From 393d2831399e07316686129418d21b671571bfe7 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 11:22:31 +0100
Subject: [PATCH 175/467] Add a "limit" to the timeline key in the example
 filter

---
 api/client-server/v2_alpha/filter.yaml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 3e564f4b..9bda2359 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -51,6 +51,7 @@ paths:
                     "not_rooms": ["!726s6s6q:example.com"]
                   },
                   "timeline": {
+                    "limit": 10,
                     "types": ["m.room.message"],
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]
@@ -119,6 +120,7 @@ paths:
                     "not_rooms": ["!726s6s6q:example.com"]
                   },
                   "timeline": {
+                    "limit": 10,
                     "types": ["m.room.message"],
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]

From 1aa916d690bac548625f5a460ad916a0c6fdf43d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 12:52:12 +0100
Subject: [PATCH 176/467] s/indicies/indices/

---
 api/client-server/v2_alpha/definitions/event_batch.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/definitions/event_batch.json b/api/client-server/v2_alpha/definitions/event_batch.json
index c7cdbfb3..ed2730c8 100644
--- a/api/client-server/v2_alpha/definitions/event_batch.json
+++ b/api/client-server/v2_alpha/definitions/event_batch.json
@@ -3,7 +3,7 @@
   "properties": {
     "events": {
       "type": "array",
-      "description": "List of indicies into an events array",
+      "description": "List of indices into an events array",
       "items": {
         "type": "integer"
       }

From f50e6d4c0ad8648dc601344b6ac67c468f4e25f8 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:02:52 +0100
Subject: [PATCH 177/467] Add x-example fields for v2 /sync

---
 api/client-server/v2_alpha/sync.yaml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 5018080e..d307a683 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -33,22 +33,26 @@ paths:
           type: string
           description: |-
             The ID of a filter created using the filter API.
+          x-example: "66696p746572"
         - in: query
           name: since
           type: string
           description: |-
             A point in time to continue a sync from.
+          x-example: "s72594_4483_1934"
         - in: query
           name: set_presence
           type: string
           description: |-
             Set the presence status of this client.
+          x-example: "online"
         - in: query
           name: timeout
           type: integer
           description: |-
             The maximum time to poll in milliseconds before returning this
             request.
+          x-example: 30000
       responses:
         200:
           description:

From 3204c2f2b6cedf748618c594b86c514c2bf0b2b2 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:04:37 +0100
Subject: [PATCH 178/467] Fix spelling

---
 api/client-server/v2_alpha/sync.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index d307a683..318a1ab5 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -108,7 +108,7 @@ paths:
               private_user_data:
                 description: |-
                   Updates to the data which is private to the user but shared
-                  amoungst their devices.
+                  amongst their devices.
                 allOf:
                   - $ref: "definitions/event_batch.json"
           examples:

From 00fd4aac264619ec6a53afecd0757149ad7c038c Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:06:04 +0100
Subject: [PATCH 179/467] s/publically/publicly/

---
 api/client-server/v2_alpha/sync.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 318a1ab5..e678d2d8 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -102,7 +102,7 @@ paths:
                         - $ref: "definitions/event_batch.json"
               public_user_data:
                 description: |-
-                  The updates to publically visible user data.
+                  The updates to publicly visible user data.
                 allOf:
                   - $ref: "definitions/event_batch.json"
               private_user_data:

From 97154cc6b2088ea30d80b0c2eff3196a40b9a484 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 13:09:54 +0100
Subject: [PATCH 180/467] s/nonce/txn_id/

---
 specification/1-client_server_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index e3817046..988f7bc4 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -242,7 +242,7 @@ To respond to this type, reply with an auth dict as follows::
     "type": "m.login.token",
     "user": "<user_id or user localpart>",
     "token": "<token>",
-    "nonce": "<client generated nonce>"
+    "txn_id": "<client generated nonce>"
   }
 
 The ``nonce`` should be a random string generated by the client for the

From db8f3c0d5969a6bc731a1e43e23359ea0dd160fe Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 28 Sep 2015 13:11:34 +0100
Subject: [PATCH 181/467] Desktop clients should have a GUI

---
 specification/0-feature_profiles.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index ab977162..86f8f3d8 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -67,7 +67,7 @@ classified as a mobile client).
 Desktop (``Desktop``)
 ~~~~~~~~~~~~~~~~~~~~~
 
-This is a native application which can run in its own environment outside a
+This is a native GUI application which can run in its own environment outside a
 browser.
 
 Command Line Interface (``CLI``)

From 41bc09ea22e8b18d0e509e666e30033dd3143e04 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:20:01 +0100
Subject: [PATCH 182/467] Split the event_filter into a event_filter and a
 room_event_filter that extends it.

So that we don't include "rooms" and "not_rooms" keys for the public_user_data
and private_user_data filters.
---
 .../v2_alpha/definitions/event_filter.json    | 16 --------------
 .../definitions/room_event_filter.json        | 21 +++++++++++++++++++
 .../v2_alpha/definitions/sync_filter.json     |  6 +++---
 3 files changed, 24 insertions(+), 19 deletions(-)
 create mode 100644 api/client-server/v2_alpha/definitions/room_event_filter.json

diff --git a/api/client-server/v2_alpha/definitions/event_filter.json b/api/client-server/v2_alpha/definitions/event_filter.json
index 76d8be64..c15b8133 100644
--- a/api/client-server/v2_alpha/definitions/event_filter.json
+++ b/api/client-server/v2_alpha/definitions/event_filter.json
@@ -37,22 +37,6 @@
       "items": {
         "type": "string"
       }
-    },
-    "rooms": {
-      "type": "array",
-      "description":
-        "A list of room IDs to include. If this list is absent then all rooms are included.",
-      "items": {
-        "type": "string"
-      }
-    },
-    "not_rooms": {
-      "type": "array",
-      "description":
-        "A list of room IDs to exclude. If this list is absent then no rooms are excluded.",
-      "items": {
-        "type": "string"
-      }
     }
   }
 }
diff --git a/api/client-server/v2_alpha/definitions/room_event_filter.json b/api/client-server/v2_alpha/definitions/room_event_filter.json
new file mode 100644
index 00000000..c234448e
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/room_event_filter.json
@@ -0,0 +1,21 @@
+{
+  "type": "object",
+  "properties": {
+    "rooms": {
+      "type": "array",
+      "description":
+        "A list of room IDs to include. If this list is absent then all rooms are included.",
+      "items": {
+        "type": "string"
+      }
+    },
+    "not_rooms": {
+      "type": "array",
+      "description":
+        "A list of room IDs to exclude. If this list is absent then no rooms are excluded.",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/definitions/sync_filter.json b/api/client-server/v2_alpha/definitions/sync_filter.json
index bfc8b51d..4ad3aa82 100644
--- a/api/client-server/v2_alpha/definitions/sync_filter.json
+++ b/api/client-server/v2_alpha/definitions/sync_filter.json
@@ -7,17 +7,17 @@
         "state": {
           "description":
             "The state events to include for rooms.",
-          "allOf": [{"$ref": "definitions/event_filter.json"}]
+          "allOf": [{"$ref": "definitions/room_event_filter.json"}]
         },
         "timeline": {
           "description":
             "The message and state update events to include for rooms.",
-          "allOf": [{"$ref": "definitions/event_filter.json"}]
+          "allOf": [{"$ref": "definitions/room_event_filter.json"}]
         },
         "ephemeral": {
           "description":
             "The events that aren't recorded in the room history, e.g. typing and receipts, to include for rooms.",
-          "allOf": [{"$ref": "definitions/event_filter.json"}]
+          "allOf": [{"$ref": "definitions/room_event_filter.json"}]
         }
       }
     },

From 940e22940dbde376db926b1b64fcda411cceffdd Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:28:58 +0100
Subject: [PATCH 183/467] Document how the "not_foo" keys interact with the
 "foo" keys

---
 api/client-server/v2_alpha/definitions/event_filter.json      | 4 ++--
 api/client-server/v2_alpha/definitions/room_event_filter.json | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v2_alpha/definitions/event_filter.json b/api/client-server/v2_alpha/definitions/event_filter.json
index c15b8133..269bb5f0 100644
--- a/api/client-server/v2_alpha/definitions/event_filter.json
+++ b/api/client-server/v2_alpha/definitions/event_filter.json
@@ -17,7 +17,7 @@
     "not_types": {
       "type": "array",
       "description":
-        "A list of event types to exclude. If this list is absent then no event types are excluded.",
+        "A list of event types to exclude. If this list is absent then no event types are excluded. A matching type will be excluded even if it is listed in the 'types' filter",
       "items": {
         "type": "string"
       }
@@ -33,7 +33,7 @@
     "not_senders": {
       "type": "array",
       "description":
-        "A list of sender IDs to exclude. If this list is absent then no senders are excluded.",
+        "A list of sender IDs to exclude. If this list is absent then no senders are excluded. A matching sender will be excluded even if it is listed in the 'senders' filter",
       "items": {
         "type": "string"
       }
diff --git a/api/client-server/v2_alpha/definitions/room_event_filter.json b/api/client-server/v2_alpha/definitions/room_event_filter.json
index c234448e..5be0fcd2 100644
--- a/api/client-server/v2_alpha/definitions/room_event_filter.json
+++ b/api/client-server/v2_alpha/definitions/room_event_filter.json
@@ -1,5 +1,6 @@
 {
   "type": "object",
+  "allOf": [{"$ref": "definitions/event_filter.json"}],
   "properties": {
     "rooms": {
       "type": "array",
@@ -12,7 +13,7 @@
     "not_rooms": {
       "type": "array",
       "description":
-        "A list of room IDs to exclude. If this list is absent then no rooms are excluded.",
+        "A list of room IDs to exclude. If this list is absent then no rooms are excluded. A matching room will be excluded even if it is listed in the 'rooms' filter",
       "items": {
         "type": "string"
       }

From c2fc1a2fb19c6a46cc2bde736222f855f0f8ba3d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 28 Sep 2015 13:41:31 +0100
Subject: [PATCH 184/467] Add basic module template.

This has the core sections for events/server/client/security.
---
 specification/modules/_template.rst | 47 +++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100644 specification/modules/_template.rst

diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst
new file mode 100644
index 00000000..6ea7aaa3
--- /dev/null
+++ b/specification/modules/_template.rst
@@ -0,0 +1,47 @@
+Module Heading
+==============
+
+.. _module:short-name:
+
+A short summary of the module. What features does this module provide? An anchor
+should be specified at the top of the module using the format ``module:name``.
+
+Complicated modules may wish to have architecture diagrams or event flows
+(e.g. VoIP call flows) here. Custom subsections can be included but they should
+be used *sparingly* to reduce the risk of putting client or server behaviour
+information in these custom sections.
+
+Events
+------
+List the new event types introduced by this module, if any. If there are no
+new events, this section can be omitted. Event types should be done as
+subsections. The section is intended to document the "common shared event
+structure" between client and server. Deviations from this shared structure
+should be documented in the relevant behaviour section.
+
+example.event.type
+~~~~~~~~~~~~~~~~~~
+There should be JSON Schema docs for this event. You can insert a template like
+so:
+
+{{example_event_type_event}}
+
+Client behaviour
+----------------
+List any new HTTP endpoints. List the steps the client needs to take to
+correctly process this module. Listing what data structures the client should be
+storing to aid implementation is recommended. 
+
+Server behaviour
+----------------
+Does the server need to handle any of the new events in a special way (e.g.
+typing timeouts, presence). Advice on how to persist events and/or requests are
+recommended to aid implementation.
+
+Security considerations
+-----------------------
+This includes privacy leaks: for example leaking presence info. How do
+misbehaving clients or servers impact this module? This section should always be
+included, if only to say "we've thought about it but there isn't anything to do
+here".
+

From c115b4c2f4cfda43bdb6409edf9f14bfb8004151 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:47:04 +0100
Subject: [PATCH 185/467] Document the valid values for the "set_presence"
 parameter

---
 api/client-server/v2_alpha/sync.yaml | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index e678d2d8..0fe9173d 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -43,9 +43,14 @@ paths:
         - in: query
           name: set_presence
           type: string
+          enum: ["offline"]
           description: |-
-            Set the presence status of this client.
-          x-example: "online"
+            Controls whether the client is automatically marked as online by
+            polling this API. If this parameter is omitted then the client is
+            automatically marked as online when it uses this API. Otherwise if
+            the parameter is set to "offline" then the client is not marked as
+            being online when it uses this API.
+          x-example: "offline"
         - in: query
           name: timeout
           type: integer

From 14b42a41d653f601d7f6a7cc61539a1ccf7f55c7 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 13:58:07 +0100
Subject: [PATCH 186/467] Document wildcard's in filters

---
 api/client-server/v2_alpha/definitions/event_filter.json  | 8 ++++----
 .../v2_alpha/definitions/room_event_filter.json           | 4 ++--
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/api/client-server/v2_alpha/definitions/event_filter.json b/api/client-server/v2_alpha/definitions/event_filter.json
index 269bb5f0..1cdcb1f4 100644
--- a/api/client-server/v2_alpha/definitions/event_filter.json
+++ b/api/client-server/v2_alpha/definitions/event_filter.json
@@ -9,7 +9,7 @@
     "types": {
       "type": "array",
       "description":
-        "A list of event types to include. If this list is absent then all event types are included.",
+        "A list of event types to include. If this list is absent then all event types are included. A '*' can be used as a wildcard to match any sequence of characters.",
       "items": {
         "type": "string"
       }
@@ -17,7 +17,7 @@
     "not_types": {
       "type": "array",
       "description":
-        "A list of event types to exclude. If this list is absent then no event types are excluded. A matching type will be excluded even if it is listed in the 'types' filter",
+        "A list of event types to exclude. If this list is absent then no event types are excluded. A matching type will be excluded even if it is listed in the 'types' filter. A '*' can be used as a wildcard to match any sequence of characters.",
       "items": {
         "type": "string"
       }
@@ -25,7 +25,7 @@
     "senders": {
       "type": "array",
       "description":
-        "A list of senders IDs to include. If this list is absent then all senders are included.",
+        "A list of senders IDs to include. If this list is absent then all senders are included. A '*' can be used as a wildcard to match any sequence of characters.",
       "items": {
         "type": "string"
       }
@@ -33,7 +33,7 @@
     "not_senders": {
       "type": "array",
       "description":
-        "A list of sender IDs to exclude. If this list is absent then no senders are excluded. A matching sender will be excluded even if it is listed in the 'senders' filter",
+        "A list of sender IDs to exclude. If this list is absent then no senders are excluded. A matching sender will be excluded even if it is listed in the 'senders' filter. A '*' can be used as a wildcard to match any sequence of characters.",
       "items": {
         "type": "string"
       }
diff --git a/api/client-server/v2_alpha/definitions/room_event_filter.json b/api/client-server/v2_alpha/definitions/room_event_filter.json
index 5be0fcd2..86375781 100644
--- a/api/client-server/v2_alpha/definitions/room_event_filter.json
+++ b/api/client-server/v2_alpha/definitions/room_event_filter.json
@@ -5,7 +5,7 @@
     "rooms": {
       "type": "array",
       "description":
-        "A list of room IDs to include. If this list is absent then all rooms are included.",
+        "A list of room IDs to include. If this list is absent then all rooms are included. A '*' can be used as a wildcard to match any sequence of characters.",
       "items": {
         "type": "string"
       }
@@ -13,7 +13,7 @@
     "not_rooms": {
       "type": "array",
       "description":
-        "A list of room IDs to exclude. If this list is absent then no rooms are excluded. A matching room will be excluded even if it is listed in the 'rooms' filter",
+        "A list of room IDs to exclude. If this list is absent then no rooms are excluded. A matching room will be excluded even if it is listed in the 'rooms' filter. A '*' can be used as a wildcard to match any sequence of characters.",
       "items": {
         "type": "string"
       }

From 643468e9141902499550ace65da1bf8eb7e53bc5 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 28 Sep 2015 14:01:54 +0100
Subject: [PATCH 187/467] Mention swagger. Clarify how event type template vars
 are formed.

---
 specification/modules/_template.rst | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst
index 6ea7aaa3..967892a5 100644
--- a/specification/modules/_template.rst
+++ b/specification/modules/_template.rst
@@ -19,18 +19,24 @@ subsections. The section is intended to document the "common shared event
 structure" between client and server. Deviations from this shared structure
 should be documented in the relevant behaviour section.
 
-example.event.type
-~~~~~~~~~~~~~~~~~~
-There should be JSON Schema docs for this event. You can insert a template like
-so:
+m.example.event.type
+~~~~~~~~~~~~~~~~~~~~
+There should be JSON Schema docs for this event. Once there is JSON schema,
+there will be a template variable with dots in the event type replaced with
+underscores. You can insert a template like so:
 
-{{example_event_type_event}}
+{{m_example_event_type_event}}
 
 Client behaviour
 ----------------
-List any new HTTP endpoints. List the steps the client needs to take to
+List any new HTTP endpoints. These endpoints should be docced using Swagger. You
+can insert a template for swagger docs like so:
+
+{{name-of-yaml-file-without-file-ext_http_api}}
+
+List the steps the client needs to take to
 correctly process this module. Listing what data structures the client should be
-storing to aid implementation is recommended. 
+storing to aid implementation is recommended.
 
 Server behaviour
 ----------------

From 70518ae655eb9fae6ac49a3a1ddea700c7defc44 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 28 Sep 2015 14:03:54 +0100
Subject: [PATCH 188/467] Clarify template suffixes

---
 specification/modules/_template.rst | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst
index 967892a5..bb662c05 100644
--- a/specification/modules/_template.rst
+++ b/specification/modules/_template.rst
@@ -23,14 +23,16 @@ m.example.event.type
 ~~~~~~~~~~~~~~~~~~~~
 There should be JSON Schema docs for this event. Once there is JSON schema,
 there will be a template variable with dots in the event type replaced with
-underscores. You can insert a template like so:
+underscores and the suffix ``_event``. You can insert a template like so:
 
 {{m_example_event_type_event}}
 
 Client behaviour
 ----------------
-List any new HTTP endpoints. These endpoints should be docced using Swagger. You
-can insert a template for swagger docs like so:
+List any new HTTP endpoints. These endpoints should be docced using Swagger.
+Once there is Swagger, there will be a template variable based on the name of
+the YAML file with the suffix ``_http_api``. You can insert a template for
+swagger docs like so:
 
 {{name-of-yaml-file-without-file-ext_http_api}}
 

From 5abea1f2bc2d36789164a4682cb8a668fb684845 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 28 Sep 2015 14:07:34 +0100
Subject: [PATCH 189/467] Minor tweaks

---
 specification/modules/_template.rst | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/specification/modules/_template.rst b/specification/modules/_template.rst
index bb662c05..9eee9843 100644
--- a/specification/modules/_template.rst
+++ b/specification/modules/_template.rst
@@ -15,12 +15,12 @@ Events
 ------
 List the new event types introduced by this module, if any. If there are no
 new events, this section can be omitted. Event types should be done as
-subsections. The section is intended to document the "common shared event
+subsections. This section is intended to document the "common shared event
 structure" between client and server. Deviations from this shared structure
 should be documented in the relevant behaviour section.
 
-m.example.event.type
-~~~~~~~~~~~~~~~~~~~~
+``m.example.event.type``
+~~~~~~~~~~~~~~~~~~~~~~~~
 There should be JSON Schema docs for this event. Once there is JSON schema,
 there will be a template variable with dots in the event type replaced with
 underscores and the suffix ``_event``. You can insert a template like so:
@@ -29,7 +29,7 @@ underscores and the suffix ``_event``. You can insert a template like so:
 
 Client behaviour
 ----------------
-List any new HTTP endpoints. These endpoints should be docced using Swagger.
+List any new HTTP endpoints. These endpoints should be documented using Swagger.
 Once there is Swagger, there will be a template variable based on the name of
 the YAML file with the suffix ``_http_api``. You can insert a template for
 swagger docs like so:
@@ -37,14 +37,15 @@ swagger docs like so:
 {{name-of-yaml-file-without-file-ext_http_api}}
 
 List the steps the client needs to take to
-correctly process this module. Listing what data structures the client should be
-storing to aid implementation is recommended.
+correctly process this module. List what data structures the client should be
+storing in order to aid implementation.
 
 Server behaviour
 ----------------
 Does the server need to handle any of the new events in a special way (e.g.
 typing timeouts, presence). Advice on how to persist events and/or requests are
-recommended to aid implementation.
+recommended to aid implementation. Federation-specific logic should be included
+here.
 
 Security considerations
 -----------------------

From 9dd3b073948740b202e7301624eef6beaebd27c0 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 14:00:56 +0100
Subject: [PATCH 190/467] Allow '.' characters in event field names to be
 escaped with '\' so that fields including a '.' can be included in a filter.

I considered replacing '.' with '/'. Since '/' was less likely to appear
in event field names. However if we used '\' to escape a literal '/' we
risk confusing it with the JSON escape '\/'.
---
 api/client-server/v2_alpha/definitions/sync_filter.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/definitions/sync_filter.json b/api/client-server/v2_alpha/definitions/sync_filter.json
index 4ad3aa82..b4f87e5f 100644
--- a/api/client-server/v2_alpha/definitions/sync_filter.json
+++ b/api/client-server/v2_alpha/definitions/sync_filter.json
@@ -39,7 +39,7 @@
     "event_fields": {
       "type": "array",
       "description":
-        "List of event fields to include. If this list is absent then all fields are included. The entries may include '.' charaters to indicate sub-fields. So ['content.body'] will include the 'body' field of the 'content' object. A server may include more fields than were requested.",
+        "List of event fields to include. If this list is absent then all fields are included. The entries may include '.' charaters to indicate sub-fields. So ['content.body'] will include the 'body' field of the 'content' object. A literal '.' character in a field name may be escaped using a '\\'. A server may include more fields than were requested.",
       "items": {
         "type": "string"
       }

From 6ad6c40147cf5d80801aa076f76dad2bd54d609e Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 28 Sep 2015 14:11:45 +0100
Subject: [PATCH 191/467] List the allowed values for the 'event_format' as an
 enum

---
 api/client-server/v2_alpha/definitions/sync_filter.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/definitions/sync_filter.json b/api/client-server/v2_alpha/definitions/sync_filter.json
index b4f87e5f..b909830b 100644
--- a/api/client-server/v2_alpha/definitions/sync_filter.json
+++ b/api/client-server/v2_alpha/definitions/sync_filter.json
@@ -34,7 +34,8 @@
     "event_format": {
       "description":
         "The format to use for events. 'client' will return the events in a format suitable for clients. 'federation' will return the raw event as receieved over federation. The default is 'client'.",
-      "type": "string"
+      "type": "string",
+      "enum": ["client", "federation"]
     },
     "event_fields": {
       "type": "array",

From 4776e0c04ce70a6a9bc11ec24e3b65c38049573d Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 14:49:55 +0100
Subject: [PATCH 192/467] Add creation_content key

---
 specification/1-client_server_api.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 93e3cb90..260832c9 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -750,6 +750,16 @@ options which can be set when creating a room:
     This will tell the server to invite everyone in the list to the newly
     created room.
 
+``creation_content``
+  Type:
+    Object
+  Optional:
+    Yes
+  Value:
+    Extra keys to be added to the content of the ``m.room.create``
+  Description:
+    Allows clients to add keys to the content of ``m.room.create``.
+
 Example::
 
   {

From ce53a171800d765b9bcc86e019ec11d4583ed4ea Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 14:51:10 +0100
Subject: [PATCH 193/467] Add txn_id rationale

---
 specification/1-client_server_api.rst | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 988f7bc4..89698c66 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -251,6 +251,13 @@ request. The same ``nonce`` should be used if retrying the request.
 There are many ways a client may receive a ``token``, including via an email or
 from an existing logged in device.
 
+The ``txn_id`` may be used by the server to disallow other devices from using
+the token, thus providing "single use" tokens while still allowing the device
+to retry the request. This would be done by tying the token to the ``txn_id``
+server side, as well as potentially invalidating the token completely once the
+device has successfully logged in (e.g. when we receive a request from the
+newly provisioned access_token).
+
 OAuth2-based
 ~~~~~~~~~~~~
 :Type:

From 6e6bc8a5a9745627eb172dd98c77c140e89f1a95 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 14:51:44 +0100
Subject: [PATCH 194/467] Mandate macaroon

---
 specification/1-client_server_api.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 89698c66..893aec73 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -258,6 +258,8 @@ server side, as well as potentially invalidating the token completely once the
 device has successfully logged in (e.g. when we receive a request from the
 newly provisioned access_token).
 
+The ``token`` must be a macaroon.
+
 OAuth2-based
 ~~~~~~~~~~~~
 :Type:

From affc2cfc923222da6a10bd6c21dd7d24c8031747 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 28 Sep 2015 16:39:54 +0100
Subject: [PATCH 195/467] Add 'm.federate' flag to 'm.room.create;

---
 event-schemas/schema/v1/m.room.create | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index f2e1ee92..bcbb10d2 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -12,6 +12,10 @@
                 "creator": {
                     "type": "string",
                     "description": "The ``user_id`` of the room creator. This is set by the homeserver."
+                },
+                "m.federate": {
+                    "type": "boolean",
+                    "description": "Whether users on other servers can join this room."
                 }
             },
             "required": ["creator"]

From 50e1b4c3a7147cb2a03fab56ff0403b998ef559e Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 29 Sep 2015 09:17:33 +0100
Subject: [PATCH 196/467] Fix up rst

---
 api/client-server/v1/sync.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index dbfa39b8..8fe30056 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -270,9 +270,9 @@ paths:
                     invite:
                         type: object
                         title: "InviteEvent"
-                        description: "The invite event if `membership` is `invite`"
+                        description: "The invite event if ``membership`` is ``invite``"
                         allOf:
-                          - "$ref": "m.room.member"
+                          - "$ref": "v1-event-schema/m.room.member"
                     messages:
                       type: object
                       title: PaginationChunk

From 083a76096f0edf6bf7810bd4f617d532499367ae Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 29 Sep 2015 12:41:07 +0100
Subject: [PATCH 197/467] Support viewing the spec at head

---
 scripts/speculator/main.go | 34 ++++++++++++++++++++++------------
 1 file changed, 22 insertions(+), 12 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index b4fe2aef..2609fa7e 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -136,20 +136,30 @@ func generateAt(repo, sha string) (dst string, err error) {
 }
 
 func serveSpec(w http.ResponseWriter, req *http.Request) {
-	pr, err := lookupPullRequest(*req.URL, "/spec")
-	if err != nil {
-		writeError(w, 400, err)
-		return
-	}
+	var cloneURL string
+	var sha string
+
+	if strings.ToLower(req.URL.Path) == "/spec/head" {
+		cloneURL = "https://github.com/matrix-org/matrix-doc.git"
+		sha = "HEAD"
+	} else {
+		pr, err := lookupPullRequest(*req.URL, "/spec")
+		if err != nil {
+			writeError(w, 400, err)
+			return
+		}
 
-	// We're going to run whatever Python is specified in the pull request, which
-	// may do bad things, so only trust people we trust.
-	if err := checkAuth(pr); err != nil {
-		writeError(w, 403, err)
-		return
+		// We're going to run whatever Python is specified in the pull request, which
+		// may do bad things, so only trust people we trust.
+		if err := checkAuth(pr); err != nil {
+			writeError(w, 403, err)
+			return
+		}
+		cloneURL = pr.Head.Repo.CloneURL
+		sha = pr.Head.SHA
 	}
 
-	dst, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
+	dst, err := generateAt(cloneURL, sha)
 	defer os.RemoveAll(dst)
 	if err != nil {
 		writeError(w, 500, err)
@@ -287,7 +297,7 @@ func listPulls(w http.ResponseWriter, req *http.Request) {
 		s += fmt.Sprintf(`<li>%d: <a href="%s">%s</a>: <a href="%s">%s</a>: <a href="spec/%d">spec</a> <a href="diff/html/%d">spec diff</a> <a href="diff/rst/%d">rst diff</a></li>`,
 			pull.Number, pull.User.HTMLURL, pull.User.Login, pull.HTMLURL, pull.Title, pull.Number, pull.Number, pull.Number)
 	}
-	s += "</ul></body>"
+	s += `</ul><div><a href="spec/head">View the spec at head</a></div></body>`
 	io.WriteString(w, s)
 }
 

From 7b4c8a9f68624e106f3b0ee8aca502d98b88d615 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 29 Sep 2015 16:19:27 +0100
Subject: [PATCH 198/467] Indicate default for m.federate key

---
 event-schemas/schema/v1/m.room.create | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.create b/event-schemas/schema/v1/m.room.create
index bcbb10d2..348b6d86 100644
--- a/event-schemas/schema/v1/m.room.create
+++ b/event-schemas/schema/v1/m.room.create
@@ -15,7 +15,7 @@
                 },
                 "m.federate": {
                     "type": "boolean",
-                    "description": "Whether users on other servers can join this room."
+                    "description": "Whether users on other servers can join this room. Defaults to ``true`` if key does not exist."
                 }
             },
             "required": ["creator"]

From 83b9497664f030ac2e1d3bae1a5faddb8425250f Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 29 Sep 2015 16:21:10 +0100
Subject: [PATCH 199/467] Add context and example

---
 specification/1-client_server_api.rst | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 260832c9..dd58b357 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -756,7 +756,8 @@ options which can be set when creating a room:
   Optional:
     Yes
   Value:
-    Extra keys to be added to the content of the ``m.room.create``
+    Extra keys to be added to the content of the ``m.room.create``. The server
+    will clober certain keys, e.g. ``creator``.
   Description:
     Allows clients to add keys to the content of ``m.room.create``.
 
@@ -766,7 +767,10 @@ Example::
     "visibility": "public",
     "room_alias_name": "thepub",
     "name": "The Grand Duke Pub",
-    "topic": "All about happy hour"
+    "topic": "All about happy hour",
+    "creation_content": {
+        "m.federate": false
+    }
   }
 
 The home server will create a ``m.room.create`` event when the room is created,

From 82c27884baed11d5dd61bb84786ce05d5c5e47c3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 29 Sep 2015 16:29:16 +0100
Subject: [PATCH 200/467] Make E2E optional

---
 specification/0-feature_profiles.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 86f8f3d8..01997dd5 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -17,7 +17,7 @@ Summary
 ============================ ===== =========== ======== ========= ===== =====
   Module / Profile            Web   Embed-Web   Mobile   Desktop   CLI   IoT
 ============================ ===== =========== ======== ========= ===== =====
- `End-to-End Encryption`_     YES                YES       YES     YES
+ `End-to-End Encryption`_                                             
  `Instant Messaging`_         YES    YES         YES       YES     YES   YES
  `Presence`_                  YES                YES       YES     YES
  `Push Notifications`_                           YES

From cdf9f011e9eb85cde0840548b1b9a1e5b7885b4e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 29 Sep 2015 16:33:34 +0100
Subject: [PATCH 201/467] Add room history visibility as a module.

---
 specification/0-feature_profiles.rst         | 2 ++
 specification/modules/history_visibility.rst | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 01997dd5..59e015e9 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -25,6 +25,7 @@ Summary
  `Typing Notifications`_      YES                YES       YES     YES
  `VoIP`_                      YES                YES       YES
  `Content Repository`_        YES                YES       YES     YES
+ `History Visibility`_        YES                YES       YES     YES
 ============================ ===== =========== ======== ========= ===== =====
 
 *Please see each module for more details on what clients need to implement.*
@@ -37,6 +38,7 @@ Summary
 .. _Typing Notifications: `module:typing`_
 .. _VoIP: `module:voip`_
 .. _Content Repository: `module:content`_
+.. _History Visibility: `module:history-visibility`_
 
 Clients
 -------
diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst
index 01c2e419..371282bd 100644
--- a/specification/modules/history_visibility.rst
+++ b/specification/modules/history_visibility.rst
@@ -1,6 +1,8 @@
 Room History Visibility
 -----------------------
 
+.. _module:history-visibility:
+
 Whether a member of a room can see the events that happened in a room from
 before they joined the room is controlled by the ``history_visibility`` key
 of the ``m.room.history_visibility`` state event. The valid values for

From 0a04672d764e070fc0199c958b8c006498314fbf Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 29 Sep 2015 17:57:44 +0100
Subject: [PATCH 202/467] Start converting the presence module. Add Rationale
 admonition.

---
 scripts/nature.css                    |   6 ++
 specification/1-client_server_api.rst |  21 +++++
 specification/modules/presence.rst    | 114 ++++++++++++++++----------
 3 files changed, 97 insertions(+), 44 deletions(-)

diff --git a/scripts/nature.css b/scripts/nature.css
index 9d67f689..f1b4edec 100644
--- a/scripts/nature.css
+++ b/scripts/nature.css
@@ -282,3 +282,9 @@ td[colspan]:not([colspan="1"]) {
 thead {
     background: #eeeeee;
 }
+
+div.admonition-rationale {
+    background-color: #efe;
+    border: 1px solid #ccc;
+}
+
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 893aec73..4403e0d3 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1092,6 +1092,27 @@ Profiles
 
 {{profile_http_api}}
 
+Events on Change of Profile Information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Because the profile display name and avatar information are likely to be used in
+many places of a client's display, changes to these fields cause an automatic
+propagation event to occur, informing likely-interested parties of the new
+values. This change is conveyed using two separate mechanisms:
+
+ - a ``m.room.member`` event is sent to every room the user is a member of,
+   to update the ``displayname`` and ``avatar_url``.
+ - a ``m.presence`` presence status update is sent, again containing the new
+   values of the ``displayname`` and ``avatar_url`` keys, in addition to the
+   required ``presence`` key containing the current presence state of the user.
+
+Both of these should be done automatically by the home server when a user
+successfully changes their display name or avatar URL fields.
+
+Additionally, when home servers emit room membership events for their own
+users, they should include the display name and avatar URL fields in these
+events so that clients already have these details to hand, and do not have to
+perform extra round trips to query it.
+
 Security
 --------
 
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index ddd2adff..4c84359a 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -1,63 +1,89 @@
 Presence
 ========
  
-Each user has the concept of presence information. This encodes the
-"availability" of that user, suitable for display on other user's clients.
+Each user has presence information associated with them. This encodes the
+"availability" of that user, suitable for display on other clients.
 This is transmitted as an ``m.presence`` event and is one of the few events
-which are sent *outside the context of a room*. The basic piece of presence
-information is represented by the ``presence`` key, which is an enum of one
-of the following:
+which are sent *outside the context of a room*. Their presence state is
+represented by the ``presence`` key, which is an enum of one of the following:
 
       - ``online`` : The default state when the user is connected to an event
         stream.
-      - ``unavailable`` : The user is not reachable at this time.
-      - ``offline`` : The user is not connected to an event stream.
+      - ``unavailable`` : The user is not reachable at this time e.g. they are
+        idle.
+      - ``offline`` : The user is not connected to an event stream or is
+        explicitly suppressing their profile information from being sent.
       - ``free_for_chat`` : The user is generally willing to receive messages
         moreso than default.
-      - ``hidden`` : Behaves as offline, but allows the user to see the client
-        state anyway and generally interact with client features. (Not yet
-        implemented in synapse).
-
-In addition, the server maintains a timestamp of the last time it saw a
-pro-active event from the user; either sending a message to a room, or
-changing presence state from a lower to a higher level of availability
-(thus: changing state from ``unavailable`` to ``online`` counts as a
-proactive event, whereas in the other direction it will not). This timestamp
-is presented via a key called ``last_active_ago``, which gives the relative
-number of milliseconds since the message is generated/emitted that the user
-was last seen active.
 
 Events
 ------
 
 {{presence_events}}
 
-Presence HTTP API
------------------
-.. TODO-spec
-  - Define how users receive presence invites, and how they accept/decline them
+Client behaviour
+----------------
+
+Clients can manually set/get their presence using the HTTP APIs listed below.
 
 {{presence_http_api}}
-    
 
-Events on Change of Profile Information
----------------------------------------
-Because the profile displayname and avatar information are likely to be used in
-many places of a client's display, changes to these fields cause an automatic
-propagation event to occur, informing likely-interested parties of the new
-values. This change is conveyed using two separate mechanisms:
-
- - a ``m.room.member`` event is sent to every room the user is a member of,
-   to update the ``displayname`` and ``avatar_url``.
- - a ``m.presence`` presence status update is sent, again containing the new values of the
-   ``displayname`` and ``avatar_url`` keys, in addition to the required
-   ``presence`` key containing the current presence state of the user.
-
-Both of these should be done automatically by the home server when a user
-successfully changes their displayname or avatar URL fields.
-
-Additionally, when home servers emit room membership events for their own
-users, they should include the displayname and avatar URL fields in these
-events so that clients already have these details to hand, and do not have to
-perform extra roundtrips to query it.
+Idle timeout
+~~~~~~~~~~~~
+
+Clients SHOULD implement an "idle timeout". This is a timer which fires after
+a period of inactivity on the client. The definition of inactivity varies
+depending on the client. For example, web implementations may determine
+inactivity to be not moving the mouse for a certain period of time. When this
+timer fires it should set the presence state to ``unavailable``. When the user
+becomes active again (e.g. by moving the mouse) the client should set the
+presence state to ``online``. A timeout value between 1 and 5 minutes is
+recommended. 
+
+Server behaviour
+----------------
+
+Propagating profile information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Because the profile display name and avatar information are likely to be used in
+many places of a client's display, changes to these fields SHOULD cause an
+automatic propagation event to occur, informing likely-interested parties of the
+new values. One of these change mechanisms SHOULD be via ``m.presence`` events.
+These events should set ``displayname`` and ``avatar_url`` to the new values
+along with the presence-specific keys. This SHOULD be done automatically by the
+home server when a user successfully changes their display name or avatar URL.
+
+.. admonition:: Rationale
+
+  The intention for sending this information in ``m.presence`` is so that any
+  "user list" can display the *current* name/presence for a user ID outside the
+  scope of a room (e.g. a user page which has a "start conversation" button).
+  This is bundled into a single event to avoid "flickering" on this page which
+  can occur if you received presence first and then display name later (the
+  user's name would flicker from their user ID to the display name).
+
+
+Last active ago
+~~~~~~~~~~~~~~~
+The server maintains a timestamp of the last time it saw a
+pro-active event from the user. A pro-active event may be sending a message to a
+room or changing presence state to a higher level of availability. Levels of
+availability are defined from low to high as follows:
+
+      - ``offline``
+      - ``unavailable``
+      - ``online``
+      - ``free_for_chat``
+
+Based on this list, changing state from ``unavailable`` to ``online`` counts as
+a pro-active event, whereas ``online`` to ``unavailable`` does not. This
+timestamp is presented via a key called ``last_active_ago`` which gives the
+relative number of milliseconds since the pro-active event.
+
+Security considerations
+-----------------------
+    
+Presence information is shared with all users who share a room with the target
+user. In large public rooms this could be undesirable.
 

From 218cf94ead2dacce3b3f4443250465ef2f23fd24 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 29 Sep 2015 19:10:31 +0100
Subject: [PATCH 203/467] Replace the events array with events_map inside the
 room objects. Only use indirection for the state and timeline events. Use
 event_ids to reference the state and timeline events.

---
 .../v2_alpha/definitions/event_batch.json     |   4 +-
 .../definitions/room_event_batch.json         |  12 ++
 .../v2_alpha/definitions/timeline_batch.json  |   2 +-
 api/client-server/v2_alpha/sync.yaml          | 134 ++++++++++--------
 4 files changed, 90 insertions(+), 62 deletions(-)
 create mode 100644 api/client-server/v2_alpha/definitions/room_event_batch.json

diff --git a/api/client-server/v2_alpha/definitions/event_batch.json b/api/client-server/v2_alpha/definitions/event_batch.json
index ed2730c8..75762d75 100644
--- a/api/client-server/v2_alpha/definitions/event_batch.json
+++ b/api/client-server/v2_alpha/definitions/event_batch.json
@@ -3,9 +3,9 @@
   "properties": {
     "events": {
       "type": "array",
-      "description": "List of indices into an events array",
+      "description": "List of events",
       "items": {
-        "type": "integer"
+        "type": "object"
       }
     }
   }
diff --git a/api/client-server/v2_alpha/definitions/room_event_batch.json b/api/client-server/v2_alpha/definitions/room_event_batch.json
new file mode 100644
index 00000000..fcf148f3
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/room_event_batch.json
@@ -0,0 +1,12 @@
+{
+  "type": "object",
+  "properties": {
+    "events": {
+      "type": "array",
+      "description": "List of event ids",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/definitions/timeline_batch.json b/api/client-server/v2_alpha/definitions/timeline_batch.json
index dc62724b..ddf8d341 100644
--- a/api/client-server/v2_alpha/definitions/timeline_batch.json
+++ b/api/client-server/v2_alpha/definitions/timeline_batch.json
@@ -1,6 +1,6 @@
 {
   "type": "object",
-  "allOf": [{"$ref":"definitions/event_batch.json"}],
+  "allOf": [{"$ref":"definitions/room_event_batch.json"}],
   "properties": {
     "limited": {
       "type": "boolean",
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 0fe9173d..1df7dace 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -71,13 +71,6 @@ paths:
                 description: |-
                   The batch token to supply in the ``since`` param of the next
                   ``/sync`` request.
-              events:
-                type: array
-                description: |-
-                  A list of event objects that are referred to by index in the
-                  rest of the ``sync`` response.
-                items:
-                  type: object
               rooms:
                 type: array
                 description: |-
@@ -89,11 +82,20 @@ paths:
                       type: string
                       description: |-
                         The ID of the room.
+                    event_map:
+                      type: object
+                      description: |-
+                        A map from event ID to events for this room. The events
+                        are referenced from the ``timeline`` and ``state`` keys
+                        for this room.
+                      additionalProperties:
+                        description: An event object.
+                        type: object
                     state:
                       description: |-
                         The state updates for the room.
                       allOf:
-                        - $ref: "definitions/event_batch.json"
+                        - $ref: "definitions/room_event_batch.json"
                     timeline:
                       description: |-
                         The timeline of messages and state changes in the room.
@@ -120,62 +122,76 @@ paths:
             application/json: |-
               {
                 "next_batch": "s72595_4483_1934",
-                "events": [
-                  {
-                    "sender": "@bob:example.com",
-                    "type": "com.example.weird.setting",
-                    "content": {"setting1": true, "setting2": false}
-                  },
-                  {
-                    "sender": "@alice:example.com",
-                    "type": "m.profile.display_name",
-                    "content": {"display_name": "Alice"}
-                  },
-                  {
-                    "sender": "@alice:example.com",
-                    "type": "m.presence",
-                    "content": {"presence": "online"}
-                  },
-                  {
-                    "room_id": "!726s6s6q:example.com",
-                    "type": "m.typing",
-                    "content": {"user_ids": ["@alice:example.com"]}
-                  },
-                  {
-                    "sender": "@alice:example.com",
-                    "room_id": "!726s6s6q:example.com",
-                    "type": "m.room.member",
-                    "state_key": "@alice:example.com",
-                    "content": {"membership": "join"},
-                    "origin_server_ts": 1417731086795
-                  },
-                  {
-                    "sender": "@bob:example.com",
-                    "room_id": "!726s6s6q:example.com",
-                    "type": "m.room.member",
-                    "state_key": "@bob:example.com",
-                    "content": {"membership": "join"},
-                    "origin_server_ts": 1417731086795
-                  },
-                  {
-                    "sender": "@alice:example.com",
-                    "room_id": "!726s6s6q:example.com",
-                    "type": "m.room.message",
-                    "unsigned": {"age": "124524", "txn_id": "1234"},
-                    "content": {"body": "I am a fish", "msgtype": "m.text"},
-                    "origin_server_ts": 1417731086797
-                  }
-                ],
-                "private_user_data": {"events": [0]},
-                "public_user_data": {"events": [1, 2]},
+                "private_user_data": {
+                  "events": [
+                    {
+                      "sender": "@bob:example.com",
+                      "type": "com.example.weird.setting",
+                      "content": {"setting1": true, "setting2": false}
+                    }
+                  ]
+                },
+                "public_user_data": {
+                  "events": [
+                    {
+                      "sender": "@alice:example.com",
+                      "type": "m.profile.display_name",
+                      "content": {"display_name": "Alice"}
+                    },
+                    {
+                      "sender": "@alice:example.com",
+                      "type": "m.presence",
+                      "content": {"presence": "online"}
+                    }
+                  ]
+                },
                 "rooms": [{
                   "room_id": "!726s6s6q:example.com",
-                  "ephemeral": {"events": [3]},
-                  "state": {"events": [4, 5]},
+                  "event_map": {
+                    "$66697273743031:example.com": {
+                      "sender": "@alice:example.com",
+                      "type": "m.room.member",
+                      "state_key": "@alice:example.com",
+                      "content": {"membership": "join"},
+                      "origin_server_ts": 1417731086795
+                    },
+                    "$7365636s6r6432:example.com": {
+                      "sender": "@bob:example.com",
+                      "type": "m.room.member",
+                      "state_key": "@bob:example.com",
+                      "content": {"membership": "join"},
+                      "origin_server_ts": 1417731086795
+                    },
+                    "$74686972643033:example.com": {
+                      "sender": "@alice:example.com",
+                      "type": "m.room.message",
+                      "unsigned": {"age": "124524", "txn_id": "1234"},
+                      "content": {"body": "I am a fish", "msgtype": "m.text"},
+                      "origin_server_ts": 1417731086797
+                    }
+                  },
+                  "state": {
+                    "events": [
+                      "$66697273743031:example.com",
+                      "$7365636s6r6432:example.com"
+                    ]
+                  },
                   "timeline": {
-                    "events": [5,6],
+                    "events": [
+                      "$7365636s6r6432:example.com",
+                      "$74686972643033:example.com"
+                    ],
                     "limited": true,
                     "prev_batch": "t34-23535_0_0"
+                  },
+                  "ephemeral": {
+                    "events": [
+                      {
+                        "room_id": "!726s6s6q:example.com",
+                        "type": "m.typing",
+                        "content": {"user_ids": ["@alice:example.com"]}
+                      }
+                    ]
                   }
                 }]
               }

From 4cb3f78d2bc8205c87e566e16a049732c2f3bad5 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 29 Sep 2015 19:48:48 +0100
Subject: [PATCH 204/467] Wrap the "rooms" list inside an object so that we can
 add keys for pagination later

---
 api/client-server/v2_alpha/sync.yaml | 165 ++++++++++++++-------------
 1 file changed, 85 insertions(+), 80 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 1df7dace..58a2c495 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -72,41 +72,44 @@ paths:
                   The batch token to supply in the ``since`` param of the next
                   ``/sync`` request.
               rooms:
-                type: array
-                description: |-
-                  A list of rooms that the client needs to update.
-                items:
-                  type: object
-                  properties:
-                    room_id:
-                      type: string
-                      description: |-
-                        The ID of the room.
-                    event_map:
+                type: object
+                properties:
+                  roomlist:
+                    type: array
+                    description: |-
+                      A list of rooms that the client needs to update.
+                    items:
                       type: object
-                      description: |-
-                        A map from event ID to events for this room. The events
-                        are referenced from the ``timeline`` and ``state`` keys
-                        for this room.
-                      additionalProperties:
-                        description: An event object.
-                        type: object
-                    state:
-                      description: |-
-                        The state updates for the room.
-                      allOf:
-                        - $ref: "definitions/room_event_batch.json"
-                    timeline:
-                      description: |-
-                        The timeline of messages and state changes in the room.
-                      allOf:
-                        - $ref: "definitions/timeline_batch.json"
-                    ephemeral:
-                      description: |-
-                        The ephemeral events in the room that aren't recorded
-                        in the timeline or state of the room. E.g. typing.
-                      allOf:
-                        - $ref: "definitions/event_batch.json"
+                      properties:
+                        room_id:
+                          type: string
+                          description: |-
+                            The ID of the room.
+                        event_map:
+                          type: object
+                          description: |-
+                            A map from event ID to events for this room. The events
+                            are referenced from the ``timeline`` and ``state`` keys
+                            for this room.
+                          additionalProperties:
+                            description: An event object.
+                            type: object
+                        state:
+                          description: |-
+                            The state updates for the room.
+                          allOf:
+                            - $ref: "definitions/room_event_batch.json"
+                        timeline:
+                          description: |-
+                            The timeline of messages and state changes in the room.
+                          allOf:
+                            - $ref: "definitions/timeline_batch.json"
+                        ephemeral:
+                          description: |-
+                            The ephemeral events in the room that aren't recorded
+                            in the timeline or state of the room. E.g. typing.
+                          allOf:
+                            - $ref: "definitions/event_batch.json"
               public_user_data:
                 description: |-
                   The updates to publicly visible user data.
@@ -145,53 +148,55 @@ paths:
                     }
                   ]
                 },
-                "rooms": [{
-                  "room_id": "!726s6s6q:example.com",
-                  "event_map": {
-                    "$66697273743031:example.com": {
-                      "sender": "@alice:example.com",
-                      "type": "m.room.member",
-                      "state_key": "@alice:example.com",
-                      "content": {"membership": "join"},
-                      "origin_server_ts": 1417731086795
+                "rooms": {
+                  "roomlist": [{
+                    "room_id": "!726s6s6q:example.com",
+                    "event_map": {
+                      "$66697273743031:example.com": {
+                        "sender": "@alice:example.com",
+                        "type": "m.room.member",
+                        "state_key": "@alice:example.com",
+                        "content": {"membership": "join"},
+                        "origin_server_ts": 1417731086795
+                      },
+                      "$7365636s6r6432:example.com": {
+                        "sender": "@bob:example.com",
+                        "type": "m.room.member",
+                        "state_key": "@bob:example.com",
+                        "content": {"membership": "join"},
+                        "origin_server_ts": 1417731086795
+                      },
+                      "$74686972643033:example.com": {
+                        "sender": "@alice:example.com",
+                        "type": "m.room.message",
+                        "unsigned": {"age": "124524", "txn_id": "1234"},
+                        "content": {"body": "I am a fish", "msgtype": "m.text"},
+                        "origin_server_ts": 1417731086797
+                      }
                     },
-                    "$7365636s6r6432:example.com": {
-                      "sender": "@bob:example.com",
-                      "type": "m.room.member",
-                      "state_key": "@bob:example.com",
-                      "content": {"membership": "join"},
-                      "origin_server_ts": 1417731086795
+                    "state": {
+                      "events": [
+                        "$66697273743031:example.com",
+                        "$7365636s6r6432:example.com"
+                      ]
                     },
-                    "$74686972643033:example.com": {
-                      "sender": "@alice:example.com",
-                      "type": "m.room.message",
-                      "unsigned": {"age": "124524", "txn_id": "1234"},
-                      "content": {"body": "I am a fish", "msgtype": "m.text"},
-                      "origin_server_ts": 1417731086797
+                    "timeline": {
+                      "events": [
+                        "$7365636s6r6432:example.com",
+                        "$74686972643033:example.com"
+                      ],
+                      "limited": true,
+                      "prev_batch": "t34-23535_0_0"
+                    },
+                    "ephemeral": {
+                      "events": [
+                        {
+                          "room_id": "!726s6s6q:example.com",
+                          "type": "m.typing",
+                          "content": {"user_ids": ["@alice:example.com"]}
+                        }
+                      ]
                     }
-                  },
-                  "state": {
-                    "events": [
-                      "$66697273743031:example.com",
-                      "$7365636s6r6432:example.com"
-                    ]
-                  },
-                  "timeline": {
-                    "events": [
-                      "$7365636s6r6432:example.com",
-                      "$74686972643033:example.com"
-                    ],
-                    "limited": true,
-                    "prev_batch": "t34-23535_0_0"
-                  },
-                  "ephemeral": {
-                    "events": [
-                      {
-                        "room_id": "!726s6s6q:example.com",
-                        "type": "m.typing",
-                        "content": {"user_ids": ["@alice:example.com"]}
-                      }
-                    ]
-                  }
-                }]
+                  }]
+                }
               }

From 73b4090f52f324c598db699899b22eed72503a3d Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 30 Sep 2015 10:22:12 +0100
Subject: [PATCH 205/467] Add private_chat_shared_power

---
 specification/1-client_server_api.rst | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index c5453931..329a87dd 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -756,15 +756,18 @@ options which can be set when creating a room:
   Optional:
     Yes
   Value:
-    ``private_chat`` or ``public_chat``
+    ``private_chat``, ``private_chat_shared_power`` or ``public_chat``
   Description:
     Convenience parameter for setting various default state events based on a
     preset.
 
-    Two presets are defined:
+    Three presets are defined:
 
     - ``private_chat``: Sets the ``join_rules`` to ``invite`` and
       ``history_visibility`` to ``shared``
+    - ``private_chat_shared_power``: Set the ``join_rules`` to
+      ``invite``, ``history_visibility`` to ``shared`` and gives all invitees
+      the same power level as the creator.
     - ``public_chat``: Sets the ``join_rules`` to ``public`` and
       ``history_visibility`` to ``shared``
 

From 069e4e39f4d326ef9e4188159767a1194db1ae76 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 10:22:51 +0100
Subject: [PATCH 206/467] Move presence specific sections from intro to
 presence module

---
 specification/0-intro.rst          | 43 ------------------------------
 specification/modules/presence.rst | 35 ++++++++++++++++++++----
 2 files changed, 30 insertions(+), 48 deletions(-)

diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index a5196e99..1b4e8672 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -338,49 +338,6 @@ Usage of an IS is not required in order for a client application to be part of
 the Matrix ecosystem. However, without one clients will not be able to look up
 user IDs using 3PIDs.
 
-Presence
-~~~~~~~~
-
-Each user has the concept of presence information. This encodes:
-
- * Whether the user is currently online
- * How recently the user was last active (as seen by the server)
- * Whether a given client considers the user to be currently idle
- * Arbitrary information about the user's current status (e.g. "in a meeting").
-
-This information is collated from both per-device (online; idle; last_active) and
-per-user (status) data, aggregated by the user's homeserver and transmitted as
-an ``m.presence`` event. This is one of the few events which are sent *outside
-the context of a room*. Presence events are sent to all users who subscribe to
-this user's presence through a presence list or by sharing membership of a room.
-
-.. TODO
-  How do we let users hide their presence information?
-
-.. TODO
-  The last_active specifics should be moved to the detailed presence event section
-  
-Last activity is tracked by the server maintaining a timestamp of the last time
-it saw a pro-active event from the user. Any event which could be triggered by a
-human using the application is considered pro-active (e.g. sending an event to a
-room). An example of a non-proactive client activity would be a client setting
-'idle' presence status, or polling for events. This timestamp is presented via a
-key called ``last_active_ago``, which gives the relative number of milliseconds
-since the message is generated/emitted that the user was last seen active.
-
-N.B. in v1 API, status/online/idle state are muxed into a single 'presence'
-field on the ``m.presence`` event.
-
-Presence Lists
-~~~~~~~~~~~~~~
-
-Each user's home server stores a "presence list". This stores a list of user IDs
-whose presence the user wants to follow.
-
-To be added to this list, the user being added must be invited by the list owner
-and accept the invitation. Once accepted, both user's HSes track the
-subscription.
-
 
 Profiles
 ~~~~~~~~
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index 4c84359a..1a359e5a 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -1,5 +1,23 @@
 Presence
 ========
+
+Each user has the concept of presence information. This encodes:
+
+ * Whether the user is currently online
+ * How recently the user was last active (as seen by the server)
+ * Whether a given client considers the user to be currently idle
+ * Arbitrary information about the user's current status (e.g. "in a meeting").
+
+This information is collated from both per-device (``online``, ``idle``,
+``last_active``) and per-user (status) data, aggregated by the user's homeserver
+and transmitted as an ``m.presence`` event. This is one of the few events which
+are sent *outside the context of a room*. Presence events are sent to all users
+who subscribe to this user's presence through a presence list or by sharing
+membership of a room.
+
+A presence list is a list of user IDs whose presence the user wants to follow.
+To be added to this list, the user being added must be invited by the list owner
+who must accept the invitation.
  
 Each user has presence information associated with them. This encodes the
 "availability" of that user, suitable for display on other clients.
@@ -15,6 +33,7 @@ represented by the ``presence`` key, which is an enum of one of the following:
         explicitly suppressing their profile information from being sent.
       - ``free_for_chat`` : The user is generally willing to receive messages
         moreso than default.
+ 
 
 Events
 ------
@@ -24,7 +43,8 @@ Events
 Client behaviour
 ----------------
 
-Clients can manually set/get their presence using the HTTP APIs listed below.
+Clients can manually set/get their presence/presence list using the HTTP APIs
+listed below.
 
 {{presence_http_api}}
 
@@ -43,6 +63,9 @@ recommended.
 Server behaviour
 ----------------
 
+Each user's home server stores a "presence list" per user. Once a user accepts
+a presence list, both user's HSes must track the subscription.
+
 Propagating profile information
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -58,10 +81,12 @@ home server when a user successfully changes their display name or avatar URL.
 
   The intention for sending this information in ``m.presence`` is so that any
   "user list" can display the *current* name/presence for a user ID outside the
-  scope of a room (e.g. a user page which has a "start conversation" button).
-  This is bundled into a single event to avoid "flickering" on this page which
-  can occur if you received presence first and then display name later (the
-  user's name would flicker from their user ID to the display name).
+  scope of a room e.g. for a user page. This is bundled into a single event for
+  several reasons. The user's display name can change per room. This
+  event provides the "canonical" name for the user. In addition, the name is
+  bundled into a single event for the ease of client implementations. If this
+  was not done, the client would need to search all rooms for their own
+  membership event to pull out the display name.
 
 
 Last active ago

From 931057accfdd2756686c3c896a22157b3f5faf51 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Wed, 30 Sep 2015 10:30:39 +0100
Subject: [PATCH 207/467] Add a top level presence key for the presence events
 and remove the public_user_data/private_user_data for now

---
 .../v2_alpha/definitions/sync_filter.json     |  9 ++-----
 api/client-server/v2_alpha/filter.yaml        | 22 +++++-----------
 api/client-server/v2_alpha/sync.yaml          | 26 +++----------------
 3 files changed, 11 insertions(+), 46 deletions(-)

diff --git a/api/client-server/v2_alpha/definitions/sync_filter.json b/api/client-server/v2_alpha/definitions/sync_filter.json
index b909830b..0cd6a798 100644
--- a/api/client-server/v2_alpha/definitions/sync_filter.json
+++ b/api/client-server/v2_alpha/definitions/sync_filter.json
@@ -21,14 +21,9 @@
         }
       }
     },
-    "public_user_data": {
+    "presence": {
       "description":
-        "The public user data, e.g. profile and presence, to include.",
-      "allOf": [{"$ref": "definitions/event_filter.json"}]
-    },
-    "private_user_data": {
-      "description":
-        "Events that are private to a user but shared amoungst their devices, e.g. notification settings, to include.",
+        "The presence updates to include.",
       "allOf": [{"$ref": "definitions/event_filter.json"}]
     },
     "event_format": {
diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 9bda2359..30d52844 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -62,14 +62,9 @@ paths:
                     "not_senders": ["@spam:example.com"]
                   }
                 },
-                "public_user_data": {
-                  "types": ["m.presence"]
-                },
-                "private_user_data": {
-                  "types": []
-                },
-                "server_data": {
-                  "types": []
+                "presence": {
+                  "types": ["m.presence"],
+                  "not_senders": ["@alice:example.com"]
                 },
                 "event_format": "client",
                 "event_fields": ["type", "content", "sender"]
@@ -131,14 +126,9 @@ paths:
                     "not_senders": ["@spam:example.com"]
                   }
                 },
-                "public_user_data": {
-                  "types": ["m.presence"]
-                },
-                "private_user_data": {
-                  "types": []
-                },
-                "server_data": {
-                  "types": []
+                "presence": {
+                  "types": ["m.presence"],
+                  "not_senders": ["@alice:example.com"]
                 },
                 "event_format": "client",
                 "event_fields": ["type", "content", "sender"]
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 58a2c495..238fab63 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -110,37 +110,17 @@ paths:
                             in the timeline or state of the room. E.g. typing.
                           allOf:
                             - $ref: "definitions/event_batch.json"
-              public_user_data:
+              presence:
                 description: |-
-                  The updates to publicly visible user data.
-                allOf:
-                  - $ref: "definitions/event_batch.json"
-              private_user_data:
-                description: |-
-                  Updates to the data which is private to the user but shared
-                  amongst their devices.
+                  The updates to the presence status of other users.
                 allOf:
                   - $ref: "definitions/event_batch.json"
           examples:
             application/json: |-
               {
                 "next_batch": "s72595_4483_1934",
-                "private_user_data": {
+                "presence": {
                   "events": [
-                    {
-                      "sender": "@bob:example.com",
-                      "type": "com.example.weird.setting",
-                      "content": {"setting1": true, "setting2": false}
-                    }
-                  ]
-                },
-                "public_user_data": {
-                  "events": [
-                    {
-                      "sender": "@alice:example.com",
-                      "type": "m.profile.display_name",
-                      "content": {"display_name": "Alice"}
-                    },
                     {
                       "sender": "@alice:example.com",
                       "type": "m.presence",

From 4e1e82f995f899d397ac5a21194a48973e262948 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 10:43:02 +0100
Subject: [PATCH 208/467] Fix paragraph CSS and adjust where paragraphs are in
 the RST

The CSS for `nature.css` was such that it was preventing `p` tags from
having sufficient vertical whitespace. This meant that you couldn't insert
any kind of spacing between lengthy sections (they just appeared as new lines).

This PR fixes this so you can actually have some whitespace between paragraphs.
As a result of this change, some parts of the spec appeared to have too much
whitespace. These were often sections which shouldn't have begun a new
paragraph anyway (e.g. a single sentence being an entire paragraph, `TODO`
blocks resulting in new paragraphs). This PR fixes the most offending areas
where we shouldn't have been inserting new paragraphs.
---
 scripts/nature.css                          |  2 +-
 specification/0-intro.rst                   | 69 +++++++++------------
 specification/1-client_server_api.rst       | 34 ++++------
 specification/3-application_service_api.rst | 18 +++---
 specification/4-server_server_api.rst       | 12 ++--
 specification/modules/content_repo.rst      | 24 +++----
 specification/modules/push_cs_api.rst       | 17 +++--
 specification/modules/voip_events.rst       |  9 ++-
 8 files changed, 77 insertions(+), 108 deletions(-)

diff --git a/scripts/nature.css b/scripts/nature.css
index 9d67f689..ac171923 100644
--- a/scripts/nature.css
+++ b/scripts/nature.css
@@ -245,7 +245,7 @@ div.viewcode-block:target {
 }
 
 p {
-	margin: 0;
+	/* margin: 0;  - setting this leads to no spacing between paragraphs which looks ugly */
 }
 
 ul li dd {
diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index a5196e99..93458d97 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -24,7 +24,6 @@ Introduction
   The Matrix specification is still evolving: the APIs are not yet frozen
   and this document is in places a work in progress or stale. We have made every 
   effort to clearly flag areas which are still being finalised.
-
   We're publishing it at this point because it's complete enough to be more than
   useful and provide a canonical reference to how Matrix is evolving. Our end
   goal is to mirror WHATWG's `Living Standard   
@@ -34,10 +33,9 @@ Matrix is a set of open APIs for open-federated Instant Messaging (IM), Voice
 over IP (VoIP) and Internet of Things (IoT) communication, designed to create
 and support a new global real-time communication ecosystem. The intention is to
 provide an open decentralised pubsub layer for the internet for securely
-persisting and publishing/subscribing JSON objects.
-
-This specification is the ongoing result of standardising the APIs used by the
-various components of the Matrix ecosystem to communicate with one another.
+persisting and publishing/subscribing JSON objects. This specification is the
+ongoing result of standardising the APIs used by the various components of the
+Matrix ecosystem to communicate with one another.
 
 The principles that Matrix attempts to follow are:
 
@@ -214,10 +212,8 @@ which have the form::
 There is exactly one room ID for each room. Whilst the room ID does contain a
 domain, it is simply for globally namespacing room IDs. The room does NOT
 reside on the domain specified. Room IDs are not meant to be human readable.
-They are case-sensitive.
-
-The following conceptual diagram shows an ``m.room.message`` event being sent to 
-the room ``!qporfwt:matrix.org``::
+They are case-sensitive. The following conceptual diagram shows an
+``m.room.message`` event being sent to the room ``!qporfwt:matrix.org``::
 
        { @alice:matrix.org }                             { @bob:domain.com }
                |                                                 ^
@@ -258,28 +254,28 @@ the room ``!qporfwt:matrix.org``::
 Federation maintains *shared data structures* per-room between multiple home
 servers. The data is split into ``message events`` and ``state events``.
 
-``Message events`` describe transient 'once-off' activity in a room such as an
-instant messages, VoIP call setups, file transfers, etc. They generally describe
-communication activity.
+Message events: 
+  These describe transient 'once-off' activity in a room such as an
+  instant messages, VoIP call setups, file transfers, etc. They generally
+  describe communication activity.
 
-``State events`` describe updates to a given piece of persistent information
-('state') related to a room, such as the room's name, topic, membership,
-participating servers, etc. State is modelled as a lookup table of key/value
-pairs per room, with each key being a tuple of ``state_key`` and ``event type``.
-Each state event updates the value of a given key.
+State events:
+  These describe updates to a given piece of persistent information
+  ('state') related to a room, such as the room's name, topic, membership,
+  participating servers, etc. State is modelled as a lookup table of key/value
+  pairs per room, with each key being a tuple of ``state_key`` and ``event type``.
+  Each state event updates the value of a given key.
 
 The state of the room at a given point is calculated by considering all events
 preceding and including a given event in the graph. Where events describe the
 same state, a merge conflict algorithm is applied. The state resolution
 algorithm is transitive and does not depend on server state, as it must
 consistently select the same event irrespective of the server or the order the
-events were received in.
-
-Events are signed by the originating server (the signature includes the parent
-relations, type, depth and payload hash) and are pushed over federation to the
-participating servers in a room, currently using full mesh topology. Servers may
-also request backfill of events over federation from the other servers
-participating in a room.
+events were received in. Events are signed by the originating server (the
+signature includes the parent relations, type, depth and payload hash) and are
+pushed over federation to the participating servers in a room, currently using
+full mesh topology. Servers may also request backfill of events over federation
+from the other servers participating in a room.
 
 
 Room Aliases
@@ -324,12 +320,10 @@ Users in Matrix are identified via their matrix user ID (MXID). However,
 existing 3rd party ID namespaces can also be used in order to identify Matrix
 users. A Matrix "Identity" describes both the user ID and any other existing IDs
 from third party namespaces *linked* to their account.
-
 Matrix users can *link* third-party IDs (3PIDs) such as email addresses, social
 network accounts and phone numbers to their user ID. Linking 3PIDs creates a
 mapping from a 3PID to a user ID. This mapping can then be used by Matrix
 users in order to discover the MXIDs of their contacts.
-
 In order to ensure that the mapping from 3PID to user ID is genuine, a globally
 federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID
 and persist and replicate the mappings.
@@ -410,6 +404,10 @@ dedicated API.  The API is symmetrical to managing Profile data.
 API Standards
 -------------
 
+.. TODO
+  Need to specify any HMAC or access_token lifetime/ratcheting tricks
+  We need to specify capability negotiation for extensible transports
+
 The mandatory baseline for communication in Matrix is exchanging JSON objects
 over HTTP APIs. HTTPS is mandated as the baseline for server-server
 (federation) communication.  HTTPS is recommended for client-server
@@ -417,20 +415,11 @@ communication, although HTTP may be supported as a fallback to support basic
 HTTP clients. More efficient optional transports for client-server
 communication will in future be supported as optional extensions - e.g. a
 packed binary encoding over stream-cipher encrypted TCP socket for
-low-bandwidth/low-roundtrip mobile usage.
-
-.. TODO
-  We need to specify capability negotiation for extensible transports
-
-For the default HTTP transport, all API calls use a Content-Type of
-``application/json``.  In addition, all strings MUST be encoded as UTF-8.
-
-Clients are authenticated using opaque ``access_token`` strings (see
-`Client Authentication`_ for details), passed as a query string parameter on
-all requests.
-
-.. TODO
-  Need to specify any HMAC or access_token lifetime/ratcheting tricks
+low-bandwidth/low-roundtrip mobile usage. For the default HTTP transport, all
+API calls use a Content-Type of ``application/json``.  In addition, all strings
+MUST be encoded as UTF-8. Clients are authenticated using opaque
+``access_token`` strings (see `Client Authentication`_ for details), passed as a
+query string parameter on all requests.
 
 Any errors which occur at the Matrix API level MUST return a "standard error
 response". This is a JSON object which looks like::
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 893aec73..7d8d57e4 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -672,13 +672,10 @@ to add keys that are, for example offensive or illegal. Since some events
 cannot be simply deleted, e.g. membership events, we instead 'redact' events.
 This involves removing all keys from an event that are not required by the
 protocol. This stripped down event is thereafter returned anytime a client or
-remote server requests it.
-
-Events that have been redacted include a ``redacted_because`` key whose value
-is the event that caused it to be redacted, which may include a reason.
-
-Redacting an event cannot be undone, allowing server owners to delete the
-offending content from the databases.
+remote server requests it. Redacting an event cannot be undone, allowing server
+owners to delete the offending content from the databases. Events that have been
+redacted include a ``redacted_because`` key whose value is the event that caused
+it to be redacted, which may include a reason.
 
 .. TODO
   Currently, only room admins can redact events by sending a ``m.room.redaction``
@@ -709,12 +706,10 @@ one of the following event types:
 .. TODO
   Need to update m.room.power_levels to reflect new power levels formatting
 
-The redaction event should be added under the key ``redacted_because``.
-
-When a client receives a redaction event it should change the redacted event
+The redaction event should be added under the key ``redacted_because``. When a
+client receives a redaction event it should change the redacted event
 in the same way a server does.
 
-
 Rooms
 -----
 
@@ -853,18 +848,14 @@ Permissions
 
 Permissions for rooms are done via the concept of power levels - to do any
 action in a room a user must have a suitable power level. Power levels are
-stored as state events in a given room.
-
-The power levels required for operations and the power levels for users are
-defined in ``m.room.power_levels``, where both a default and specific users'
-power levels can be set.
-
+stored as state events in a given room. The power levels required for operations
+and the power levels for users are defined in ``m.room.power_levels``, where
+both a default and specific users' power levels can be set.
 By default all users have a power level of 0, other than the room creator whose
 power level defaults to 100. Users can grant other users increased power levels
 up to their own power level. For example, user A with a power level of 50 could
 increase the power level of user B to a maximum of level 50. Power levels for
 users are tracked per-room even if the user is not present in the room.
-
 The keys contained in ``m.room.power_levels`` determine the levels required for
 certain operations such as kicking, banning and sending state events. See
 `m.room.power_levels`_ for more information.
@@ -881,10 +872,9 @@ room. There are several states in which a user may be, in relation to a room:
  - Banned (the user is not allowed to join the room)
 
 Some rooms require that users be invited to it before they can join; others
-allow anyone to join.
-
-Whether a given room is an "invite-only" room is determined by the room config
-key ``m.room.join_rules``. It can have one of the following values:
+allow anyone to join. Whether a given room is an "invite-only" room is
+determined by the room config key ``m.room.join_rules``. It can have one of the
+following values:
 
 ``public``
   This room is free for anyone to join without an invite.
diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index e982390b..bdde7789 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -4,11 +4,9 @@ Application Service API
 The Matrix client-server API and server-server APIs provide the means to
 implement a consistent self-contained federated messaging fabric. However, they
 provide limited means of implementing custom server-side behaviour in Matrix
-(e.g. gateways, filters, extensible hooks etc).
-
-The Application Service API defines a standard API to allow such extensible 
-functionality to be implemented irrespective of the underlying homeserver
-implementation.
+(e.g. gateways, filters, extensible hooks etc). The Application Service API
+defines a standard API to allow such extensible functionality to be implemented
+irrespective of the underlying homeserver implementation.
 
 .. TODO-spec
   Add in Client-Server services? Overview of bots? Seems weird to be in the spec
@@ -18,12 +16,10 @@ Passive Application Services
 ----------------------------
 "Passive" application services can only observe events from a given home server.
 They cannot prevent events from being sent, nor can they modify the content of
-the event being sent.
-
-In order to observe events from a homeserver, the homeserver needs to be
-configured to pass certain types of traffic to the application service.  This
-is achieved by manually configuring the homeserver with information about the
-AS..
+the event being sent. In order to observe events from a homeserver, the
+homeserver needs to be configured to pass certain types of traffic to the
+application service.  This is achieved by manually configuring the homeserver
+with information about the AS.
 
 .. NOTE::
   Previously, application services could register with a homeserver via HTTP
diff --git a/specification/4-server_server_api.rst b/specification/4-server_server_api.rst
index 8d1f8898..c5ff2b87 100644
--- a/specification/4-server_server_api.rst
+++ b/specification/4-server_server_api.rst
@@ -59,13 +59,11 @@ and an optional TLS port.
 .. **
 
 If the port is present then the server is discovered by looking up an AAAA or
-A record for the DNS name and connecting to the specified TLS port.
-
-If the port is absent then the server is discovered by looking up a
-``_matrix._tcp`` SRV record for the DNS name. If this record does not exist
-then the server is discovered by looking up an AAAA or A record on the DNS
-name and taking the default fallback port number of 8448.
-
+A record for the DNS name and connecting to the specified TLS port. If the port
+is absent then the server is discovered by looking up a ``_matrix._tcp`` SRV
+record for the DNS name. If this record does not exist then the server is
+discovered by looking up an AAAA or A record on the DNS name and taking the
+default fallback port number of 8448.
 Home servers may use SRV records to load balance requests between multiple TLS
 endpoints or to failover to another endpoint if an endpoint fails.
 
diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index 2c45ced7..5a3cf9b9 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -56,21 +56,21 @@ Homeservers must never upscale images.
 Security considerations
 -----------------------
 
-Clients may try to upload very large files. Homeservers should not store files
-that are too large and should not serve them to clients.
+ - Clients may try to upload very large files. Homeservers should not store files
+   that are too large and should not serve them to clients.
 
-Clients may try to upload very large images. Homeservers should not attempt to
-generate thumbnails for images that are too large.
+ - Clients may try to upload very large images. Homeservers should not attempt to
+   generate thumbnails for images that are too large.
 
-Remote homeservers may host very large files or images. Homeserver should not
-proxy or thumbnail large files or images from remote homeservers.
+ - Remote homeservers may host very large files or images. Homeserver should not
+   proxy or thumbnail large files or images from remote homeservers.
 
-Clients may try to upload a large number of files. Homeservers should limit the
-number and total size of media that can be uploaded by clients.
+ - Clients may try to upload a large number of files. Homeservers should limit the
+   number and total size of media that can be uploaded by clients.
 
-Clients may try to access a large number of remote files through a homeserver.
-Homeservers should restrict the number and size of remote files that it caches.
+ - Clients may try to access a large number of remote files through a homeserver.
+   Homeservers should restrict the number and size of remote files that it caches.
 
-Clients or remote homeservers may try to upload malicious files targeting
-vulnerabilities in either the homeserver thumbnailing or the client decoders.
+ - Clients or remote homeservers may try to upload malicious files targeting
+   vulnerabilities in either the homeserver thumbnailing or the client decoders.
 
diff --git a/specification/modules/push_cs_api.rst b/specification/modules/push_cs_api.rst
index b78b44fd..c6301926 100644
--- a/specification/modules/push_cs_api.rst
+++ b/specification/modules/push_cs_api.rst
@@ -99,16 +99,13 @@ be redundant. Actions for the highest priority rule and only that rule apply
 (for example, a set_tweak action in a lower priority rule will not apply if a
 higher priority rule matches, even if that rule does not specify any tweaks).
 
-Rules also have an identifier, rule_id, which is a string. The rule_id is
-unique within the kind of rule and scope: rule_ids need not be unique between
-rules of the same kind on different devices.
-
-A home server may also have server default rules of each kind and in each scope.
-Server default rules are lower priority than user-defined rules in each scope.
-Server default rules (and only server default rules) begin with a dot ('.')
-character.
-
-In addition, all rules may be enabled or disabled. Disabled rules never match.
+Rules also have an identifier, ``rule_id``, which is a string. The ``rule_id``
+is unique within the kind of rule and scope: ``rule_ids`` need not be unique
+between rules of the same kind on different devices. A home server may also have
+server default rules of each kind and in each scope. Server default rules are
+lower priority than user-defined rules in each scope. Server default rules (and
+only server default rules) begin with a dot ('.') character. In addition, all
+rules may be enabled or disabled. Disabled rules never match.
 
 If no rules match an event, the Home Server should not notify for the message
 (that is to say, the default action is "dont-notify"). Events that the user sent
diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst
index a5468237..33998cd9 100644
--- a/specification/modules/voip_events.rst
+++ b/specification/modules/voip_events.rst
@@ -44,11 +44,10 @@ Glare
 This specification aims to address the problem of two users calling each other
 at roughly the same time and their invites crossing on the wire. It is a far
 better experience for the users if their calls are connected if it is clear
-that their intention is to set up a call with one another.
-
-In Matrix, calls are to rooms rather than users (even if those rooms may only
-contain one other user) so we consider calls which are to the same room. The
-rules for dealing with such a situation are as follows:
+that their intention is to set up a call with one another. In Matrix, calls are
+to rooms rather than users (even if those rooms may only contain one other user)
+so we consider calls which are to the same room. The rules for dealing with such
+a situation are as follows:
 
  - If an invite to a room is received whilst the client is preparing to send an
    invite to the same room, the client should cancel its outgoing call and

From a49d5f67f1ee8e06c2c7a3e0baaba5f120f7a4e3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 10:54:06 +0100
Subject: [PATCH 209/467] Delete rather than comment out the offending CSS

---
 scripts/nature.css | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/scripts/nature.css b/scripts/nature.css
index ac171923..7c4af305 100644
--- a/scripts/nature.css
+++ b/scripts/nature.css
@@ -244,10 +244,6 @@ div.viewcode-block:target {
     border-bottom: 1px solid #ac9;
 }
 
-p {
-	/* margin: 0;  - setting this leads to no spacing between paragraphs which looks ugly */
-}
-
 ul li dd {
 	margin-top: 0;
 }

From ede43fbe9fc59b0f4eff83d7e1e8cc0d69c180a6 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 14:31:26 +0100
Subject: [PATCH 210/467] Modify how speculator uses git repos

Modified so it has a "master" repo and local A/B repos which pull from the
"master". This saves an extra git clone operation per HTTP request.
---
 scripts/speculator/main.go | 72 +++++++++++++++++++++++++++-----------
 1 file changed, 51 insertions(+), 21 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 2609fa7e..6ee80150 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -24,6 +24,7 @@ import (
 	"strconv"
 	"strings"
 	"syscall"
+	"time"
 )
 
 type PullRequest struct {
@@ -58,16 +59,24 @@ func (u *User) IsTrusted() bool {
 	return allowedMembers[u.Login]
 }
 
-const pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls"
+const (
+	pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls"
+	matrixDocCloneURL = "https://github.com/matrix-org/matrix-doc.git"
+)
+
+
+func gitClone(url string, shared bool) (string, error) {
+	directory := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10))
+	cmd := exec.Command("git", "clone", url, directory)
+	if shared {
+		cmd.Args = append(cmd.Args, "--shared")
+	}
 
-func gitClone(url string) (string, error) {
-	dst := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10))
-	cmd := exec.Command("git", "clone", url, dst)
 	err := cmd.Run()
 	if err != nil {
 		return "", fmt.Errorf("error cloning repo: %v", err)
 	}
-	return dst, nil
+	return directory, nil
 }
 
 func gitCheckout(path, sha string) error {
@@ -80,6 +89,16 @@ func gitCheckout(path, sha string) error {
 	return nil
 }
 
+func gitFetch(path string) error {
+	cmd := exec.Command("git", "fetch")
+	cmd.Dir = path
+	err := cmd.Run()
+	if err != nil {
+		return fmt.Errorf("error fetching repo: %v", err)
+	}
+	return nil
+}
+
 func lookupPullRequest(url url.URL, pathPrefix string) (*PullRequest, error) {
 	if !strings.HasPrefix(url.Path, pathPrefix+"/") {
 		return nil, fmt.Errorf("invalid path passed: %s expect %s/123", url.Path, pathPrefix)
@@ -119,10 +138,18 @@ func writeError(w http.ResponseWriter, code int, err error) {
 	io.WriteString(w, fmt.Sprintf("%v\n", err))
 }
 
+type server struct {
+	matrixDocCloneURL string
+}
+
 // generateAt generates spec from repo at sha.
 // Returns the path where the generation was done.
-func generateAt(repo, sha string) (dst string, err error) {
-	dst, err = gitClone(repo)
+func (s *server) generateAt(sha string) (dst string, err error) {
+	err = gitFetch(s.matrixDocCloneURL)
+	if err != nil {
+		return
+	}
+	dst, err = gitClone(s.matrixDocCloneURL, true)
 	if err != nil {
 		return
 	}
@@ -135,12 +162,10 @@ func generateAt(repo, sha string) (dst string, err error) {
 	return
 }
 
-func serveSpec(w http.ResponseWriter, req *http.Request) {
-	var cloneURL string
+func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 	var sha string
 
 	if strings.ToLower(req.URL.Path) == "/spec/head" {
-		cloneURL = "https://github.com/matrix-org/matrix-doc.git"
 		sha = "HEAD"
 	} else {
 		pr, err := lookupPullRequest(*req.URL, "/spec")
@@ -155,11 +180,10 @@ func serveSpec(w http.ResponseWriter, req *http.Request) {
 			writeError(w, 403, err)
 			return
 		}
-		cloneURL = pr.Head.Repo.CloneURL
 		sha = pr.Head.SHA
 	}
 
-	dst, err := generateAt(cloneURL, sha)
+	dst, err := s.generateAt(sha)
 	defer os.RemoveAll(dst)
 	if err != nil {
 		writeError(w, 500, err)
@@ -181,7 +205,7 @@ func checkAuth(pr *PullRequest) error {
 	return nil
 }
 
-func serveRSTDiff(w http.ResponseWriter, req *http.Request) {
+func (s *server) serveRSTDiff(w http.ResponseWriter, req *http.Request) {
 	pr, err := lookupPullRequest(*req.URL, "/diff/rst")
 	if err != nil {
 		writeError(w, 400, err)
@@ -195,14 +219,14 @@ func serveRSTDiff(w http.ResponseWriter, req *http.Request) {
 		return
 	}
 
-	base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA)
+	base, err := s.generateAt(pr.Base.SHA)
 	defer os.RemoveAll(base)
 	if err != nil {
 		writeError(w, 500, err)
 		return
 	}
 
-	head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
+	head, err := s.generateAt(pr.Head.SHA)
 	defer os.RemoveAll(head)
 	if err != nil {
 		writeError(w, 500, err)
@@ -219,7 +243,7 @@ func serveRSTDiff(w http.ResponseWriter, req *http.Request) {
 	w.Write(diff.Bytes())
 }
 
-func serveHTMLDiff(w http.ResponseWriter, req *http.Request) {
+func (s *server) serveHTMLDiff(w http.ResponseWriter, req *http.Request) {
 	pr, err := lookupPullRequest(*req.URL, "/diff/html")
 	if err != nil {
 		writeError(w, 400, err)
@@ -233,14 +257,14 @@ func serveHTMLDiff(w http.ResponseWriter, req *http.Request) {
 		return
 	}
 
-	base, err := generateAt(pr.Base.Repo.CloneURL, pr.Base.SHA)
+	base, err := s.generateAt(pr.Base.SHA)
 	defer os.RemoveAll(base)
 	if err != nil {
 		writeError(w, 500, err)
 		return
 	}
 
-	head, err := generateAt(pr.Head.Repo.CloneURL, pr.Head.SHA)
+	head, err := s.generateAt(pr.Head.SHA)
 	defer os.RemoveAll(head)
 	if err != nil {
 		writeError(w, 500, err)
@@ -327,9 +351,15 @@ func main() {
 		"Kegsay":        true,
 		"NegativeMjark": true,
 	}
-	http.HandleFunc("/spec/", serveSpec)
-	http.HandleFunc("/diff/rst/", serveRSTDiff)
-	http.HandleFunc("/diff/html/", serveHTMLDiff)
+	rand.Seed(time.Now().Unix())
+	masterCloneDir, err := gitClone(matrixDocCloneURL, false)
+	if err != nil {
+		log.Fatal(err)
+	}
+	s := server{masterCloneDir}
+	http.HandleFunc("/spec/", s.serveSpec)
+	http.HandleFunc("/diff/rst/", s.serveRSTDiff)
+	http.HandleFunc("/diff/html/", s.serveHTMLDiff)
 	http.HandleFunc("/healthz", serveText("ok"))
 	http.HandleFunc("/", listPulls)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))

From b6f0b67da66d4d9d286ad6efb641f79583456cbc Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 15:21:32 +0100
Subject: [PATCH 211/467] Speed up continuserv

Ignore .git directory as that shouldn't affect spec generation. Also, when
we receive writes from the OS, wait a bit before re-generating the spec to
clump together multiple writes rather than re-generating one after another
and waiting for no more writes before serving the request.
---
 scripts/continuserv/main.go | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index 658ae0fb..573c2c95 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -17,6 +17,7 @@ import (
 	"strings"
 	"sync"
 	"sync/atomic"
+	"time"
 
 	fsnotify "gopkg.in/fsnotify.v1"
 )
@@ -67,7 +68,6 @@ func watchFS(ch chan struct{}, w *fsnotify.Watcher) {
 		select {
 		case e := <-w.Events:
 			if filter(e) {
-				wg.Add(1)
 				fmt.Printf("Noticed change to %s, re-generating spec\n", e.Name)
 				ch <- struct{}{}
 			}
@@ -98,6 +98,11 @@ func filter(e fsnotify.Event) bool {
 		return false
 	}
 
+	// Ignore the .git directory - It's very noisy
+	if strings.Contains(e.Name, "/.git/") {
+		return false
+	}
+
 	// Avoid infinite cycles being caused by writing actual output
 	if strings.Contains(e.Name, "/tmp/") || strings.Contains(e.Name, "/gen/") {
 		return false
@@ -133,8 +138,20 @@ func populateOnce(dir string) {
 }
 
 func doPopulate(ch chan struct{}, dir string) {
-	for _ = range ch {
-		populateOnce(dir)
+	var pending int
+	for {
+		select {
+		case <-ch:
+			if pending == 0 {
+				wg.Add(1)
+			}
+			pending++
+		case <-time.After(10 * time.Millisecond):
+			if pending > 0 {
+				pending = 0
+				populateOnce(dir)
+			}
+		}
 	}
 }
 

From 2b7e02c0805b81d1c2ad72245f4dc13bdb56a445 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 11:26:02 +0100
Subject: [PATCH 212/467] Add sections for typing. Add swagger, JSON schema and
 example m.typing event

---
 api/client-server/v1/typing.yaml              | 77 +++++++++++++++++++
 event-schemas/examples/v1/m.typing            |  7 ++
 event-schemas/schema/v1/m.typing              | 28 +++++++
 .../modules/typing_notifications.rst          | 47 +++++------
 4 files changed, 131 insertions(+), 28 deletions(-)
 create mode 100644 api/client-server/v1/typing.yaml
 create mode 100644 event-schemas/examples/v1/m.typing
 create mode 100644 event-schemas/schema/v1/m.typing

diff --git a/api/client-server/v1/typing.yaml b/api/client-server/v1/typing.yaml
new file mode 100644
index 00000000..737c6928
--- /dev/null
+++ b/api/client-server/v1/typing.yaml
@@ -0,0 +1,77 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Typing API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/typing/{userId}":
+    put:
+      summary: Informs the server that the user has started or stopped typing.
+      description: |-
+        This tells the server that the user is typing for the next N
+        milliseconds where N is the value specified in the ``timeout`` key.
+        Alternatively, if ``typing`` is ``false``, it tells the server that the
+        user has stopped typing.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: userId
+          description: The user who has started to type.
+          required: true
+          x-example: "@alice:example.com"
+        - in: path
+          type: string
+          name: roomId
+          description: The room in which the user is typing.
+          required: true
+          x-example: "!wefh3sfukhs:example.com"
+        - in: body
+          name: typingState
+          description: The current typing state.
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "typing": true,
+                "timeout": 30000
+              }
+            properties:
+              typing:
+                type: boolean
+                description: |-
+                  Whether the user is typing or not. If ``false``, the ``timeout``
+                  key can be omitted.
+              timeout:
+                type: integer
+                description: The length of time in milliseconds to mark this user as typing.
+            required: ["typing"]
+      responses:
+        200:
+          description: The new typing state was set.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+
diff --git a/event-schemas/examples/v1/m.typing b/event-schemas/examples/v1/m.typing
new file mode 100644
index 00000000..bd53f6fb
--- /dev/null
+++ b/event-schemas/examples/v1/m.typing
@@ -0,0 +1,7 @@
+{
+    "type": "m.typing",
+    "room_id": "!z0mnsuiwhifuhwwfw:matrix.org",
+    "content": {
+        "user_ids": ["@alice:matrix.org", "@bob:example.com"]
+    }
+}
\ No newline at end of file
diff --git a/event-schemas/schema/v1/m.typing b/event-schemas/schema/v1/m.typing
new file mode 100644
index 00000000..b712f6ec
--- /dev/null
+++ b/event-schemas/schema/v1/m.typing
@@ -0,0 +1,28 @@
+{
+    "type": "object",
+    "title": "Typing Event",
+    "description": "Informs the client of the list of users currently typing.",
+    "properties": {
+        "content": {
+            "type": "object",
+            "properties": {
+                "user_ids": {
+                    "type": "array",
+                    "items": {
+                        "type": "string"
+                    },
+                    "description": "The list of user IDs typing in this room, if any."
+                }
+            },
+            "required": ["user_ids"]
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.typing"]
+        },
+        "room_id": {
+            "type": "string"
+        }
+    },
+    "required": ["type", "room_id", "content"]
+}
diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst
index 25b714ab..b32e3411 100644
--- a/specification/modules/typing_notifications.rst
+++ b/specification/modules/typing_notifications.rst
@@ -1,45 +1,30 @@
 Typing Notifications
---------------------
+====================
 
-Client APIs
-~~~~~~~~~~~
+Events
+------
 
-To set "I am typing for the next N msec"::
+{{m_typing_event}}
 
-  PUT .../rooms/<room_id>/typing/<user_id>
-  Content:  { "typing": true, "timeout": N }
-  # timeout is in milliseconds; suggested no more than 20 or 30 seconds
+Client behaviour
+----------------
+
+ - suggested no more than 20-30 seconds
 
 This should be re-sent by the client to continue informing the server the user
 is still typing; a safety margin of 5 seconds before the expected
 timeout runs out is recommended. Just keep declaring a new timeout, it will
 replace the old one.
 
-To set "I am no longer typing"::
-
-  PUT ../rooms/<room_id>/typing/<user_id>
-  Content: { "typing": false }
-
-Client Events
-~~~~~~~~~~~~~
-
-All room members will receive an event on the event stream::
-
-  {
-    "type": "m.typing",
-    "room_id": "!room-id-here:matrix.org",
-    "content": {
-      "user_ids": ["list of", "every user", "who is", "currently typing"]
-    }
-  }
-
-The client must use this list to *REPLACE* its knowledge of every user who is
+Event: The client must use this list to *REPLACE* its knowledge of every user who is
 currently typing. The reason for this is that the server DOES NOT remember
 users who are not currently typing, as that list gets big quickly. The client
 should mark as not typing, any user ID who is not in that list.
 
-Server APIs
-~~~~~~~~~~~
+{{typing_http_api}}
+
+Server behaviour
+----------------
 
 Servers will emit EDUs in the following form::
 
@@ -59,3 +44,9 @@ originating HSes to ensure they eventually send "stop" notifications.
   ((This will eventually need addressing, as part of the wider typing/presence
   timer addition work))
 
+Security considerations
+-----------------------
+
+Clients may not wish to inform everyone in a room that they are typing and
+instead only specific users in the room.
+

From a82f2ad4ac0b622899fcc039b05d2cd14c26e37c Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 11:55:34 +0100
Subject: [PATCH 213/467] Flesh out typing module

---
 specification/0-intro.rst                     |  2 +
 .../modules/typing_notifications.rst          | 39 ++++++++++++-------
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index 93458d97..5fec59c5 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -180,6 +180,8 @@ of a "Room".
 Event Graphs
 ~~~~~~~~~~~~
 
+.. _sect:event-graph:
+
 Events exchanged in the context of a room are stored in a directed acyclic graph
 (DAG) called an ``event graph``. The partial ordering of this graph gives the
 chronological ordering of events within the room. Each event in the graph has a
diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst
index b32e3411..110cd134 100644
--- a/specification/modules/typing_notifications.rst
+++ b/specification/modules/typing_notifications.rst
@@ -1,6 +1,13 @@
 Typing Notifications
 ====================
 
+Users often desire to see when another user is typing. This can be achieved
+using typing notifications. These are ephemeral events scoped to a ``room_id``.
+This means they do not form part of the `Event Graph`_ but still have a
+``room_id`` key.
+
+.. _Event Graph: `sect:event-graph`_
+
 Events
 ------
 
@@ -9,24 +16,28 @@ Events
 Client behaviour
 ----------------
 
- - suggested no more than 20-30 seconds
-
-This should be re-sent by the client to continue informing the server the user
-is still typing; a safety margin of 5 seconds before the expected
-timeout runs out is recommended. Just keep declaring a new timeout, it will
-replace the old one.
-
-Event: The client must use this list to *REPLACE* its knowledge of every user who is
-currently typing. The reason for this is that the server DOES NOT remember
-users who are not currently typing, as that list gets big quickly. The client
-should mark as not typing, any user ID who is not in that list.
+When a client receives an ``m.typing`` event, it MUST use the user ID list to
+**REPLACE** its knowledge of every user who is currently typing. The reason for
+this is that the server *does not remember* users who are not currently typing
+as that list gets big quickly. The client should mark as not typing any user ID
+who is not in that list.
+
+It is recommended that clients store a ``boolean`` indicating whether the user
+is typing or not. Whilst this value is ``true`` a timer should fire periodically
+every N seconds to send a typing HTTP request. The value of N is recommended to
+be no more than 20-30 seconds. This request should be re-sent by the client to
+continue informing the server the user is still typing. As subsequent
+requests will replace older requests, a safety margin of 5 seconds before the
+expected timeout runs out is recommended. When the user stops typing, the
+state change of the ``boolean`` to ``false`` should trigger another HTTP request
+to inform the server that the user has stopped typing.
 
 {{typing_http_api}}
 
 Server behaviour
 ----------------
 
-Servers will emit EDUs in the following form::
+Servers MUST emit typing EDUs in the following form::
 
   {
     "type": "m.typing",
@@ -37,8 +48,8 @@ Servers will emit EDUs in the following form::
     }
   }
 
-Server EDUs don't (currently) contain timing information; it is up to
-originating HSes to ensure they eventually send "stop" notifications.
+This does not contain timing information so it is up to originating homeservers
+to ensure they eventually send "stop" notifications.
 
 .. TODO
   ((This will eventually need addressing, as part of the wider typing/presence

From 097dc501805891c594dc218d6751fb529d2dd6a0 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 15:45:29 +0100
Subject: [PATCH 214/467] Minor tweaks

---
 specification/modules/typing_notifications.rst | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/specification/modules/typing_notifications.rst b/specification/modules/typing_notifications.rst
index 110cd134..74bd76bc 100644
--- a/specification/modules/typing_notifications.rst
+++ b/specification/modules/typing_notifications.rst
@@ -1,10 +1,10 @@
 Typing Notifications
 ====================
 
-Users often desire to see when another user is typing. This can be achieved
-using typing notifications. These are ephemeral events scoped to a ``room_id``.
-This means they do not form part of the `Event Graph`_ but still have a
-``room_id`` key.
+Users may wish to be informed when another user is typing in a room. This can be
+achieved using typing notifications. These are ephemeral events scoped to a
+``room_id``. This means they do not form part of the `Event Graph`_ but still
+have a ``room_id`` key.
 
 .. _Event Graph: `sect:event-graph`_
 
@@ -37,7 +37,8 @@ to inform the server that the user has stopped typing.
 Server behaviour
 ----------------
 
-Servers MUST emit typing EDUs in the following form::
+Servers MUST emit typing EDUs in a different form to ``m.typing`` events which
+are shown to clients. This form looks like::
 
   {
     "type": "m.typing",

From 9964dd1401ac0127a8476b14106274c9d84d0fbd Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 30 Sep 2015 16:11:31 +0100
Subject: [PATCH 215/467] Make explicit the state event only includes some keys

---
 event-schemas/schema/v1/m.room.member | 1 +
 1 file changed, 1 insertion(+)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 32302b85..c0fb103c 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -39,6 +39,7 @@
             "items": {
                 "type": "object",
                 "title": "StateEvent",
+                "description": "A stripped down state event, with only the ``type``, ``state_key`` and ``content`` keys.",
                 "properties": {
                     "type": {
                         "type": "string"

From 0320e8cef32d59ac84131f445937ddc2c05523a3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 16:41:47 +0100
Subject: [PATCH 216/467] Table tweaks from PR

---
 specification/0-feature_profiles.rst | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 59e015e9..1d88311b 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -14,19 +14,19 @@ in order for it to be classified as that profile.
 Summary
 -------
 
-============================ ===== =========== ======== ========= ===== =====
-  Module / Profile            Web   Embed-Web   Mobile   Desktop   CLI   IoT
-============================ ===== =========== ======== ========= ===== =====
- `End-to-End Encryption`_                                             
- `Instant Messaging`_         YES    YES         YES       YES     YES   YES
- `Presence`_                  YES                YES       YES     YES
- `Push Notifications`_                           YES
- `Receipts`_                  YES                YES       YES     YES
- `Typing Notifications`_      YES                YES       YES     YES
- `VoIP`_                      YES                YES       YES
- `Content Repository`_        YES                YES       YES     YES
- `History Visibility`_        YES                YES       YES     YES
-============================ ===== =========== ======== ========= ===== =====
+============================ ========== ========== ========== ========== ==========
+  Module / Profile              Web       Mobile    Desktop       CLI     Embedded
+============================ ========== ========== ========== ========== ==========
+ `Instant Messaging`_         Required   Required   Required   Required   Optional
+ `Presence`_                  Required   Required   Required   Required   Optional
+ `Push Notifications`_        Optional   Required   Optional   Optional   Optional
+ `Receipts`_                  Required   Required   Required   Required   Optional
+ `Typing Notifications`_      Required   Required   Required   Required   Optional
+ `VoIP`_                      Required   Required   Required   Optional   Optional
+ `Content Repository`_        Required   Required   Required   Optional   Optional
+ `History Visibility`_        Required   Required   Required   Required   Optional
+ `End-to-End Encryption`_     Optional   Optional   Optional   Optional   Optional 
+============================ ========== ========== ========== ========== ==========
 
 *Please see each module for more details on what clients need to implement.*
 

From be9402b66fbb49c4c23c57e7aec93d1f1efc03f3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 16:43:09 +0100
Subject: [PATCH 217/467] Move feature profiles section to modules as a
 sub-section

---
 specification/0-feature_profiles.rst | 2 +-
 specification/targets.yaml           | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 1d88311b..0580c18a 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -25,7 +25,7 @@ Summary
  `VoIP`_                      Required   Required   Required   Optional   Optional
  `Content Repository`_        Required   Required   Required   Optional   Optional
  `History Visibility`_        Required   Required   Required   Required   Optional
- `End-to-End Encryption`_     Optional   Optional   Optional   Optional   Optional 
+ `End-to-End Encryption`_     Optional   Optional   Optional   Optional   Optional
 ============================ ========== ========== ========== ========== ==========
 
 *Please see each module for more details on what clients need to implement.*
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 62585c69..d77bf8b5 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -2,11 +2,11 @@ targets:
   main:  # arbitrary name to identify this build target
     files:  # the sort order of files to cat
       - 0-intro.rst
-      - { 1: 0-feature_profiles.rst }
       - 1-client_server_api.rst
       - { 1: 0-events.rst }
       - { 1: 0-event_signing.rst }
       - 2-modules.rst
+      - { 1: 0-feature_profiles.rst }
       - { 1: "group:modules" }  # reference a group of files
       - 3-application_service_api.rst
       - 4-server_server_api.rst

From 91b6347f74c4e01ab62b56ee784b735c97d60df0 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 16:48:47 +0100
Subject: [PATCH 218/467] Explain what 'embedded' clients are

---
 specification/0-feature_profiles.rst | 30 ++++++++++++++++++----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 0580c18a..5a7e5513 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -50,14 +50,6 @@ This is a web page which heavily uses Matrix for communication. Single-page web
 apps would be classified as a stand-alone web client, as would multi-page web
 apps which use Matrix on nearly every page.
 
-Embedded web (``EmbedWeb``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This is a Matrix client which is embedded in another website, e.g. using
-iframes. These embedded clients are typically for a single purpose
-related to the website in question, and are not intended to be fully-fledged
-communication apps.
-
 Mobile (``Mobile``)
 ~~~~~~~~~~~~~~~~~~~
 
@@ -77,9 +69,25 @@ Command Line Interface (``CLI``)
 
 This is a client which is used via a text-based terminal.
 
-Internet of Things (``IoT``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Embedded (``Embedded``)
+~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a client which is embedded into another application or an embedded
+device.
+
+Application
++++++++++++
+
+This is a Matrix client which is embedded in another website, e.g. using
+iframes. These embedded clients are typically for a single purpose
+related to the website in question, and are not intended to be fully-fledged
+communication apps.
+
+Device
+++++++
 
 This is a client which is typically running on an embedded device such as a
-kettle, fridge or car.
+kettle, fridge or car. These clients tend to perform a few operations and run
+in a resource constrained environment. Like embedded applications, they are
+not intended to be fully-fledged communication systems.
 

From 6c3e70d2721554827d14d27ff07e892976ffcd24 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 30 Sep 2015 17:32:44 +0100
Subject: [PATCH 219/467] Start fleshing out voip module

---
 specification/1-client_server_api.rst |  2 +
 specification/modules/voip_events.rst | 84 ++++++++++++++++++---------
 2 files changed, 59 insertions(+), 27 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 7d8d57e4..db88bbd6 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -427,6 +427,8 @@ the complete dataset is provided in "chunk".
 Events
 ------
 
+.. _sect:events:
+
 Overview
 ~~~~~~~~
 
diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst
index 33998cd9..f1b8ae05 100644
--- a/specification/modules/voip_events.rst
+++ b/specification/modules/voip_events.rst
@@ -1,17 +1,18 @@
 Voice over IP
--------------
-Matrix can also be used to set up VoIP calls. This is part of the core
-specification, although is at a relatively early stage. Voice (and video) over
-Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like
-any other event. This means that clients must only send call events to rooms
-with exactly two participants as currently the WebRTC standard is based around
-two-party communication.
+=============
 
-{{voip_events}}
+This module outlines how two users in a room can set up a Voice over IP (VoIP)
+call to each other. Voice and video calls are built upon the WebRTC 1.0 standard.
+Call signalling is achieved by sending `message events`_ to the room. As a result,
+this means that clients MUST only send call events to rooms with exactly two
+participants as currently the WebRTC standard is based around two-party
+communication.
+
+.. _message events: `sect:events`_
 
 Message Exchange
-~~~~~~~~~~~~~~~~
-A call is set up with messages exchanged as follows:
+----------------
+A call is set up with message events exchanged as follows:
 
 ::
 
@@ -38,28 +39,57 @@ Or a rejected call:
 
 Calls are negotiated according to the WebRTC specification.
 
+Events
+------
+
+{{voip_events}}
+
+Client behaviour
+----------------
 
 Glare
 ~~~~~
-This specification aims to address the problem of two users calling each other
-at roughly the same time and their invites crossing on the wire. It is a far
-better experience for the users if their calls are connected if it is clear
-that their intention is to set up a call with one another. In Matrix, calls are
-to rooms rather than users (even if those rooms may only contain one other user)
-so we consider calls which are to the same room. The rules for dealing with such
-a situation are as follows:
-
- - If an invite to a room is received whilst the client is preparing to send an
-   invite to the same room, the client should cancel its outgoing call and
-   instead automatically accept the incoming call on behalf of the user.
- - If an invite to a room is received after the client has sent an invite to
-   the same room and is waiting for a response, the client should perform a
-   lexicographical comparison of the call IDs of the two calls and use the
-   lesser of the two calls, aborting the greater. If the incoming call is the
-   lesser, the client should accept this call on behalf of the user.
+
+"Glare" is a problem which occurs when two users call each other at roughly the
+same time. This results in the call failing to set up as there already is an
+incoming/outgoing call. A glare resolution algorithm can be used to determine
+which call to hangup and which call to answer. If both clients implement the
+same algorithm then they will both select the same call and the call will be
+successfully connected.
+
+
+As calls are "placed" to rooms rather than users, the glare resolution algorithm
+outlined below is only considered for calls which are to the same room. The
+algorithm is as follows:
+
+ - If an invite to a room is received whilst the client is **preparing to send**
+   an invite to the same room:
+
+    * the client should cancel its outgoing call and instead
+      automatically accept the incoming call on behalf of the user.
+
+ - If an invite to a room is received **after the client has sent** an invite to
+   the same room and is waiting for a response:
+
+    * the client should perform a lexicographical comparison of the call IDs of
+      the two calls and use the **lesser** of the two calls, aborting the
+      greater. If the incoming call is the lesser, the client should accept
+      this call on behalf of the user.
+
 
 The call setup should appear seamless to the user as if they had simply placed
-a call and the other party had accepted. Thusly, any media stream that had been
+a call and the other party had accepted. This means any media stream that had been
 setup for use on a call should be transferred and used for the call that
 replaces it.
 
+Server behaviour
+----------------
+
+TURN Servers
+~~~~~~~~~~~~
+
+Security considerations
+-----------------------
+
+
+

From d092b22848e4ec63aee7c157e2bc2203d6d48d87 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 09:23:08 +0100
Subject: [PATCH 220/467] Rename to 'Managing history visibility'

---
 specification/0-feature_profiles.rst | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index 5a7e5513..b9f12b74 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -14,19 +14,19 @@ in order for it to be classified as that profile.
 Summary
 -------
 
-============================ ========== ========== ========== ========== ==========
-  Module / Profile              Web       Mobile    Desktop       CLI     Embedded
-============================ ========== ========== ========== ========== ==========
- `Instant Messaging`_         Required   Required   Required   Required   Optional
- `Presence`_                  Required   Required   Required   Required   Optional
- `Push Notifications`_        Optional   Required   Optional   Optional   Optional
- `Receipts`_                  Required   Required   Required   Required   Optional
- `Typing Notifications`_      Required   Required   Required   Required   Optional
- `VoIP`_                      Required   Required   Required   Optional   Optional
- `Content Repository`_        Required   Required   Required   Optional   Optional
- `History Visibility`_        Required   Required   Required   Required   Optional
- `End-to-End Encryption`_     Optional   Optional   Optional   Optional   Optional
-============================ ========== ========== ========== ========== ==========
+===================================== ========== ========== ========== ========== ==========
+  Module / Profile                       Web       Mobile    Desktop       CLI     Embedded
+===================================== ========== ========== ========== ========== ==========
+ `Instant Messaging`_                  Required   Required   Required   Required   Optional
+ `Presence`_                           Required   Required   Required   Required   Optional
+ `Push Notifications`_                 Optional   Required   Optional   Optional   Optional
+ `Receipts`_                           Required   Required   Required   Required   Optional
+ `Typing Notifications`_               Required   Required   Required   Required   Optional
+ `VoIP`_                               Required   Required   Required   Optional   Optional
+ `Content Repository`_                 Required   Required   Required   Optional   Optional
+ `Managing History Visibility`_        Required   Required   Required   Required   Optional
+ `End-to-End Encryption`_              Optional   Optional   Optional   Optional   Optional
+===================================== ========== ========== ========== ========== ==========
 
 *Please see each module for more details on what clients need to implement.*
 
@@ -38,7 +38,7 @@ Summary
 .. _Typing Notifications: `module:typing`_
 .. _VoIP: `module:voip`_
 .. _Content Repository: `module:content`_
-.. _History Visibility: `module:history-visibility`_
+.. _Managing History Visibility: `module:history-visibility`_
 
 Clients
 -------

From e82661413e3bf51dd248778b0f3dd9a83b29efb9 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 11:04:42 +0100
Subject: [PATCH 221/467] Add /turnServer endpoint

---
 api/client-server/v1/voip.yaml        | 68 +++++++++++++++++++++++++++
 specification/modules/voip_events.rst | 30 +++++++-----
 2 files changed, 85 insertions(+), 13 deletions(-)
 create mode 100644 api/client-server/v1/voip.yaml

diff --git a/api/client-server/v1/voip.yaml b/api/client-server/v1/voip.yaml
new file mode 100644
index 00000000..5fdf1ca7
--- /dev/null
+++ b/api/client-server/v1/voip.yaml
@@ -0,0 +1,68 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Voice over IP API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/turnServer":
+    get:
+      summary: Obtain TURN server credentials.
+      description: |-
+        This API provides credentials for the client to use when initiating
+        calls.
+      security:
+        - accessToken: []
+      responses:
+        200:
+          description: The TURN server credentials.
+          examples:
+            application/json: |-
+              {
+                "username":"1443779631:@user:example.com",
+                "password":"JlKfBy1QwLrO20385QyAtEyIv0=",
+                "uris":[
+                  "turn:turn.example.com:3478?transport=udp",
+                  "turn:10.20.30.40:3478?transport=tcp",
+                  "turns:10.20.30.40:443?transport=tcp"
+                ],
+                "ttl":86400
+              }
+          schema:
+            type: object
+            properties:
+              username:
+                type: string
+                description: |-
+                  The username to use.
+              password:
+                type: string
+                description: |-
+                  The password to use.
+              uris:
+                type: array
+                items:
+                  type: string
+                description: A list of TURN URIs
+              ttl:
+                type: integer
+                description: The time-to-live in seconds
+            required: ["username", "password", "uris", "ttl"]
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+
diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst
index f1b8ae05..1bf1ea1d 100644
--- a/specification/modules/voip_events.rst
+++ b/specification/modules/voip_events.rst
@@ -10,8 +10,14 @@ communication.
 
 .. _message events: `sect:events`_
 
-Message Exchange
+Events
+------
+
+{{voip_events}}
+
+Client behaviour
 ----------------
+
 A call is set up with message events exchanged as follows:
 
 ::
@@ -39,14 +45,6 @@ Or a rejected call:
 
 Calls are negotiated according to the WebRTC specification.
 
-Events
-------
-
-{{voip_events}}
-
-Client behaviour
-----------------
-
 Glare
 ~~~~~
 
@@ -72,7 +70,7 @@ algorithm is as follows:
    the same room and is waiting for a response:
 
     * the client should perform a lexicographical comparison of the call IDs of
-      the two calls and use the **lesser** of the two calls, aborting the
+      the two calls and use the *lesser* of the two calls, aborting the
       greater. If the incoming call is the lesser, the client should accept
       this call on behalf of the user.
 
@@ -85,11 +83,17 @@ replaces it.
 Server behaviour
 ----------------
 
-TURN Servers
-~~~~~~~~~~~~
+The server MAY provide a TURN server which clients can use to contact the
+remote party. This server should be accessible via the HTTP endpoint listed
+below.
+
+{{voip_http_api}}
+
 
 Security considerations
 -----------------------
 
-
+Calls should only be placed to rooms with one other user in them. If they are
+placed to group chat rooms it is possible that another user will intercept and
+answer the call.
 

From 3b73b07babbcd86dbb8318f4813b27322a689848 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 11:11:08 +0100
Subject: [PATCH 222/467] Clarifications that room invites are m.call.invites
 not actual invites

---
 specification/modules/voip_events.rst | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst
index 1bf1ea1d..9d27c23b 100644
--- a/specification/modules/voip_events.rst
+++ b/specification/modules/voip_events.rst
@@ -60,14 +60,14 @@ As calls are "placed" to rooms rather than users, the glare resolution algorithm
 outlined below is only considered for calls which are to the same room. The
 algorithm is as follows:
 
- - If an invite to a room is received whilst the client is **preparing to send**
-   an invite to the same room:
+ - If an ``m.call.invite`` to a room is received whilst the client is
+   **preparing to send** an ``m.call.invite`` to the same room:
 
     * the client should cancel its outgoing call and instead
       automatically accept the incoming call on behalf of the user.
 
- - If an invite to a room is received **after the client has sent** an invite to
-   the same room and is waiting for a response:
+ - If an ``m.call.invite`` to a room is received **after the client has sent**
+   an ``m.call.invite`` to the same room and is waiting for a response:
 
     * the client should perform a lexicographical comparison of the call IDs of
       the two calls and use the *lesser* of the two calls, aborting the
@@ -83,9 +83,9 @@ replaces it.
 Server behaviour
 ----------------
 
-The server MAY provide a TURN server which clients can use to contact the
-remote party. This server should be accessible via the HTTP endpoint listed
-below.
+The homeserver MAY provide a TURN server which clients can use to contact the
+remote party. The following HTTP API endpoints will be used by clients in order
+to get information about the TURN server.
 
 {{voip_http_api}}
 

From 365a9076b93500cd87e85e68129e6ba7729f7625 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 12:11:26 +0100
Subject: [PATCH 223/467] Add nested dict template support; Add x-pattern

For cases where event schema specify `patternProperties` it would be nice
to give that pattern a "human-readable" form rather than a raw regex. This
is now supported by specifying `x-pattern` in the value part of the specified
pattern e.g. `patternProperties:{ "^.*":{ x-pattern: "$THING", ... } }`

Templating had limited record type descriptions limited to value primitives
e.g. `{string: integer}`. It now supports inspecting the values recursively
if the value is `object`.

Updated `m.receipt` to take both these points into account to make it read
better. Tweak receipt module text.
---
 event-schemas/schema/v1/m.receipt    |  9 ++++---
 specification/modules/receipts.rst   | 37 ++++++++++++----------------
 templating/matrix_templates/units.py | 26 +++++++++++++++----
 3 files changed, 43 insertions(+), 29 deletions(-)

diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index 0f365eed..5be232ad 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -5,17 +5,20 @@
     "properties": {
         "content": {
             "type": "object",
-            "description": "The event ids which the receipts relate to.",
             "patternProperties": {
                 "^\\$": {
                     "type": "object",
-                    "description": "The types of the receipts.",
+                    "x-pattern": "$EVENT_ID",
+                    "description": "The mapping of event ID to receipt type. The event ID is the ID which the receipts relate to and *not* an ID for the receipt itself. The key in the object is an enum which must be ``read``.",
                     "additionalProperties": {
                         "type": "object",
-                        "description": "User ids of the receipts",
+                        "title": "Users",
                         "patternProperties": {
                             "^@": {
                                 "type": "object",
+                                "title": "Receipt",
+                                "description": "The mapping of user ID to receipt. The user ID is the entity who sent this receipt.",
+                                "x-pattern": "$USER_ID",
                                 "properties": {
                                     "ts": {
                                         "type": "number",
diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst
index e2f83eea..f32c12db 100644
--- a/specification/modules/receipts.rst
+++ b/specification/modules/receipts.rst
@@ -6,8 +6,16 @@ have interacted with. For example, which events the user has read. For
 efficiency this is done as "up to" markers, i.e. marking a particular event
 as, say, ``read`` indicates the user has read all events *up to* that event.
 
-Client-Server API
-~~~~~~~~~~~~~~~~~
+Events
+------
+
+{{m_receipt_event}}
+
+Client behaviour
+----------------
+
+ - When clients should send receipts
+ - What clients should do when they receive these receipts
 
 Clients will receive receipts in the following format::
 
@@ -25,22 +33,6 @@ Clients will receive receipts in the following format::
         }
     }
 
-For example::
-
-    {
-        "type": "m.receipt",
-        "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org",
-        "content": {
-            "$1435641916114394fHBLK:matrix.org": {
-                "read": {
-                    "@erikj:jki.re": { "ts": 1436451550453 },
-                    ...
-                }
-            },
-            ...
-        }
-    }
-
 For efficiency, receipts are batched into one event per room. In the initialSync
 and v2 sync APIs the receipts are listed in a separate top level ``receipts``
 key. Each ``user_id``, ``receipt_type`` pair must be associated with only a
@@ -56,9 +48,8 @@ A client can update the markers for its user by issuing a request::
 Where the contents of the ``POST`` will be included in the content sent to
 other users. The server will automatically set the ``ts`` field.
 
-
-Server-Server API
-~~~~~~~~~~~~~~~~~
+Server behaviour
+----------------
 
 Receipts are sent across federation as EDUs with type ``m.receipt``. The
 format of the EDUs are::
@@ -75,3 +66,7 @@ format of the EDUs are::
 
 These are always sent as deltas to previously sent receipts.
 
+Security considerations
+-----------------------
+
+
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 50fa784e..473fdd82 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -49,8 +49,17 @@ def get_json_schema_object_fields(obj, enforce_title=False):
     }
     tables = [fields]
 
-    props = obj.get("properties", obj.get("patternProperties"))
     parents = obj.get("allOf")
+    props = obj.get("properties")
+    if not props:
+        props = obj.get("patternProperties")
+        if props:
+            # try to replace horrible regex key names with pretty x-pattern ones
+            for key_name in props.keys():
+                pretty_key = props[key_name].get("x-pattern")
+                if pretty_key:
+                    props[pretty_key] = props[key_name]
+                    del props[key_name]
     if not props and not parents:
         raise Exception(
             "Object %s has no properties or parents." % obj
@@ -70,10 +79,17 @@ def get_json_schema_object_fields(obj, enforce_title=False):
         if props[key_name]["type"] == "object":
             if props[key_name].get("additionalProperties"):
                 # not "really" an object, just a KV store
-                value_type = (
-                    "{string: %s}" %
-                    props[key_name]["additionalProperties"]["type"]
-                )
+                prop_val = props[key_name]["additionalProperties"]["type"]
+                if prop_val == "object":
+                    nested_object = get_json_schema_object_fields(
+                        props[key_name]["additionalProperties"],
+                        enforce_title=True
+                    )
+                    value_type = "{string: %s}" % nested_object[0]["title"]
+                    if not nested_object[0].get("no-table"):
+                        tables += nested_object
+                else:
+                    value_type = "{string: %s}" % prop_val
             else:
                 nested_object = get_json_schema_object_fields(
                     props[key_name], 

From c972dad8b383a8345864d32ba6997b5da1fd57fa Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 15:41:54 +0100
Subject: [PATCH 224/467] Flesh out receipts module. Add receipts swagger

Add templating support for v2 apis.
---
 api/client-server/v2_alpha/receipts.yaml | 68 +++++++++++++++++++++
 event-schemas/examples/v1/m.receipt      |  2 +-
 event-schemas/schema/v1/m.receipt        | 35 ++++++-----
 specification/modules/receipts.rst       | 76 ++++++++++++++----------
 templating/matrix_templates/units.py     | 30 ++++++----
 5 files changed, 151 insertions(+), 60 deletions(-)
 create mode 100644 api/client-server/v2_alpha/receipts.yaml

diff --git a/api/client-server/v2_alpha/receipts.yaml b/api/client-server/v2_alpha/receipts.yaml
new file mode 100644
index 00000000..4ef435b0
--- /dev/null
+++ b/api/client-server/v2_alpha/receipts.yaml
@@ -0,0 +1,68 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v2 Receipts API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/v2_alpha
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/receipt/{receiptType}/{eventId}":
+    post:
+      summary: Send a receipt for the given event ID.
+      description: |-
+        This API updates the marker for the given receipt type to the event ID
+        specified.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room in which to send the event.
+          required: true
+          x-example: "!wefuh21ffskfuh345:example.com"
+        - in: path
+          type: string
+          name: receiptType
+          description: The type of receipt to send.
+          required: true
+          x-example: "m.read"
+          enum: ["m.read"]
+        - in: path
+          type: string
+          name: eventId
+          description: The event ID to acknowledge up to.
+          required: true
+          x-example: "$1924376522eioj:example.com"
+        - in: body
+          description: |-
+            Extra receipt information to attach to ``content`` if any. The
+            server will automatically set the ``ts`` field.
+          schema:
+            type: object
+            example: |-
+              {}
+      responses:
+        200:
+          description: The receipt was sent.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
\ No newline at end of file
diff --git a/event-schemas/examples/v1/m.receipt b/event-schemas/examples/v1/m.receipt
index 83515317..bd0b726c 100644
--- a/event-schemas/examples/v1/m.receipt
+++ b/event-schemas/examples/v1/m.receipt
@@ -3,7 +3,7 @@
     "room_id": "!KpjVgQyZpzBwvMBsnT:matrix.org",
     "content": {
         "$1435641916114394fHBLK:matrix.org": {
-            "read": {
+            "m.read": {
                 "@rikj:jki.re": {
                     "ts": 1436451550453
                 }
diff --git a/event-schemas/schema/v1/m.receipt b/event-schemas/schema/v1/m.receipt
index 5be232ad..d0f79ac4 100644
--- a/event-schemas/schema/v1/m.receipt
+++ b/event-schemas/schema/v1/m.receipt
@@ -9,25 +9,28 @@
                 "^\\$": {
                     "type": "object",
                     "x-pattern": "$EVENT_ID",
-                    "description": "The mapping of event ID to receipt type. The event ID is the ID which the receipts relate to and *not* an ID for the receipt itself. The key in the object is an enum which must be ``read``.",
-                    "additionalProperties": {
-                        "type": "object",
-                        "title": "Users",
-                        "patternProperties": {
-                            "^@": {
-                                "type": "object",
-                                "title": "Receipt",
-                                "description": "The mapping of user ID to receipt. The user ID is the entity who sent this receipt.",
-                                "x-pattern": "$USER_ID",
-                                "properties": {
-                                    "ts": {
-                                        "type": "number",
-                                        "description": "The timestamp the receipt was sent at"
+                    "title": "Receipts",
+                    "description": "The mapping of event ID to a collection of receipts for this event ID. The event ID is the ID of the event being acknowledged and *not* an ID for the receipt itself.",
+                    "properties": {
+                        "m.read": {
+                            "type": "object",
+                            "title": "Users",
+                            "description": "A collection of users who have sent ``m.read`` receipts for this event.",
+                            "patternProperties": {
+                                "^@": {
+                                    "type": "object",
+                                    "title": "Receipt",
+                                    "description": "The mapping of user ID to receipt. The user ID is the entity who sent this receipt.",
+                                    "x-pattern": "$USER_ID",
+                                    "properties": {
+                                        "ts": {
+                                            "type": "number",
+                                            "description": "The timestamp the receipt was sent at."
+                                        }
                                     }
                                 }
                             }
-                        },
-                        "additionalProperties": false
+                        }
                     }
                 }
             },
diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst
index f32c12db..9dabba30 100644
--- a/specification/modules/receipts.rst
+++ b/specification/modules/receipts.rst
@@ -1,56 +1,63 @@
 Receipts
---------
+========
 
-Receipts are used to publish which events in a room the user or their devices
-have interacted with. For example, which events the user has read. For
-efficiency this is done as "up to" markers, i.e. marking a particular event
-as, say, ``read`` indicates the user has read all events *up to* that event.
+This module adds in support for receipts. These receipts are a form of
+acknowledgement of an event. This module defines a single acknowledgement:
+``m.read`` which indicates that the user has read up to a given event.
+
+Sending a receipt for each event can result in sending large amounts of traffic
+to a homeserver. To prevent this from becoming a problem, receipts are implemented
+using "up to" markers. This marker indicates that the acknowledgement applies
+to all events "up to and including" the event specified. For example, marking
+an event as "read" would indicate that the user had read all events *up to* the
+referenced event.
 
 Events
 ------
+Each ``user_id``, ``receipt_type`` pair must be associated with only a
+single ``event_id``.
 
 {{m_receipt_event}}
 
 Client behaviour
 ----------------
 
- - When clients should send receipts
- - What clients should do when they receive these receipts
-
-Clients will receive receipts in the following format::
+In v1 ``/initialSync``, receipts are listed in a separate top level ``receipts``
+key. In v2 ``/sync``, receipts are contained in the ``ephemeral`` block for a
+room. New receipts that come down the event streams are deltas which update
+existing mappings. Clients should replace older receipt acknowledgements based
+on ``user_id`` and ``receipt_type`` pairs. For example::
 
-     {
-        "type": "m.receipt",
-        "room_id": <room_id>,
-        "content": {
-            <event_id>: {
-                <receipt_type>: {
-                    <user_id>: { "ts": <ts>, ... },
-                    ...
-                }
-            },
-            ...
-        }
-    }
+  Client receives m.receipt:
+    user = @alice:example.com
+    receipt_type = m.read
+    event_id = $aaa:example.com
 
-For efficiency, receipts are batched into one event per room. In the initialSync
-and v2 sync APIs the receipts are listed in a separate top level ``receipts``
-key. Each ``user_id``, ``receipt_type`` pair must be associated with only a
-single ``event_id``. New receipts that come down the event streams are deltas.
-Deltas update existing mappings, clobbering based on ``user_id``,
-``receipt_type`` pairs.
+  Client receives another m.receipt:
+    user = @alice:example.com
+    receipt_type = m.read
+    event_id = $bbb:example.com
 
+  The client should replace the older acknowledgement for $aaa:example.com with
+  this one for $bbb:example.com
 
-A client can update the markers for its user by issuing a request::
+Clients should send read receipts when there is some certainty that the event in
+question has been **displayed** to the user. Simply receiving an event does not
+provide enough certainty that the user has seen the event. The user SHOULD need
+to *take some action* such as viewing the room that the event was sent to or
+dismissing a notification in order for the event to count as "read".
 
-    POST /_matrix/client/v2_alpha/rooms/<room_id>/receipt/read/<event_id>
+A client can update the markers for its user by interacting with the following
+HTTP APIs.
 
-Where the contents of the ``POST`` will be included in the content sent to
-other users. The server will automatically set the ``ts`` field.
+{{v2_receipts_http_api}}
 
 Server behaviour
 ----------------
 
+For efficiency, receipts SHOULD be batched into one event per room before
+delivering them to clients.
+
 Receipts are sent across federation as EDUs with type ``m.receipt``. The
 format of the EDUs are::
 
@@ -64,9 +71,12 @@ format of the EDUs are::
         ...
     }
 
-These are always sent as deltas to previously sent receipts.
+These are always sent as deltas to previously sent receipts. Currently only a
+single ``<receipt_type>`` should be used: ``m.read``.
 
 Security considerations
 -----------------------
 
+As receipts are sent outside the context of the event graph, there are no
+integrity checks performed on the contents of ``m.receipt`` events.
 
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 473fdd82..eca52acb 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -19,6 +19,7 @@ import yaml
 V1_CLIENT_API = "../api/client-server/v1"
 V1_EVENT_EXAMPLES = "../event-schemas/examples/v1"
 V1_EVENT_SCHEMA = "../event-schemas/schema/v1"
+V2_CLIENT_API = "../api/client-server/v2_alpha"
 CORE_EVENT_SCHEMA = "../event-schemas/schema/v1/core-event-schema"
 CHANGELOG = "../CHANGELOG.rst"
 TARGETS = "../specification/targets.yaml"
@@ -336,18 +337,27 @@ class MatrixUnits(Units):
         }
 
     def load_swagger_apis(self):
-        path = V1_CLIENT_API
+        paths = [
+            V1_CLIENT_API, V2_CLIENT_API
+        ]
         apis = {}
-        for filename in os.listdir(path):
-            if not filename.endswith(".yaml"):
+        for path in paths:
+            is_v2 = (path == V2_CLIENT_API)
+            if not os.path.exists(V2_CLIENT_API):
+                self.log("Skipping v2 apis: %s does not exist.", V2_CLIENT_API)
                 continue
-            self.log("Reading swagger API: %s" % filename)
-            with open(os.path.join(path, filename), "r") as f:
-                # strip .yaml
-                group_name = filename[:-5]
-                api = yaml.load(f.read())
-                api["__meta"] = self._load_swagger_meta(api, group_name)
-                apis[group_name] = api
+            for filename in os.listdir(path):
+                if not filename.endswith(".yaml"):
+                    continue
+                self.log("Reading swagger API: %s" % filename)
+                with open(os.path.join(path, filename), "r") as f:
+                    # strip .yaml
+                    group_name = filename[:-5]
+                    if is_v2:
+                        group_name = "v2_" + group_name
+                    api = yaml.load(f.read())
+                    api["__meta"] = self._load_swagger_meta(api, group_name)
+                    apis[group_name] = api
         return apis
 
     def load_common_event_fields(self):

From 560cd7a58fe6ecdb713dd3cd30b58b3e98a04d3d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 15:54:45 +0100
Subject: [PATCH 225/467] This isn't javascript. s/,/%/

---
 templating/matrix_templates/units.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index eca52acb..71b6acc6 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -344,7 +344,7 @@ class MatrixUnits(Units):
         for path in paths:
             is_v2 = (path == V2_CLIENT_API)
             if not os.path.exists(V2_CLIENT_API):
-                self.log("Skipping v2 apis: %s does not exist.", V2_CLIENT_API)
+                self.log("Skipping v2 apis: %s does not exist." % V2_CLIENT_API)
                 continue
             for filename in os.listdir(path):
                 if not filename.endswith(".yaml"):

From 87b6dd845e0e6c685800bbc13f9459a7d21ad59e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 17:55:16 +0100
Subject: [PATCH 226/467] Flesh out content repo; modify templating to support
 headers

Edit content-repo.yaml to include examples and headers.
Restructure content module to conform to the module template.
Adjust the HTTP API template to give 1 more char to the response
param to fit "Content-Disposition" correctly.
Edit the templating system to support displaying enums for
swagger APIs (before it was just JSON schema). Also add support
for introspecting headers from swagger. Finally, replace - with
_ when forming the {{ template_var }} else things whine.
---
 api/client-server/v1/content-repo.yaml        | 49 +++++++++++++++++--
 specification/modules/content_repo.rst        | 30 ++++++++++--
 .../matrix_templates/templates/http-api.tmpl  | 12 ++---
 templating/matrix_templates/units.py          | 22 +++++++--
 4 files changed, 96 insertions(+), 17 deletions(-)

diff --git a/api/client-server/v1/content-repo.yaml b/api/client-server/v1/content-repo.yaml
index fe3d1dc3..aa47dc18 100644
--- a/api/client-server/v1/content-repo.yaml
+++ b/api/client-server/v1/content-repo.yaml
@@ -15,16 +15,22 @@ paths:
       summary: Upload some content to the content repository.
       produces: ["application/json"]
       parameters:
+        - in: header
+          name: Content-Type
+          type: string
+          description: The content type of the file being uploaded
+          x-example: "Content-Type: audio/mpeg"
         - in: body
-          name: content
+          name: "<content>"
           description: The content to be uploaded.
           required: true
           schema:
             type: string
+            example: "<bytes>"
             format: byte
       responses:
         200:
-          description: Information about the uploaded content.
+          description: The MXC URI for the uploaded content.
           schema:
             type: object
             required: ["content_uri"]
@@ -32,6 +38,11 @@ paths:
               content_uri:
                 type: string
                 description: "The MXC URI to the uploaded content."
+          examples:
+            "application/json": |-
+              {
+                "content_uri": "mxc://example.com/AQwafuaFswefuhsfAFAgsw"
+              }
   "/download/{serverName}/{mediaId}":
     get:
       summary: "Download content from the content repository."
@@ -40,20 +51,32 @@ paths:
         - in: path
           type: string
           name: serverName
+          x-example: matrix.org
           required: true
           description: |
             The server name from the ``mxc://`` URI (the authoritory component)
         - in: path
           type: string
           name: mediaId
+          x-example: ascERGshawAWawugaAcauga
           required: true
           description: |
             The media ID from the ``mxc://`` URI (the path component)
       responses:
         200:
-          description: "The content downloaded."
+          description: "The content that was previously uploaded."
+          headers:
+            Content-Type:
+              description: "The content type of the file that was previously uploaded."
+              x-example: "audio/mpeg"
+              type: "string"
+            Content-Disposition:
+              description: "The name of the file that was previously uploaded, if set."
+              x-example: "attachment;filename=03-cool.mp3"
+              type: "string"
           schema:
             type: file
+            name: "<file>"
   "/thumbnail/{serverName}/{mediaId}":
     get:
       summary: "Download a thumbnail of the content from the content repository."
@@ -63,31 +86,47 @@ paths:
           type: string
           name: serverName
           required: true
+          x-example: matrix.org
           description: |
             The server name from the ``mxc://`` URI (the authoritory component)
         - in: path
           type: string
           name: mediaId
+          x-example: ascERGshawAWawugaAcauga
           required: true
           description: |
             The media ID from the ``mxc://`` URI (the path component)
         - in: query
           type: integer
+          x-example: 64
           name: width
-          description: The desired width of the thumbnail.
+          description: |-
+            The *desired* width of the thumbnail. The actual thumbnail may not
+            match the size specified.
         - in: query
           type: integer
+          x-example: 64
           name: height
-          description: The desired height of the thumbnail.
+          description: |-
+            The *desired* height of the thumbnail. The actual thumbnail may not
+            match the size specified.
         - in: query
           type: string
           enum: ["crop", "scale"]
           name: method
+          x-example: "scale"
           description: The desired resizing method.
       responses:
         200:
           description: "A thumbnail of the requested content."
+          headers:
+            Content-Type:
+              description: "The content type of the thumbnail."
+              x-example: "image/jpeg"
+              type: "string"
+              enum: ["image/jpeg", "image/png"]
           schema:
             type: file
+            name: "<file>"
             
 
diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index 83333f37..de464e20 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -3,8 +3,23 @@ Content repository
 
 .. _module:content:
 
-HTTP API
---------
+This module allows users to upload content to their homeserver which is
+retrievable from other homeservers. Its' purpose is to allow users to share
+attachments in a room. Content locations are represented as Matrix Content (MXC)
+URIs. They look like::
+
+  mxc://<server-name>/<media-id>
+
+  <server-name> : The name of the homeserver where this content can be found, e.g. matrix.org
+  <media-id> : An opaque ID which identifies the content.
+
+Client behaviour
+----------------
+
+Clients can upload and download content using the following HTTP APIs.
+
+{{content_repo_http_api}}
+
 
 Uploads are POSTed to a resource which returns a token which is used to GET
 the download.  Uploads are POSTed to the sender's local homeserver, but are
@@ -49,6 +64,9 @@ width and height are close to the requested size and the aspect matches
 the requested size. The client should scale the image if it needs to fit
 within a given rectangle.
 
+Server behaviour
+----------------
+
 Homeservers may generate thumbnails for content uploaded to remote
 homeservers themselves or may rely on the remote homeserver to thumbnail
 the content. Homeservers may return thumbnails of a different size to that
@@ -58,13 +76,19 @@ Homeservers must never upscale images.
 Security considerations
 -----------------------
 
+The HTTP GET endpoint does not require any authentication. Knowing the URL of
+the content is sufficient to retrieve the content, even if the entity isn't in
+the room.
+
+Homeservers have additional concerns:
+
  - Clients may try to upload very large files. Homeservers should not store files
    that are too large and should not serve them to clients.
 
  - Clients may try to upload very large images. Homeservers should not attempt to
    generate thumbnails for images that are too large.
 
- - Remote homeservers may host very large files or images. Homeserver should not
+ - Remote homeservers may host very large files or images. Homeservers should not
    proxy or thumbnail large files or images from remote homeservers.
 
  - Clients may try to upload a large number of files. Homeservers should limit the
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index eb3f3e64..472c9d7a 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -31,18 +31,18 @@ Response format:
 {% for table in endpoint.res_tables -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-================== ================= ===========================================
+=================== ================= ==========================================
       Param              Type                 Description
-================== ================= ===========================================
+=================== ================= ==========================================
 {% for row in table.rows -%}
 {# -#}
-{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
+{# Row type needs to prepend spaces to line up with the type column (20 ch) -#}
 {# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
-{#      It also needs to then wrap inside the desc col (43 ch width) -#}
+{#      It also needs to then wrap inside the desc col (42 ch width) -#}
 {# -#}
-{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(37)}}
+{{row.key}}{{row.type|indent(20-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(18 - (row.type|length))) |indent_block(38)}}
 {% endfor -%}
-================== ================= ===========================================
+=================== ================= ==========================================
 
 {% endfor %}
 {% endif -%}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 50fa784e..04f3bae1 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -151,6 +151,13 @@ class MatrixUnits(Units):
 
                     # assign value expected for this param
                     val_type = param.get("type") # integer/string
+
+                    if param.get("enum"):
+                        val_type = "enum"
+                        desc += (
+                            " One of: %s" % json.dumps(param.get("enum"))
+                        )
+
                     refType = Units.prop(param, "schema/$ref/") # Error,Event
                     schemaFmt = Units.prop(param, "schema/format") # bytes e.g. uploads
                     if not val_type and refType:
@@ -255,7 +262,7 @@ class MatrixUnits(Units):
                     res_type = Units.prop(good_response, "schema/type")
                     if res_type and res_type not in ["object", "array"]:
                         # response is a raw string or something like that
-                        endpoint["res_tables"].append({
+                        good_table = {
                             "title": None,
                             "rows": [{
                                 "key": good_response["schema"].get("name", ""),
@@ -263,7 +270,16 @@ class MatrixUnits(Units):
                                 "desc": res.get("description", ""),
                                 "req_str": ""
                             }]
-                        })
+                        }
+                        if good_response.get("headers"):
+                            for (header_name, header) in good_response.get("headers").iteritems():
+                                good_table["rows"].append({
+                                    "key": header_name,
+                                    "type": "Header<" + header["type"] + ">",
+                                    "desc": header["description"],
+                                    "req_str": ""
+                                })
+                        endpoint["res_tables"].append(good_table)
                     elif res_type and Units.prop(good_response, "schema/properties"):
                         # response is an object:
                         schema = good_response["schema"]
@@ -328,7 +344,7 @@ class MatrixUnits(Units):
             self.log("Reading swagger API: %s" % filename)
             with open(os.path.join(path, filename), "r") as f:
                 # strip .yaml
-                group_name = filename[:-5]
+                group_name = filename[:-5].replace("-", "_")
                 api = yaml.load(f.read())
                 api["__meta"] = self._load_swagger_meta(api, group_name)
                 apis[group_name] = api

From 8c4d7f50510edb2a8e0932248efed1949bf1715a Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 1 Oct 2015 18:03:34 +0100
Subject: [PATCH 227/467] Do not try to parse non-json request examples as json

---
 api/check_examples.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index a0cd0658..f08b2dc1 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -34,7 +34,7 @@ def check_parameter(filepath, request, parameter):
     example = None
     try:
         example_json = schema.get('example')
-        if example_json:
+        if example_json and not schema.get("format") == "byte":
             example = json.loads(example_json)
     except Exception as e:
         raise ValueError("Error parsing JSON example request for %r" % (

From 30232f20aa571cbd402ff56eaa970bbc0fc8d34c Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 1 Oct 2015 19:13:09 -0500
Subject: [PATCH 228/467] speculator: Merge after fetching, so that /spec/head
 works

---
 scripts/speculator/main.go | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 6ee80150..442400df 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -60,11 +60,10 @@ func (u *User) IsTrusted() bool {
 }
 
 const (
-	pullsPrefix = "https://api.github.com/repos/matrix-org/matrix-doc/pulls"
+	pullsPrefix       = "https://api.github.com/repos/matrix-org/matrix-doc/pulls"
 	matrixDocCloneURL = "https://github.com/matrix-org/matrix-doc.git"
 )
 
-
 func gitClone(url string, shared bool) (string, error) {
 	directory := path.Join("/tmp/matrix-doc", strconv.FormatInt(rand.Int63(), 10))
 	cmd := exec.Command("git", "clone", url, directory)
@@ -80,21 +79,22 @@ func gitClone(url string, shared bool) (string, error) {
 }
 
 func gitCheckout(path, sha string) error {
-	cmd := exec.Command("git", "checkout", sha)
-	cmd.Dir = path
-	err := cmd.Run()
-	if err != nil {
-		return fmt.Errorf("error checking out repo: %v", err)
+	return runGitCommand(path, []string{"checkout", sha})
+}
+
+func gitFetchAndMerge(path string) error {
+	if err := runGitCommand(path, []string{"fetch"}); err != nil {
+		return err
 	}
-	return nil
+	return runGitCommand(path, []string{"merge"})
 }
 
-func gitFetch(path string) error {
-	cmd := exec.Command("git", "fetch")
+func runGitCommand(path string, args []string) error {
+	cmd := exec.Command("git", args...)
 	cmd.Dir = path
 	err := cmd.Run()
 	if err != nil {
-		return fmt.Errorf("error fetching repo: %v", err)
+		return fmt.Errorf("error running %s: %v", strings.Join(cmd.Args, " "), err)
 	}
 	return nil
 }
@@ -145,7 +145,7 @@ type server struct {
 // generateAt generates spec from repo at sha.
 // Returns the path where the generation was done.
 func (s *server) generateAt(sha string) (dst string, err error) {
-	err = gitFetch(s.matrixDocCloneURL)
+	err = gitFetchAndMerge(s.matrixDocCloneURL)
 	if err != nil {
 		return
 	}

From a69e03f57778b9ef743cb03430db472e1b0a28a9 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 1 Oct 2015 19:15:30 -0500
Subject: [PATCH 229/467] speculator: Report listening port

---
 scripts/speculator/main.go | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 6ee80150..ece65372 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -362,6 +362,8 @@ func main() {
 	http.HandleFunc("/diff/html/", s.serveHTMLDiff)
 	http.HandleFunc("/healthz", serveText("ok"))
 	http.HandleFunc("/", listPulls)
+
+	fmt.Printf("Listening on port %d\n", *port)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 }
 

From af7d2ca9fcc7ebb49aa4d35fa06b7dcdb1494447 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 1 Oct 2015 22:19:11 -0500
Subject: [PATCH 230/467] Update 3pid invite spec

This takes into account:
 1) That finding the existing servers of a room is hard
 2) Federation
---
 api/client-server/v1/membership.yaml          | 29 +++++++-------
 event-schemas/examples/v1/m.room.member       |  7 +++-
 .../examples/v1/m.room.third_party_invite     | 14 +++++++
 event-schemas/schema/v1/m.room.member         | 24 +++++++++++-
 .../schema/v1/m.room.third_party_invite       | 37 ++++++++++++++++++
 specification/modules/third_party_invites.rst | 39 +++++++++++++++++++
 specification/targets.yaml                    |  1 +
 7 files changed, 135 insertions(+), 16 deletions(-)
 create mode 100644 event-schemas/examples/v1/m.room.third_party_invite
 create mode 100644 event-schemas/schema/v1/m.room.third_party_invite
 create mode 100644 specification/modules/third_party_invites.rst

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index c4d658fb..ff35fa35 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -159,26 +159,29 @@ paths:
         If the identity server does not know a Matrix user identifier for the
         passed third party identifier, the homeserver will issue an invitation
         which can be accepted upon providing proof of ownership of the third
-        party identifier. This is achieved by requesting a nonce and digest from
-        the identity server. When a user binds the invited third party
-        identifier to a Matrix user ID, the identity server will give the user a
-        list of pending invitations, each containing:
+        party identifier. This is achieved by the identity server generating a
+        token, which it gives to the inviting homeserver. The homeserver will
+        add an ``m.room.third_party_invite`` event into the graph for the room,
+        containing that token.
+
+        When a user binds the invited third party identifier to a Matrix user ID,
+        the identity server will give the user a list of pending invitations,
+        each containing:
 
           - The room ID to which they were invited
 
-          - The digest given to the homeserver
+          - The token given to the homeserver
 
-          - A secret which, when appended to the nonce, digests to the above digest,
-            i.e. digest = sha256(nonce + secret)
+          - A signature of the token, signed with the identity server's private key
 
-        The digest algorithm to be used is SHA256.
+          - The matrix user ID who invited them to the room
 
         If the identity server did know the Matrix user identifier for the
         third party identifier, the home server will append a ``m.room.member``
         event to the room.
 
-        If a digest and nonce are requested from the identity server, the home
-        server will append a ``m.room.token_based_invite`` event to the room.
+        If a token is requested from the identity server, the home server will
+        append a ``m.room.third_party_invite`` event to the room.
       security:
         - accessToken: []
       parameters:
@@ -195,13 +198,13 @@ paths:
             type: object
             example: |-
               {
-                "identity_server": "matrix.org",
+                "id_server": "matrix.org",
                 "medium": "email",
                 "address": "cheeky@monkey.com",
                 "display_name": "A very cheeky monkey"
               }
             properties:
-              identity_server:
+              id_server:
                 type: string
                 description: The hostname+port of the identity server which should be used for third party identifier lookups.
               medium:
@@ -213,7 +216,7 @@ paths:
               display_name:
                 type: string
                 description: A user-friendly string describing who has been invited. It should not contain the address of the invitee, to avoid leaking mappings between third party identities and matrix user IDs.
-            required: ["identity_server", "medium", "address", "display_name"]
+            required: ["id_server", "medium", "address", "display_name"]
       responses:
         200:
           description: The user has been invited to join the room.
diff --git a/event-schemas/examples/v1/m.room.member b/event-schemas/examples/v1/m.room.member
index b9cd2671..279eb484 100644
--- a/event-schemas/examples/v1/m.room.member
+++ b/event-schemas/examples/v1/m.room.member
@@ -3,7 +3,12 @@
   "content": {
     "membership": "join",
     "avatar_url": "mxc://localhost/SEsfnsuifSDFSSEF#auto",
-    "displayname": "Alice Margatroid"
+    "displayname": "Alice Margatroid",
+    "token": "pc98",
+    "public_key": "abc123",
+    "key_validity_url": "https://magic.forest/verifykey",
+    "signature": "q1w2e3",
+    "sender": "@zun:zun.soft"
   },
   "state_key": "@alice:localhost",
   "origin_server_ts": 1431961217939,
diff --git a/event-schemas/examples/v1/m.room.third_party_invite b/event-schemas/examples/v1/m.room.third_party_invite
new file mode 100644
index 00000000..82ac5248
--- /dev/null
+++ b/event-schemas/examples/v1/m.room.third_party_invite
@@ -0,0 +1,14 @@
+{
+  "age": 242352,
+  "content": {
+    "display_name": "Alice Margatroid",
+    "key_validity_url": "https://magic.forest/verifykey",
+    "public_key": "abc123"
+  },
+  "state_key": "pc98",
+  "origin_server_ts": 1431961217939,
+  "event_id": "$WLGTSEFSEF:localhost",
+  "type": "m.room.third_party_invite",
+  "room_id": "!Cuyf34gef24t:localhost",
+  "sender": "@example:localhost"
+}
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index c0fb103c..a3be9a1d 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "The current membership state of a user in the room.",
-    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail.",
+    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. The token, public_key, key_validity_url, signature, and sender properties either must all or none be set - they are set if the invite was an ``m.room.third_party_invite`` event, and absent if the invite was an ``m.room.member`` event.",
     "allOf": [{
         "$ref": "core-event-schema/state_event.json"
     }],
@@ -21,7 +21,27 @@
                 "displayname": {
                     "type": ["string", "null"],
                     "description": "The display name for this user, if any. This is added by the homeserver."
-                }
+                },
+		"token": {
+		    "type": "string",
+		    "description": "A token which must be correctly signed, in order to join the room."
+		},
+		"key_validity_url": {
+		    "type": "string",
+		    "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
+		},
+		"public_key": {
+		    "type": "string",
+		    "description": "A base64-encoded ed25519 key with which token must be signed."
+		},
+		"signature": {
+		    "type": "string",
+		    "description": "A base64-encoded signature of token with public_key."
+		},
+		"sender": {
+		    "type": "string",
+		    "description": "The matrix user ID of the user who send the invite which is being used."
+		}
             },
             "required": ["membership"]
         },
diff --git a/event-schemas/schema/v1/m.room.third_party_invite b/event-schemas/schema/v1/m.room.third_party_invite
new file mode 100644
index 00000000..eae8c015
--- /dev/null
+++ b/event-schemas/schema/v1/m.room.third_party_invite
@@ -0,0 +1,37 @@
+{
+    "$schema": "http://json-schema.org/draft-04/schema#",
+    "type": "object",
+    "title": "An invitation to a room issued to a third party identifier, rather than a matrix user ID.",
+    "description": "Acts as an ``m.room.member`` invite event, where there isn't a target user_id to invite. This event contains a token and a public key whose private key must be used to sign the token. Any user who can present that signature may use this invitation to join the target room.",
+    "allOf": [{
+        "$ref": "core#/definitions/state_event"
+    }],
+    "properties": {
+        "content": {
+            "type": "object",
+            "properties": {
+                "display_name": {
+                    "type": "string",
+                    "description": "A user-readable string which represents the user who has been invited. This should not contain the user's third party ID, as otherwise when the invite is accepted it would leak the association between the matrix ID and the third party ID."
+                },
+		"key_validity_url": {
+		    "type": "string",
+		    "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
+		},
+		"public_key": {
+		    "type": "string",
+		    "description": "A base64-encoded ed25519 key with which token must be signed."
+		}
+            },
+            "required": ["display_name", "key_validity_url", "public_key"]
+        },
+        "state_key": {
+            "type": "string",
+            "description": "The token, of which a signature must be produced in order to join the room."
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.room.third_party_invite"]
+        }
+    }
+}
diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
new file mode 100644
index 00000000..ae0d74e8
--- /dev/null
+++ b/specification/modules/third_party_invites.rst
@@ -0,0 +1,39 @@
+Third party invites
+===================
+
+.. _module:third_party_invites:
+
+This module adds in support for inviting new members to a room where their
+Matrix user ID is not known, instead addressing them by a third party identifier
+such as an email address.
+
+Events
+------
+
+{{m_room_third_party_invite_event}}
+
+Client behaviour
+----------------
+
+A client asks a server to invite a user by their third party identifier.
+
+See the documentation for /invite for more information.
+
+Server behaviour
+----------------
+
+All homeservers MUST verify that sign(``token``, ``public_key``) = ``signature``.
+
+If a client of the current homeserver is joining by an
+``m.room.third_party_invite``, that homesever MUST validate that the public
+key used for signing is still valid, by checking ``key_validity_url``.
+
+If a homeserver is joining a room for the first time because of an
+``m.room.third_party_invite``, the server which is already participating in the
+room MUST validate that the public key used for signing is still valid, by
+checking ``key_validity_url``.
+
+No other homeservers may reject the joining of the room on the basis of
+``key_validity_url``, this is so that all homeservers have a consistent view of
+the room.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index d77bf8b5..4b404ff1 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -22,6 +22,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/content_repo.rst
     - modules/end_to_end_encryption.rst
     - modules/history_visibility.rst
+    - modules/third_party_invites.rst
     - modules/push_overview.rst
       # relative depth
     - { 1: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }

From e996933a29badce5676ebc2ecd52ff8374cd44b1 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 1 Oct 2015 22:21:34 -0500
Subject: [PATCH 231/467] Banish the tabs

---
 .../schema/v1/m.room.third_party_invite          | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.third_party_invite b/event-schemas/schema/v1/m.room.third_party_invite
index eae8c015..dc010fe7 100644
--- a/event-schemas/schema/v1/m.room.third_party_invite
+++ b/event-schemas/schema/v1/m.room.third_party_invite
@@ -14,14 +14,14 @@
                     "type": "string",
                     "description": "A user-readable string which represents the user who has been invited. This should not contain the user's third party ID, as otherwise when the invite is accepted it would leak the association between the matrix ID and the third party ID."
                 },
-		"key_validity_url": {
-		    "type": "string",
-		    "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
-		},
-		"public_key": {
-		    "type": "string",
-		    "description": "A base64-encoded ed25519 key with which token must be signed."
-		}
+        "key_validity_url": {
+            "type": "string",
+            "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
+        },
+        "public_key": {
+            "type": "string",
+            "description": "A base64-encoded ed25519 key with which token must be signed."
+        }
             },
             "required": ["display_name", "key_validity_url", "public_key"]
         },

From fdccf35c4745147db601c06673e3641fa5a7ad70 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 1 Oct 2015 22:23:24 -0500
Subject: [PATCH 232/467] Banish more tabs

---
 event-schemas/schema/v1/m.room.member | 40 +++++++++++++--------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index a3be9a1d..47ce4d6c 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -22,26 +22,26 @@
                     "type": ["string", "null"],
                     "description": "The display name for this user, if any. This is added by the homeserver."
                 },
-		"token": {
-		    "type": "string",
-		    "description": "A token which must be correctly signed, in order to join the room."
-		},
-		"key_validity_url": {
-		    "type": "string",
-		    "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
-		},
-		"public_key": {
-		    "type": "string",
-		    "description": "A base64-encoded ed25519 key with which token must be signed."
-		},
-		"signature": {
-		    "type": "string",
-		    "description": "A base64-encoded signature of token with public_key."
-		},
-		"sender": {
-		    "type": "string",
-		    "description": "The matrix user ID of the user who send the invite which is being used."
-		}
+        "token": {
+            "type": "string",
+            "description": "A token which must be correctly signed, in order to join the room."
+        },
+        "key_validity_url": {
+            "type": "string",
+            "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
+        },
+        "public_key": {
+            "type": "string",
+            "description": "A base64-encoded ed25519 key with which token must be signed."
+        },
+        "signature": {
+            "type": "string",
+            "description": "A base64-encoded signature of token with public_key."
+        },
+        "sender": {
+            "type": "string",
+            "description": "The matrix user ID of the user who send the invite which is being used."
+        }
             },
             "required": ["membership"]
         },

From 9cda004848b230b943dd3fcf7310e9c549b96b3b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 1 Oct 2015 22:27:19 -0500
Subject: [PATCH 233/467] Fix ref

---
 event-schemas/schema/v1/m.room.third_party_invite | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.third_party_invite b/event-schemas/schema/v1/m.room.third_party_invite
index dc010fe7..ba66100f 100644
--- a/event-schemas/schema/v1/m.room.third_party_invite
+++ b/event-schemas/schema/v1/m.room.third_party_invite
@@ -4,7 +4,7 @@
     "title": "An invitation to a room issued to a third party identifier, rather than a matrix user ID.",
     "description": "Acts as an ``m.room.member`` invite event, where there isn't a target user_id to invite. This event contains a token and a public key whose private key must be used to sign the token. Any user who can present that signature may use this invitation to join the target room.",
     "allOf": [{
-        "$ref": "core#/definitions/state_event"
+        "$ref": "core-event-schema/state_event.json"
     }],
     "properties": {
         "content": {

From 3d9dbe42e691f446830e348eb680297728dadede Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 2 Oct 2015 10:21:48 +0100
Subject: [PATCH 234/467] Bump to swagger-parser 3.2.1 - remove x- keys on
 headers

Removed x- keys due to https://github.com/BigstickCarpet/swagger-parser/issues/23
---
 api/client-server/v1/content-repo.yaml |  5 -----
 api/package.json                       |  2 +-
 api/validator.js                       | 10 +++++-----
 3 files changed, 6 insertions(+), 11 deletions(-)

diff --git a/api/client-server/v1/content-repo.yaml b/api/client-server/v1/content-repo.yaml
index aa47dc18..8e6e8d1a 100644
--- a/api/client-server/v1/content-repo.yaml
+++ b/api/client-server/v1/content-repo.yaml
@@ -68,15 +68,12 @@ paths:
           headers:
             Content-Type:
               description: "The content type of the file that was previously uploaded."
-              x-example: "audio/mpeg"
               type: "string"
             Content-Disposition:
               description: "The name of the file that was previously uploaded, if set."
-              x-example: "attachment;filename=03-cool.mp3"
               type: "string"
           schema:
             type: file
-            name: "<file>"
   "/thumbnail/{serverName}/{mediaId}":
     get:
       summary: "Download a thumbnail of the content from the content repository."
@@ -122,11 +119,9 @@ paths:
           headers:
             Content-Type:
               description: "The content type of the thumbnail."
-              x-example: "image/jpeg"
               type: "string"
               enum: ["image/jpeg", "image/png"]
           schema:
             type: file
-            name: "<file>"
             
 
diff --git a/api/package.json b/api/package.json
index 15193493..84b9dd7b 100644
--- a/api/package.json
+++ b/api/package.json
@@ -10,6 +10,6 @@
   "license": "ISC",
   "dependencies": {
     "nopt": "^3.0.2",
-    "swagger-parser": "^2.4.1"
+    "swagger-parser": "^3.2.1"
   }
 }
diff --git a/api/validator.js b/api/validator.js
index 3b89a5a3..0d76c09d 100644
--- a/api/validator.js
+++ b/api/validator.js
@@ -26,11 +26,10 @@ if (!opts.schema) {
 }
 
 
-var errFn = function(err, api, metadata) {
+var errFn = function(err, api) {
     if (!err) {
         return;
     }
-    console.log(metadata);
     console.error(err);
     process.exit(1);
 };
@@ -46,11 +45,12 @@ if (isDir) {
         files.forEach(function(f) {
             var suffix = ".yaml";
             if (f.indexOf(suffix, f.length - suffix.length) > 0) {
-                parser.parse(path.join(opts.schema, f), function(err, api, metadata) {
+                parser.validate(path.join(opts.schema, f), function(err, api, metadata) {
                     if (!err) {
                         console.log("%s is valid.", f);
                     }
                     else {
+                        console.error("%s is not valid.", f);
                         errFn(err, api, metadata);
                     }
                 });
@@ -59,12 +59,12 @@ if (isDir) {
     });
 }
 else{
-    parser.parse(opts.schema, function(err, api, metadata) {
+    parser.validate(opts.schema, function(err, api) {
         if (!err) {
             console.log("%s is valid", opts.schema);
         }
         else {
-            errFn(err, api, metadata);
+            errFn(err, api);
         }
     });
 };

From dbc72c43aca270d86c128caba1cde434642bcb8a Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 2 Oct 2015 10:28:29 +0100
Subject: [PATCH 235/467] s/private_chat_shared_power/trusted_private_chat/

---
 specification/1-client_server_api.rst | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 329a87dd..1c64f021 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -756,7 +756,7 @@ options which can be set when creating a room:
   Optional:
     Yes
   Value:
-    ``private_chat``, ``private_chat_shared_power`` or ``public_chat``
+    ``private_chat``, ``trusted_private_chat`` or ``public_chat``
   Description:
     Convenience parameter for setting various default state events based on a
     preset.
@@ -765,9 +765,9 @@ options which can be set when creating a room:
 
     - ``private_chat``: Sets the ``join_rules`` to ``invite`` and
       ``history_visibility`` to ``shared``
-    - ``private_chat_shared_power``: Set the ``join_rules`` to
-      ``invite``, ``history_visibility`` to ``shared`` and gives all invitees
-      the same power level as the creator.
+    - ``trusted_private_chat``: Set the ``join_rules`` to ``invite``,
+      ``history_visibility`` to ``shared`` and gives all invitees the same
+      power level as the creator.
     - ``public_chat``: Sets the ``join_rules`` to ``public`` and
       ``history_visibility`` to ``shared``
 

From 912a8ca7605278322cc8f8c921002022ea4e34c7 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 2 Oct 2015 10:30:45 +0100
Subject: [PATCH 236/467] Be more explicit about keys that are clobbered

---
 specification/1-client_server_api.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index dd58b357..40b46be0 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -757,7 +757,8 @@ options which can be set when creating a room:
     Yes
   Value:
     Extra keys to be added to the content of the ``m.room.create``. The server
-    will clober certain keys, e.g. ``creator``.
+    will clober the following keys: ``creator``. Future versions of this
+    spec may allow the server to clobber other keys if required.
   Description:
     Allows clients to add keys to the content of ``m.room.create``.
 

From 4dabcd112ef787c2f09ddbd61415e145e1b146e3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 2 Oct 2015 10:44:50 +0100
Subject: [PATCH 237/467] Remove redundant info now we have the http api
 template. Minor tweaks to display of schema with no names but a type

---
 specification/modules/content_repo.rst | 46 +++++---------------------
 templating/matrix_templates/units.py   |  3 +-
 2 files changed, 11 insertions(+), 38 deletions(-)

diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index de464e20..9ac5e199 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -10,9 +10,15 @@ URIs. They look like::
 
   mxc://<server-name>/<media-id>
 
-  <server-name> : The name of the homeserver where this content can be found, e.g. matrix.org
+  <server-name> : The name of the homeserver where this content originated, e.g. matrix.org
   <media-id> : An opaque ID which identifies the content.
 
+Uploads are POSTed to a resource on the user's local homeserver which returns a
+token which is used to GET the download. Content is downloaded from the
+recipient's local homeserver, which must first transfer the content from the
+origin homeserver using the same API (unless the origin and destination
+homeservers are the same).
+
 Client behaviour
 ----------------
 
@@ -20,42 +26,8 @@ Clients can upload and download content using the following HTTP APIs.
 
 {{content_repo_http_api}}
 
-
-Uploads are POSTed to a resource which returns a token which is used to GET
-the download.  Uploads are POSTed to the sender's local homeserver, but are
-downloaded from the recipient's local homeserver, which must thus first transfer
-the content from the origin homeserver using the same API (unless the origin
-and destination homeservers are the same).  The upload/download API is::
-
-    => POST /_matrix/media/v1/upload HTTP/1.1
-       Content-Type: <media-type>
-
-       <media>
-
-    <= HTTP/1.1 200 OK
-       Content-Type: application/json
-
-       { "content-uri": "mxc://<server-name>/<media-id>" }
-
-    => GET /_matrix/media/v1/download/<server-name>/<media-id> HTTP/1.1
-
-    <= HTTP/1.1 200 OK
-       Content-Type: <media-type>
-       Content-Disposition: attachment;filename=<upload-filename>
-
-       <media>
-
-Clients can get thumbnails by supplying a desired width and height and
-thumbnailing method::
-
-    => GET /_matrix/media/v1/thumbnail/<server_name>
-            /<media-id>?width=<w>&height=<h>&method=<m> HTTP/1.1
-
-    <= HTTP/1.1 200 OK
-       Content-Type: image/jpeg or image/png
-
-       <thumbnail>
-
+Thumbnails
+~~~~~~~~~~
 The thumbnail methods are "crop" and "scale". "scale" tries to return an
 image where either the width or the height is smaller than the requested
 size. The client should then scale and letterbox the image if it needs to
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 04f3bae1..1e449b5d 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -260,12 +260,13 @@ class MatrixUnits(Units):
                 if good_response:
                     self.log("Found a 200 response for this API")
                     res_type = Units.prop(good_response, "schema/type")
+                    res_name = Units.prop(good_response, "schema/name")
                     if res_type and res_type not in ["object", "array"]:
                         # response is a raw string or something like that
                         good_table = {
                             "title": None,
                             "rows": [{
-                                "key": good_response["schema"].get("name", ""),
+                                "key": "<" + res_type + ">" if not res_name else res_name,
                                 "type": res_type,
                                 "desc": res.get("description", ""),
                                 "req_str": ""

From a73cc50aa963f05d86783272a1ab105a70a84f34 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 2 Oct 2015 13:03:39 +0100
Subject: [PATCH 238/467] s|client/api/v2_alpha|client/v2_alpha|

---
 api/client-server/v2_alpha/filter.yaml | 2 +-
 api/client-server/v2_alpha/sync.yaml   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 30d52844..37a0a3aa 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -5,7 +5,7 @@ info:
 host: localhost:8008
 schemes:
   - https
-basePath: /_matrix/client/api/v2_alpha
+basePath: /_matrix/client/v2_alpha
 consumes:
   - application/json
 produces:
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 238fab63..7daa33f5 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -5,7 +5,7 @@ info:
 host: localhost:8008
 schemes:
   - https
-basePath: /_matrix/client/api/v2_alpha
+basePath: /_matrix/client/v2_alpha
 consumes:
   - application/json
 produces:

From 0e8f1b5475f70d8436ec3a1d19886c2fe71ca677 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 2 Oct 2015 07:33:26 -0500
Subject: [PATCH 239/467] Quote args

---
 scripts/speculator/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 442400df..07ffa44e 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -94,7 +94,7 @@ func runGitCommand(path string, args []string) error {
 	cmd.Dir = path
 	err := cmd.Run()
 	if err != nil {
-		return fmt.Errorf("error running %s: %v", strings.Join(cmd.Args, " "), err)
+		return fmt.Errorf("error running %q: %v", strings.Join(cmd.Args, " "), err)
 	}
 	return nil
 }

From 9d8dc7971dd697c50747432784ae4ccc80b6e313 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 2 Oct 2015 08:33:12 -0500
Subject: [PATCH 240/467] Review comments

---
 event-schemas/examples/v1/m.room.member       | 12 +++--
 event-schemas/schema/v1/m.room.member         | 47 ++++++++++-------
 specification/modules/third_party_invites.rst | 52 ++++++++++++++++---
 3 files changed, 80 insertions(+), 31 deletions(-)

diff --git a/event-schemas/examples/v1/m.room.member b/event-schemas/examples/v1/m.room.member
index 279eb484..a5ab79b5 100644
--- a/event-schemas/examples/v1/m.room.member
+++ b/event-schemas/examples/v1/m.room.member
@@ -4,11 +4,13 @@
     "membership": "join",
     "avatar_url": "mxc://localhost/SEsfnsuifSDFSSEF#auto",
     "displayname": "Alice Margatroid",
-    "token": "pc98",
-    "public_key": "abc123",
-    "key_validity_url": "https://magic.forest/verifykey",
-    "signature": "q1w2e3",
-    "sender": "@zun:zun.soft"
+    "third_party_invite": {
+        "token": "pc98",
+        "public_key": "abc123",
+        "key_validity_url": "https://magic.forest/verifykey",
+        "signature": "q1w2e3",
+        "sender": "@zun:zun.soft"
+    }
   },
   "state_key": "@alice:localhost",
   "origin_server_ts": 1431961217939,
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 47ce4d6c..56104328 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -22,26 +22,33 @@
                     "type": ["string", "null"],
                     "description": "The display name for this user, if any. This is added by the homeserver."
                 },
-        "token": {
-            "type": "string",
-            "description": "A token which must be correctly signed, in order to join the room."
-        },
-        "key_validity_url": {
-            "type": "string",
-            "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
-        },
-        "public_key": {
-            "type": "string",
-            "description": "A base64-encoded ed25519 key with which token must be signed."
-        },
-        "signature": {
-            "type": "string",
-            "description": "A base64-encoded signature of token with public_key."
-        },
-        "sender": {
-            "type": "string",
-            "description": "The matrix user ID of the user who send the invite which is being used."
-        }
+                "third_party_invite": {
+                    "type": "object",
+                    "title": "invite",
+                    "properties": {
+                        "token": {
+                            "type": "string",
+                            "description": "A token which must be correctly signed, in order to join the room."
+                        },
+                        "key_validity_url": {
+                            "type": "string",
+                            "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
+                        },
+                        "public_key": {
+                            "type": "string",
+                            "description": "A base64-encoded ed25519 key with which token must be signed."
+                        },
+                        "signature": {
+                            "type": "string",
+                            "description": "A base64-encoded signature of token with public_key."
+                        },
+                        "sender": {
+                            "type": "string",
+                            "description": "The matrix user ID of the user who send the invite which is being used."
+                        }
+                    },
+                    "required": ["token", "key_validity_url", "public_key", "signature", "sender"]
+                }
             },
             "required": ["membership"]
         },
diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index ae0d74e8..3f5a38a4 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -7,6 +7,22 @@ This module adds in support for inviting new members to a room where their
 Matrix user ID is not known, instead addressing them by a third party identifier
 such as an email address.
 
+There are two flows here; one if a Matrix user ID is known for the third party
+identifier, and one if not. Either way, the client calls /invite with the
+details of the third party identifier.
+
+The homeserver asks the identity server whether a Matrix user ID is known for
+that identifier. If it is, an invite is simply issued for that user.
+
+If it is not, the homeserver asks the identity server to record the details of
+the invitation, and to notify the client of this pending invitation if it gets
+a binding for this identifier in the future. The identity server returns to the
+homeserver a token, as well as its public key.
+
+If a client then tries to join the room in the future, it will be allowed to if
+it presents both the token, and a signature of that token from the identity
+server which can be verified with the public key.
+
 Events
 ------
 
@@ -17,23 +33,47 @@ Client behaviour
 
 A client asks a server to invite a user by their third party identifier.
 
-See the documentation for /invite for more information.
-
 Server behaviour
 ----------------
 
-All homeservers MUST verify that sign(``token``, ``public_key``) = ``signature``.
+All homeservers MUST verify that sig(``token``, ``public_key``) = ``signature``.
 
 If a client of the current homeserver is joining by an
 ``m.room.third_party_invite``, that homesever MUST validate that the public
-key used for signing is still valid, by checking ``key_validity_url``.
+key used for signing is still valid, by checking ``key_validity_url``. It does
+this by making an HTTP GET request to ``key_validity_url``, with the querystring
+?public_key=``public_key``. A JSON object will be returned, and the key is
+considered valid if the object contains a key named ``valid`` whose value is
+``true``. If this cannot be verified, the invitation must be rejected.
 
 If a homeserver is joining a room for the first time because of an
 ``m.room.third_party_invite``, the server which is already participating in the
 room MUST validate that the public key used for signing is still valid, by
-checking ``key_validity_url``.
+checking ``key_validity_url`` in the above described way.
 
 No other homeservers may reject the joining of the room on the basis of
 ``key_validity_url``, this is so that all homeservers have a consistent view of
-the room.
+the room. They may, however, indicate to their clients that a member's'
+membership is questionable.
+
+For example:
+
+If room R has two participating homeservers, H1, H2
+
+And user A on H1 invites a third party identifier to room R
+
+H1 asks the identity server for a binding to a Matrix user ID, and has none,
+so issues an ``m.room.third_party_invite`` event to the room.
+
+When the third party user validates their identity, they are told about the
+invite, and ask their homeserver, H3, to join the room.
+
+H3 validates that sign(``token``, ``public_key``) = ``signature``, and may check
+``key_validity_url``.
+
+H3 then asks H1 to join it to the room. H1 *must* validate that
+sign(``token``, ``public_key``) = ``signature`` *and* check ``key_validity_url``.
+
+Having validated these things, H1 writes the join event to the room, and H3
+begins participating in the room. H2 *must* accept this event.
 

From 458383585f6e58b588171d37584daccba49ff5bb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 2 Oct 2015 15:03:55 +0100
Subject: [PATCH 241/467] Stub sections

---
 specification/modules/instant_messaging.rst | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 43a06aa1..619226df 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -3,6 +3,9 @@ Instant Messaging
 
 .. _module:im:
 
+This module adds support for sending human-readable messages to a room. It also
+adds human-readable information to the room itself such as a room name and topic.
+
 Events
 ------
 
@@ -15,7 +18,7 @@ Events
 {{m_room_topic_event}}
 
 m.room.message msgtypes
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
 
 .. TODO-spec
    How a client should handle unknown message types.
@@ -27,3 +30,13 @@ outlined below.
 
 {{msgtype_events}}
 
+
+Client behaviour
+----------------
+
+Server behaviour
+----------------
+
+Security considerations
+-----------------------
+

From b29fdebec22ab12b7cc8c613a6018d9a96883744 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 2 Oct 2015 09:14:31 -0500
Subject: [PATCH 242/467] Give example of verification dance

---
 specification/modules/third_party_invites.rst | 23 ++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index 3f5a38a4..3365501e 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -41,7 +41,28 @@ All homeservers MUST verify that sig(``token``, ``public_key``) = ``signature``.
 If a client of the current homeserver is joining by an
 ``m.room.third_party_invite``, that homesever MUST validate that the public
 key used for signing is still valid, by checking ``key_validity_url``. It does
-this by making an HTTP GET request to ``key_validity_url``, with the querystring
+this by making an HTTP GET request to ``key_validity_url``:
+
+Schema::
+
+    => GET $key_validity_url?public_key=$public_key
+    <= HTTP/1.1 200 OK
+    {
+        "valid": true|false
+    }
+
+
+Example::
+
+    key_validity_url = https://identity.server/is_valid
+    public_key = ALJWLAFQfqffQHFqFfeqFUOEHf4AIHfefh4
+    => GET https://identity.server/is_valid?public_key=ALJWLAFQfqffQHFqFfeqFUOEHf4AIHfefh4
+    <= HTTP/1.1 200 OK
+    {
+        "valid": true
+    }
+
+with the querystring
 ?public_key=``public_key``. A JSON object will be returned, and the key is
 considered valid if the object contains a key named ``valid`` whose value is
 ``true``. If this cannot be verified, the invitation must be rejected.

From c8ddf1af09581ca5d54a1e21561d9adce2b3d99d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 2 Oct 2015 16:00:15 +0100
Subject: [PATCH 243/467] Add changelog

---
 CHANGELOG.rst | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 78aebddc..66f3d0a8 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -6,6 +6,29 @@
 .. in Jenkins. Comments like this are ignored by both RST and the templating
 .. system. Add the newest release notes beneath this comment.
 
+Specification changes in v0.2.0 (2015-10-02)
+============================================
+
+This update restructures the specification and begins to aggressively standardise
+on using Swagger and JSON Schema to document HTTP endpoints and Events
+respectively. It also introduces a number of new concepts to Matrix.
+
+Additions:
+ - New section: Feature Profiles.
+ - New section: Receipts.
+ - New section: Room history visibility.
+ - New event: ``m.receipt``.
+ - New event: ``m.room.canonical_alias``
+ - New event: ``m.room.history_visibility``
+ - New keys: ``/createRoom`` - allows room "presets" using ``preset`` and
+   ``initial_state`` keys.
+ - New endpoint: ``/tokenrefresh`` - Related to refreshing access tokens.
+
+Modifications:
+ - Convert most of the older HTTP APIs to Swagger documentation.
+ - Convert most of the older event formats to JSON Schema.
+ - Move selected client-server sections to be "Modules".
+
 Specification changes in v0.1.0 (2015-06-01)
 ============================================
 - First numbered release.

From bde003fe8618195122dd300b03913dc1b28710c3 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 2 Oct 2015 16:14:24 +0100
Subject: [PATCH 244/467] Split the rooms out into a separate top level key.

Divide the rooms into separate groups in preparation for adding tag
support.

Further subdivide the rooms into "joined/invited/archived" based the
membership of the user in the room because that membership affects what
events the user can view from the room. E.g only users that are joined
to a room may see the ephemeral events for the room.
---
 api/client-server/v2_alpha/sync.yaml | 114 +++++++++++++++++----------
 1 file changed, 74 insertions(+), 40 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 7daa33f5..7a003745 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -5,7 +5,7 @@ info:
 host: localhost:8008
 schemes:
   - https
-basePath: /_matrix/client/v2_alpha
+basePath: /_matrix/client/api/v2_alpha
 consumes:
   - application/json
 produces:
@@ -73,43 +73,71 @@ paths:
                   ``/sync`` request.
               rooms:
                 type: object
-                properties:
-                  roomlist:
-                    type: array
-                    description: |-
-                      A list of rooms that the client needs to update.
-                    items:
+                description: |-
+                  The updates to rooms, grouped according to the filter. By
+                  default there is a single ``default`` group.
+                additionalProperties:
+                    joined:
+                      type: array
+                      description: |-
+                        A list of room ids that the user is a member of that
+                        have updates.
+                      items:
+                        type: string.
+                    invited:
+                      type: array
+                      description: |-
+                        A list of room ids that the user has been invited to.
+                        The entries in the room_map will have a ``invite`` key
+                        containing the ``m.room.member`` event of the invite.
+                      items:
+                        type: string.
+                    archived:
+                      type: array
+                      description: |-
+                        A list of room ids that the user has left or been
+                        banned from. The entries in the room_map will have a
+                        ``state`` key and a ``timeline`` key. But will lack the
+                        ``emphemeral`` key.
+                      items:
+                        type: string.
+              room_map:
+                description: |-
+                  Map from room id to the updates for that room. The room ids
+                  are referenced from the ``rooms`` key.
+                type: object
+                additionalProperties:
+                  type: object
+                  properties:
+                    room_id:
+                      type: string
+                      description: |-
+                        The ID of the room.
+                    event_map:
                       type: object
-                      properties:
-                        room_id:
-                          type: string
-                          description: |-
-                            The ID of the room.
-                        event_map:
-                          type: object
-                          description: |-
-                            A map from event ID to events for this room. The events
-                            are referenced from the ``timeline`` and ``state`` keys
-                            for this room.
-                          additionalProperties:
-                            description: An event object.
-                            type: object
-                        state:
-                          description: |-
-                            The state updates for the room.
-                          allOf:
-                            - $ref: "definitions/room_event_batch.json"
-                        timeline:
-                          description: |-
-                            The timeline of messages and state changes in the room.
-                          allOf:
-                            - $ref: "definitions/timeline_batch.json"
-                        ephemeral:
-                          description: |-
-                            The ephemeral events in the room that aren't recorded
-                            in the timeline or state of the room. E.g. typing.
-                          allOf:
-                            - $ref: "definitions/event_batch.json"
+                      description: |-
+                        A map from event ID to events for this room. The events
+                        are referenced from the ``timeline`` and ``state`` keys
+                        for this room.
+                      additionalProperties:
+                        description: An event object.
+                        type: object
+                    state:
+                      description: |-
+                        The state updates for the room.
+                      allOf:
+                        - $ref: "definitions/room_event_batch.json"
+                    timeline:
+                      description: |-
+                        The timeline of messages and state changes in the room.
+                      allOf:
+                        - $ref: "definitions/timeline_batch.json"
+                    ephemeral:
+                      description: |-
+                        The ephemeral events in the room that aren't recorded
+                        in the timeline or state of the room. E.g. typing.
+                      allOf:
+                        - $ref: "definitions/event_batch.json"
               presence:
                 description: |-
                   The updates to the presence status of other users.
@@ -129,8 +157,14 @@ paths:
                   ]
                 },
                 "rooms": {
-                  "roomlist": [{
-                    "room_id": "!726s6s6q:example.com",
+                  "default": {
+                    "invited": [],
+                    "archived": [],
+                    "joined": ["!726s6s6q:example.com"]
+                  }
+                },
+                "room_map": {
+                  "!726s6s6q:example.com": {
                     "event_map": {
                       "$66697273743031:example.com": {
                         "sender": "@alice:example.com",
@@ -177,6 +211,6 @@ paths:
                         }
                       ]
                     }
-                  }]
+                  }
                 }
               }

From 047419f2ad9b9b72ed7de144d9c97bcc17e62030 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 2 Oct 2015 16:21:48 +0100
Subject: [PATCH 245/467] Remove the room_id since it is redundent. Remove text
 about invite event handling till we've thought about it some more.

---
 api/client-server/v2_alpha/sync.yaml | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 7a003745..6b7a1b2a 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -88,8 +88,6 @@ paths:
                       type: array
                       description: |-
                         A list of room ids that the user has been invited to.
-                        The entries in the room_map will have a ``invite`` key
-                        containing the ``m.room.member`` event of the invite.
                       items:
                         type: string.
                     archived:
@@ -109,10 +107,6 @@ paths:
                 additionalProperties:
                   type: object
                   properties:
-                    room_id:
-                      type: string
-                      description: |-
-                        The ID of the room.
                     event_map:
                       type: object
                       description: |-

From 417c5b53c4c6264f6b91a4bfbaaa4049da6934a0 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 2 Oct 2015 16:24:33 +0100
Subject: [PATCH 246/467] Remove duplicate sentences from merge conflicts

---
 specification/modules/presence.rst | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index cff4490b..79151c4f 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -21,11 +21,8 @@ A presence list is a list of user IDs whose presence the user wants to follow.
 To be added to this list, the user being added must be invited by the list owner
 who must accept the invitation.
  
-Each user has presence information associated with them. This encodes the
-"availability" of that user, suitable for display on other clients.
-This is transmitted as an ``m.presence`` event and is one of the few events
-which are sent *outside the context of a room*. Their presence state is
-represented by the ``presence`` key, which is an enum of one of the following:
+User's presence state is represented by the ``presence`` key, which is an enum
+of one of the following:
 
       - ``online`` : The default state when the user is connected to an event
         stream.
@@ -35,7 +32,6 @@ represented by the ``presence`` key, which is an enum of one of the following:
         explicitly suppressing their profile information from being sent.
       - ``free_for_chat`` : The user is generally willing to receive messages
         moreso than default.
- 
 
 Events
 ------

From 28fd1aa205b28adadee838547aed62e9f2a0a115 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 2 Oct 2015 16:34:06 +0100
Subject: [PATCH 247/467] Go into a bit more detail about feature profiles

---
 CHANGELOG.rst | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 66f3d0a8..6e3198ef 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -9,9 +9,14 @@
 Specification changes in v0.2.0 (2015-10-02)
 ============================================
 
-This update restructures the specification and begins to aggressively standardise
-on using Swagger and JSON Schema to document HTTP endpoints and Events
-respectively. It also introduces a number of new concepts to Matrix.
+This update fundamentally restructures the specification. The specification has
+been split into more digestible "modules" which each describe a particular
+function (e.g. typing). This was done in order make the specification easier to
+maintain and help define which modules are mandatory for certain types
+of clients. Types of clients along with the mandatory modules can be found in a
+new "Feature Profiles" section. This update also begins to aggressively
+standardise on using Swagger and JSON Schema to document HTTP endpoints and
+Events respectively. It also introduces a number of new concepts to Matrix.
 
 Additions:
  - New section: Feature Profiles.

From 8e5c832ff9fa8d8e674346e3776c08e659c306c9 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 5 Oct 2015 13:45:23 +0100
Subject: [PATCH 248/467] Flesh out more of the IM module

---
 .../schema/v1/m.room.message.feedback         |  2 +-
 specification/modules/instant_messaging.rst   | 59 +++++++++++++++++--
 2 files changed, 54 insertions(+), 7 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback
index 1bbfc1ba..2eaed999 100644
--- a/event-schemas/schema/v1/m.room.message.feedback
+++ b/event-schemas/schema/v1/m.room.message.feedback
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "MessageFeedback",
-    "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.",
+    "description": "**NB: Usage of this event is discouraged in favour of the** `receipts module`_. **Most clients will not recognise this event.** Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged.",
     "allOf": [{
         "$ref": "core-event-schema/room_event.json"
     }],
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 619226df..7934b80a 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -4,15 +4,37 @@ Instant Messaging
 .. _module:im:
 
 This module adds support for sending human-readable messages to a room. It also
-adds human-readable information to the room itself such as a room name and topic.
+adds support for associating human-readable information with the room itself
+such as a room name and topic.
 
 Events
 ------
 
 {{m_room_message_event}}
 
+
+.. admonition:: Rationale
+
+  Not all clients can display all message types. The most commonly supported
+  message type is raw text. As a result, we chose to have a textual fallback
+  display method represented by the ``body`` key. This means that even if the
+  client cannot display a particular ``msgtype``, they can still display
+  *something*, even if it is just plain text.
+
 {{m_room_message_feedback_event}}
 
+Usage of this event is discouraged for several reasons:
+ - The number of feedback events will grow very quickly with the number of users
+   in the room. This event provides no way to "batch" feedback, unlike the
+   `receipts module`_.
+ - Pairing feedback to messages gets complicated when paginating as feedback
+   arrives before the message it is acknowledging.
+ - There are no guarantees that the client has seen the event ID being
+   acknowledged.
+
+
+.. _`receipts module`: `module:receipts`_
+
 {{m_room_name_event}}
 
 {{m_room_topic_event}}
@@ -20,13 +42,10 @@ Events
 m.room.message msgtypes
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-.. TODO-spec
-   How a client should handle unknown message types.
-
-
 Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type
 of message being sent. Each type has their own required and optional keys, as
-outlined below.
+outlined below. If a client cannot display the given ``msgtype`` then it MUST
+display the fallback plain text ``body`` key instead.
 
 {{msgtype_events}}
 
@@ -34,9 +53,37 @@ outlined below.
 Client behaviour
 ----------------
 
+Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be
+uploaded using the `content repository module`_ where available. The
+resulting ``mxc://`` URI can then be used in the ``url`` key. The
+attachment SHOULD be uploaded *prior* to sending the event in order to stop a
+race condition where the recipient receives a link to a non-existent attachment.
+
+.. _`content repository module`: `module:content`_
+
+Recommendations when sending messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+- Advise using idempotent PUTs to send messages (and why)
+- Retries (exp backoff, giveup eventually allowing manual retry)
+- Queueing (bucket per room)
+
+Implementing local echo
+~~~~~~~~~~~~~~~~~~~~~~~
+- Local echo (document bug with races) - sending state. Pairing returned event ID.
+
+Displaying membership information with messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Member name linking (incl. pagination aka historical display names)
+
 Server behaviour
 ----------------
 
+- SHOULD enforce the body/msgtype keys are present (can 400 them)
+
 Security considerations
 -----------------------
 
+- Not encrypted, link to E2E module.
+- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc)
+

From 97cdd8106cec02871d58a54f1abcafe7b33481a2 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 5 Oct 2015 14:21:16 +0100
Subject: [PATCH 249/467] s|client/api/v2_alpha|/client/v2_alpha|

---
 api/client-server/v2_alpha/sync.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 6b7a1b2a..c502d926 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -5,7 +5,7 @@ info:
 host: localhost:8008
 schemes:
   - https
-basePath: /_matrix/client/api/v2_alpha
+basePath: /_matrix/client/v2_alpha
 consumes:
   - application/json
 produces:

From ed0f6d3ff38848250f830ad8ade02838d9c82f3e Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 5 Oct 2015 14:22:50 +0100
Subject: [PATCH 250/467] Typo

---
 api/client-server/v2_alpha/sync.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index c502d926..437af251 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -96,7 +96,7 @@ paths:
                         A list of room ids that the user has left or been
                         banned from. The entries in the room_map will have a
                         ``state`` key and a ``timeline`` key. But will lack the
-                        ``emphemeral`` key.
+                        ``ephemeral`` key.
                       items:
                         type: string.
               room_map:

From a7b808c5cd8ec423342a9844241a31bdc339eff8 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 5 Oct 2015 15:26:33 +0100
Subject: [PATCH 251/467] Add a invite_state key for holding the state that is
 bundled with an invite. This is kept separate from the actual state for the
 room as it may be derived from an incomplete, unverified copy of the state
 that was bundled with an invite event received over federation.

---
 api/client-server/v2_alpha/sync.yaml | 38 +++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 437af251..07078823 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -88,6 +88,9 @@ paths:
                       type: array
                       description: |-
                         A list of room ids that the user has been invited to.
+                        The entries in the room_map will have an
+                        ``invite_state`` key. But will lack the ``ephemeral``
+                        key, the ``timeline`` key and the ``state`` key.
                       items:
                         type: string.
                     archived:
@@ -132,6 +135,21 @@ paths:
                         in the timeline or state of the room. E.g. typing.
                       allOf:
                         - $ref: "definitions/event_batch.json"
+                    invite_state:
+                      description: |-
+                        The state of a room that the user has been invited to.
+                        These state events may only have the ``type``,
+                        ``state_key`` and ``content`` keys present.
+                        These events do not replace any state that the client
+                        already has for the room, for example if the client has
+                        archived the room. Instead the client should keep two
+                        separate copies of the state: the one from the
+                        ``invite_state`` and one from the archived ``state``.
+                        If the client joins the room then the current state
+                        will be given as a delta against the archived ``state``
+                        not the ``invite_state``.
+                      allOf:
+                        - $ref: "definitions/event_batch.json"
               presence:
                 description: |-
                   The updates to the presence status of other users.
@@ -152,7 +170,7 @@ paths:
                 },
                 "rooms": {
                   "default": {
-                    "invited": [],
+                    "invited": ["!696r7674:example.com"],
                     "archived": [],
                     "joined": ["!726s6s6q:example.com"]
                   }
@@ -205,6 +223,24 @@ paths:
                         }
                       ]
                     }
+                  },
+                  "!696r7674:example.com": {
+                    "invite_state": {
+                      "events": [
+                        {
+                          "sender": "@alice:example.com",
+                          "type": "m.room.name",
+                          "state_key": "@alice:example.com",
+                          "content": {"name": "My Room Name"}
+                        },
+                        {
+                          "sender": "@alice:example.com",
+                          "type": "m.room.name",
+                          "state_key": "",
+                          "content": {"membership": "invite"}
+                        }
+                      ]
+                    }
                   }
                 }
               }

From bbb5fa9398447f2196cf7c62ec457c7f717704ce Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 5 Oct 2015 15:36:01 +0100
Subject: [PATCH 252/467] Fix the state_key in the example v2 response, include
 the "sender" key in the list of keys included with invite_state events

---
 api/client-server/v2_alpha/sync.yaml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 07078823..49f42c6e 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -138,8 +138,8 @@ paths:
                     invite_state:
                       description: |-
                         The state of a room that the user has been invited to.
-                        These state events may only have the ``type``,
-                        ``state_key`` and ``content`` keys present.
+                        These state events may only have the `sender``,
+                        ``type``, ``state_key`` and ``content`` keys present.
                         These events do not replace any state that the client
                         already has for the room, for example if the client has
                         archived the room. Instead the client should keep two
@@ -230,13 +230,13 @@ paths:
                         {
                           "sender": "@alice:example.com",
                           "type": "m.room.name",
-                          "state_key": "@alice:example.com",
+                          "state_key": "",
                           "content": {"name": "My Room Name"}
                         },
                         {
                           "sender": "@alice:example.com",
                           "type": "m.room.name",
-                          "state_key": "",
+                          "state_key": "@bob:example.com",
                           "content": {"membership": "invite"}
                         }
                       ]

From 0e5b00feaaa9c337fd097e13780fa7d06158ac0d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 5 Oct 2015 16:10:25 +0100
Subject: [PATCH 253/467] Fix event type in v2 sync example

---
 api/client-server/v2_alpha/sync.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 49f42c6e..1b83d1d9 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -235,7 +235,7 @@ paths:
                         },
                         {
                           "sender": "@alice:example.com",
-                          "type": "m.room.name",
+                          "type": "m.room.member",
                           "state_key": "@bob:example.com",
                           "content": {"membership": "invite"}
                         }

From 0ce533d153e9656818dfdefa565b1356afa4b391 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 6 Oct 2015 13:42:03 +0100
Subject: [PATCH 254/467] Split the room_map into separate sections based on
 whether they are "joined/invited/archived".

Rename the room_map to rooms and remove the grouping indirection. When we
want groups then we can add them under a separate key, either at the
top-level or as part of the events themselves.
---
 api/client-server/v2_alpha/sync.yaml | 295 ++++++++++++++-------------
 1 file changed, 153 insertions(+), 142 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 1b83d1d9..7d60d639 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -74,82 +74,92 @@ paths:
               rooms:
                 type: object
                 description: |-
-                  The updates to rooms, grouped according to the filter. By
-                  default there is a single ``default`` group.
-                additionalProperties:
-                    joined:
-                      type: array
-                      description: |-
-                        A list of room ids that the user is a member of that
-                        have updates.
-                      items:
-                        type: string.
-                    invited:
-                      type: array
-                      description: |-
-                        A list of room ids that the user has been invited to.
-                        The entries in the room_map will have an
-                        ``invite_state`` key. But will lack the ``ephemeral``
-                        key, the ``timeline`` key and the ``state`` key.
-                      items:
-                        type: string.
-                    archived:
-                      type: array
-                      description: |-
-                        A list of room ids that the user has left or been
-                        banned from. The entries in the room_map will have a
-                        ``state`` key and a ``timeline`` key. But will lack the
-                        ``ephemeral`` key.
-                      items:
-                        type: string.
-              room_map:
-                description: |-
-                  Map from room id to the updates for that room. The room ids
-                  are referenced from the ``rooms`` key.
-                type: object
-                additionalProperties:
-                  type: object
-                  properties:
-                    event_map:
+                  Updates to rooms.
+                properties:
+                  joined:
+                    type: object
+                    additionalProperties:
+                      type: object
+                      properties:
+                        event_map:
+                          type: object
+                          description: |-
+                            A map from event ID to events for this room. The
+                            events are referenced from the ``timeline`` and
+                            ``state`` keys for this room.
+                          additionalProperties:
+                            description: An event object.
+                            type: object
+                        state:
+                          description: |-
+                            The state updates for the room.
+                          allOf:
+                            - $ref: "definitions/room_event_batch.json"
+                        timeline:
+                          description: |-
+                            The timeline of messages and state changes in the
+                            room.
+                          allOf:
+                            - $ref: "definitions/timeline_batch.json"
+                        ephemeral:
+                          description: |-
+                            The ephemeral events in the room that aren't
+                            recorded in the timeline or state of the room.
+                            e.g. typing.
+                          allOf:
+                            - $ref: "definitions/event_batch.json"
+                  invited:
+                    type: object
+                    description: |-
+                      The rooms that the user has been invited to.
+                    additionalProperties:
                       type: object
-                      description: |-
-                        A map from event ID to events for this room. The events
-                        are referenced from the ``timeline`` and ``state`` keys
-                        for this room.
-                      additionalProperties:
-                        description: An event object.
-                        type: object
-                    state:
-                      description: |-
-                        The state updates for the room.
-                      allOf:
-                        - $ref: "definitions/room_event_batch.json"
-                    timeline:
-                      description: |-
-                        The timeline of messages and state changes in the room.
-                      allOf:
-                        - $ref: "definitions/timeline_batch.json"
-                    ephemeral:
-                      description: |-
-                        The ephemeral events in the room that aren't recorded
-                        in the timeline or state of the room. E.g. typing.
-                      allOf:
-                        - $ref: "definitions/event_batch.json"
-                    invite_state:
-                      description: |-
-                        The state of a room that the user has been invited to.
-                        These state events may only have the `sender``,
-                        ``type``, ``state_key`` and ``content`` keys present.
-                        These events do not replace any state that the client
-                        already has for the room, for example if the client has
-                        archived the room. Instead the client should keep two
-                        separate copies of the state: the one from the
-                        ``invite_state`` and one from the archived ``state``.
-                        If the client joins the room then the current state
-                        will be given as a delta against the archived ``state``
-                        not the ``invite_state``.
-                      allOf:
-                        - $ref: "definitions/event_batch.json"
+                      properties:
+                        invite_state:
+                          description: |-
+                            The state of a room that the user has been invited
+                            to. These state events may only have the `sender``,
+                            ``type``, ``state_key`` and ``content`` keys
+                            present. These events do not replace any state that
+                            the client already has for the room, for example if
+                            the client has archived the room. Instead the
+                            client should keep two separate copies of the
+                            state: the one from the ``invite_state`` and one
+                            from the archived ``state``. If the client joins
+                            the room then the current state will be given as a
+                            delta against the archived ``state`` not the
+                            ``invite_state``.
+                          allOf:
+                            - $ref: "definitions/event_batch.json"
+                  archived:
+                    type: object
+                    description: |-
+                      The rooms that the user has left or been banned from. The
+                      entries in the room_map will lack an ``ephemeral`` key.
+                    additionalProperties:
+                      type: object
+                      properties:
+                        event_map:
+                          type: object
+                          description: |-
+                            A map from event ID to events for this room. The
+                            events are referenced from the ``timeline`` and
+                            ``state`` keys for this room.
+                          additionalProperties:
+                            description: An event object.
+                            type: object
+                        state:
+                          description: |-
+                            The state updates for the room up to the point when
+                            the user left.
+                          allOf:
+                            - $ref: "definitions/room_event_batch.json"
+                        timeline:
+                          description: |-
+                            The timeline of messages and state changes in the
+                            room up to the point when the user left.
+                          allOf:
+                            - $ref: "definitions/timeline_batch.json"
               presence:
                 description: |-
                   The updates to the presence status of other users.
@@ -169,78 +179,79 @@ paths:
                   ]
                 },
                 "rooms": {
-                  "default": {
-                    "invited": ["!696r7674:example.com"],
-                    "archived": [],
-                    "joined": ["!726s6s6q:example.com"]
-                  }
-                },
-                "room_map": {
-                  "!726s6s6q:example.com": {
-                    "event_map": {
-                      "$66697273743031:example.com": {
-                        "sender": "@alice:example.com",
-                        "type": "m.room.member",
-                        "state_key": "@alice:example.com",
-                        "content": {"membership": "join"},
-                        "origin_server_ts": 1417731086795
-                      },
-                      "$7365636s6r6432:example.com": {
-                        "sender": "@bob:example.com",
-                        "type": "m.room.member",
-                        "state_key": "@bob:example.com",
-                        "content": {"membership": "join"},
-                        "origin_server_ts": 1417731086795
-                      },
-                      "$74686972643033:example.com": {
-                        "sender": "@alice:example.com",
-                        "type": "m.room.message",
-                        "unsigned": {"age": "124524", "txn_id": "1234"},
-                        "content": {"body": "I am a fish", "msgtype": "m.text"},
-                        "origin_server_ts": 1417731086797
-                      }
-                    },
-                    "state": {
-                      "events": [
-                        "$66697273743031:example.com",
-                        "$7365636s6r6432:example.com"
-                      ]
-                    },
-                    "timeline": {
-                      "events": [
-                        "$7365636s6r6432:example.com",
-                        "$74686972643033:example.com"
-                      ],
-                      "limited": true,
-                      "prev_batch": "t34-23535_0_0"
-                    },
-                    "ephemeral": {
-                      "events": [
-                        {
-                          "room_id": "!726s6s6q:example.com",
-                          "type": "m.typing",
-                          "content": {"user_ids": ["@alice:example.com"]}
-                        }
-                      ]
-                    }
-                  },
-                  "!696r7674:example.com": {
-                    "invite_state": {
-                      "events": [
-                        {
+                  "joined": {
+                    "!726s6s6q:example.com": {
+                      "event_map": {
+                        "$66697273743031:example.com": {
                           "sender": "@alice:example.com",
-                          "type": "m.room.name",
-                          "state_key": "",
-                          "content": {"name": "My Room Name"}
+                          "type": "m.room.member",
+                          "state_key": "@alice:example.com",
+                          "content": {"membership": "join"},
+                          "origin_server_ts": 1417731086795
                         },
-                        {
-                          "sender": "@alice:example.com",
+                        "$7365636s6r6432:example.com": {
+                          "sender": "@bob:example.com",
                           "type": "m.room.member",
                           "state_key": "@bob:example.com",
-                          "content": {"membership": "invite"}
+                          "content": {"membership": "join"},
+                          "origin_server_ts": 1417731086795
+                        },
+                        "$74686972643033:example.com": {
+                          "sender": "@alice:example.com",
+                          "type": "m.room.message",
+                          "unsigned": {"age": "124524", "txn_id": "1234"},
+                          "content": {
+                            "body": "I am a fish",
+                            "msgtype": "m.text"
+                          },
+                          "origin_server_ts": 1417731086797
                         }
-                      ]
+                      },
+                      "state": {
+                        "events": [
+                          "$66697273743031:example.com",
+                          "$7365636s6r6432:example.com"
+                        ]
+                      },
+                      "timeline": {
+                        "events": [
+                          "$7365636s6r6432:example.com",
+                          "$74686972643033:example.com"
+                        ],
+                        "limited": true,
+                        "prev_batch": "t34-23535_0_0"
+                      },
+                      "ephemeral": {
+                        "events": [
+                          {
+                            "room_id": "!726s6s6q:example.com",
+                            "type": "m.typing",
+                            "content": {"user_ids": ["@alice:example.com"]}
+                          }
+                        ]
+                      }
                     }
-                  }
+                  },
+                  "invited": {
+                    "!696r7674:example.com": {
+                      "invite_state": {
+                        "events": [
+                          {
+                            "sender": "@alice:example.com",
+                            "type": "m.room.name",
+                            "state_key": "",
+                            "content": {"name": "My Room Name"}
+                          },
+                          {
+                            "sender": "@alice:example.com",
+                            "type": "m.room.member",
+                            "state_key": "@bob:example.com",
+                            "content": {"membership": "invite"}
+                          }
+                        ]
+                      }
+                    }
+                  },
+                  "archived": {}
                 }
               }

From 39faccb00d7d22a19f08724d47833dff2e53bec3 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 09:38:46 -0500
Subject: [PATCH 255/467] CS API is not just v1 surely

---
 specification/1-client_server_api.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index c63742d1..59e6b68e 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1,5 +1,5 @@
-Client-Server API v1
-====================
+Client-Server API
+=================
 
 Overview
 --------

From 4f606cf3a7098359f68bd292dab43450d7081602 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 6 Oct 2015 17:31:05 +0100
Subject: [PATCH 256/467] Add the fields required by the templating system to
 v2 sync

---
 api/client-server/v2_alpha/sync.yaml | 29 ++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 7d60d639..266f27bc 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -72,36 +72,49 @@ paths:
                   The batch token to supply in the ``since`` param of the next
                   ``/sync`` request.
               rooms:
+                title: Rooms
                 type: object
                 description: |-
                   Updates to rooms.
                 properties:
                   joined:
+                    title: Joined
                     type: object
                     additionalProperties:
+                      title: Joined Room
                       type: object
                       properties:
                         event_map:
+                          title: EventMap
                           type: object
                           description: |-
                             A map from event ID to events for this room. The
                             events are referenced from the ``timeline`` and
                             ``state`` keys for this room.
                           additionalProperties:
+                            title: Event
                             description: An event object.
                             type: object
+                            allOf:
+                              - $ref: "core-event-schema/event.json"
                         state:
+                          title: State
+                          type: object
                           description: |-
                             The state updates for the room.
                           allOf:
                             - $ref: "definitions/room_event_batch.json"
                         timeline:
+                          title: Timeline
+                          type: object
                           description: |-
                             The timeline of messages and state changes in the
                             room.
                           allOf:
                             - $ref: "definitions/timeline_batch.json"
                         ephemeral:
+                          title: Ephemeral
+                          type: object
                           description: |-
                             The ephemeral events in the room that aren't
                             recorded in the timeline or state of the room.
@@ -109,13 +122,17 @@ paths:
                           allOf:
                             - $ref: "definitions/event_batch.json"
                   invited:
+                    title: Invited
                     type: object
                     description: |-
                       The rooms that the user has been invited to.
                     additionalProperties:
+                      title: Invited Room
                       type: object
                       properties:
                         invite_state:
+                          title: InviteState
+                          type: object
                           description: |-
                             The state of a room that the user has been invited
                             to. These state events may only have the `sender``,
@@ -132,35 +149,47 @@ paths:
                           allOf:
                             - $ref: "definitions/event_batch.json"
                   archived:
+                    title: Archived
                     type: object
                     description: |-
                       The rooms that the user has left or been banned from. The
                       entries in the room_map will lack an ``ephemeral`` key.
                     additionalProperties:
+                      title: Archived Room
                       type: object
                       properties:
                         event_map:
+                          title: EventMap
                           type: object
                           description: |-
                             A map from event ID to events for this room. The
                             events are referenced from the ``timeline`` and
                             ``state`` keys for this room.
                           additionalProperties:
+                            title: Event
                             description: An event object.
                             type: object
+                            allOf:
+                              - $ref: "core-event-schema/event.json"
                         state:
+                          title: State
+                          type: object
                           description: |-
                             The state updates for the room up to the point when
                             the user left.
                           allOf:
                             - $ref: "definitions/room_event_batch.json"
                         timeline:
+                          title: Timeline
+                          type: object
                           description: |-
                             The timeline of messages and state changes in the
                             room up to the point when the user left.
                           allOf:
                             - $ref: "definitions/timeline_batch.json"
               presence:
+                title: Presence
+                type: object
                 description: |-
                   The updates to the presence status of other users.
                 allOf:

From 509d178d5892f8ed218b5919fcc36a77cb14109a Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 6 Oct 2015 17:40:28 +0100
Subject: [PATCH 257/467] Fix the receipts API to be valid swagger

---
 api/client-server/v2_alpha/definitions/error.yaml | 10 ++++++++++
 api/client-server/v2_alpha/receipts.yaml          |  3 ++-
 2 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 api/client-server/v2_alpha/definitions/error.yaml

diff --git a/api/client-server/v2_alpha/definitions/error.yaml b/api/client-server/v2_alpha/definitions/error.yaml
new file mode 100644
index 00000000..20312ae4
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/error.yaml
@@ -0,0 +1,10 @@
+type: object
+description: A Matrix-level Error
+properties:
+  errcode:
+    type: string
+    description: An error code.
+  error:
+    type: string
+    description: A human-readable error message.
+required: ["errcode"]
\ No newline at end of file
diff --git a/api/client-server/v2_alpha/receipts.yaml b/api/client-server/v2_alpha/receipts.yaml
index 4ef435b0..b60f72e6 100644
--- a/api/client-server/v2_alpha/receipts.yaml
+++ b/api/client-server/v2_alpha/receipts.yaml
@@ -47,6 +47,7 @@ paths:
           required: true
           x-example: "$1924376522eioj:example.com"
         - in: body
+          name: receipt
           description: |-
             Extra receipt information to attach to ``content`` if any. The
             server will automatically set the ``ts`` field.
@@ -65,4 +66,4 @@ paths:
         429:
           description: This request was rate-limited.
           schema:
-            "$ref": "definitions/error.yaml"
\ No newline at end of file
+            "$ref": "definitions/error.yaml"

From 034241eb20cb0958bccb78f939c21d036c95d63f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 6 Oct 2015 17:43:04 +0100
Subject: [PATCH 258/467] Missing symlink for the v2 swagger APIs

---
 api/client-server/v2_alpha/core-event-schema | 1 +
 1 file changed, 1 insertion(+)
 create mode 120000 api/client-server/v2_alpha/core-event-schema

diff --git a/api/client-server/v2_alpha/core-event-schema b/api/client-server/v2_alpha/core-event-schema
new file mode 120000
index 00000000..b020e6da
--- /dev/null
+++ b/api/client-server/v2_alpha/core-event-schema
@@ -0,0 +1 @@
+../../../event-schemas/schema/v1/core-event-schema
\ No newline at end of file

From 7ae25731134fbca4a53bc2a43b8a58ec6b7a2e52 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 13:36:59 -0500
Subject: [PATCH 259/467] fix barefaced lie regarding the passivity of ASes

---
 specification/3-application_service_api.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index bdde7789..a6e82137 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -14,7 +14,8 @@ irrespective of the underlying homeserver implementation.
 
 Passive Application Services
 ----------------------------
-"Passive" application services can only observe events from a given home server.
+"Passive" application services can only observe events from a given home server,
+and inject events into a room they are participating in.
 They cannot prevent events from being sent, nor can they modify the content of
 the event being sent. In order to observe events from a homeserver, the
 homeserver needs to be configured to pass certain types of traffic to the

From f759dc30d2896622a3df31460ed43f08c957b229 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 14:13:15 -0500
Subject: [PATCH 260/467] major refresh of the existing FAQ content (WIP)

---
 supporting-docs/_posts/2015-08-19-faq.md | 247 ++++++++++++++++-------
 1 file changed, 172 insertions(+), 75 deletions(-)

diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index 517b2ccf..8bfbb0f9 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -37,59 +37,92 @@ FAQ Content
 
 ##### What is Matrix?
 
-Matrix is an ambitious new open standard for open, distributed,
-real-time communication over IP. It defines interoperable Instant
-Messaging and VoIP, providing pragmatic HTTP APIs and open source
-reference implementations for creating and running your own real-time
-communication infrastructure.
+Matrix is an open standard for interoperable, decentralised,
+real-time communication over IP. It can be used to power Instant
+Messaging, VoIP/WebRTC signalling, Internet of Things communication - or anywhere
+you need a standard HTTP API for publishing and subscribing to 
+data whilst tracking the conversation history.
+
+|
+
+Matrix defines the standard, and provides open source reference implementations
+of Matrix-compatible Servers, Clients, Client SDKs and Application Services
+to help you create new communication solutions or extend the capabilities
+and reach of existing ones.
 
 ##### What is Matrix's Mission?
 
-Matrix.org's initial inspiration and goal has been to fix the problem of
-fragmented IP communications. But Matrix's real potential and ultimate
-mission is to be a generic messaging and data synchronisation system for
-the web - allowing people, services and devices to easily communicate
-with each other with full history.
+Matrix's initial goal is to fix the problem of fragmented IP communications:
+letting users message and call each other without having to care what app
+the other user is on - making it as easy as sending an email.
+
+|
+
+The longer term goal is for Matrix to act as a generic HTTP messaging and data
+synchronisation system for the whole web - allowing people, services and devices
+to easily communicate with each other, empowering users to own and control their
+data and select the services and vendors they want to use.
 
 ##### What does Matrix provide?
 
-Today Matrix provides a new [open standard](/docs/spec),
-[APIs](/docs/api) to integrate a service to the Matrix ecosystem and
-reference [open source
-implementations](http://github.com/matrix-org/synapse) of the standard.
+Matrix provides:
+
+-   [Open Standard](/docs/spec) HTTP APIs for transferring JSON messages (e.g. instant messages, WebRTC signalling), including:
+  -   [Client\<-\>Server API](/docs/spec#client-server-api-v1) - defines how Matrix compatible clients communicate with Matrix homeservers.
+  -   [Server\<-\>Server API](/docs/spec#federation-api) - defines how Matrix homeservers exchange messages and synchronise history with each other.
+  -   [Application Service API](/docs/spec/#application-service-api) - defines how to extend the functionality of Matrix with 'integrations' and bridge to other networks.
+  -   [Modules](/docs/spec/#modules) - specifies features that must be implemented by particular classes of clients.
+-   Open source reference implementations of:
+  -   Clients (Web (React), iOS, Android)
+  -   Client SDKs (Javascript, Web (React), iOS, Android)
+  -   Homeservers (Synapse)
+  -   Application Services (bridges to IRC, Slack, Skype, Lync and more...)
+-   The actual ecosystem and community of everyone running Matrix servers and services
+-   Loads of 3rd party contributions of clients, SDKs, servers and services.
+
+You can find the full list of Matrix enabled projects at https://matrix.org/blog/try-matrix-now.
 
 ##### What does this mean for users?
 
 The aim is to provide an analogous ecosystem to email - one where you
 can communicate with pretty much anyone, without caring what app or
 server they are using, using whichever app & server you chose to use,
-and a nice neutral identity system like an e-mail address or phone
+and use a neutral identity system like an e-mail address or phone
 number to discover people to talk to.
 
 ##### What kind of company is Matrix.org?
 
 Matrix is an open initiative which acts as a neutral custodian of the
 Matrix standard. It's not actually incorporated anywhere at the moment
-but we are looking at the best legal structure for the future. We are
-committed to keeping the Matrix project open.
+but we are looking at the best legal structure for the future (and as
+of October 2015 we have hopefully found one). Whatever the legal
+structure, we are committed to keeping the Matrix project open.
 
 ##### Who is funding Matrix.org?
 
-We have been given permission by our employers, Amdocs, to work on
-Matrix as an independent non-profit initiative.
+Most of the current core contributors to Matrix work at
+[Amdocs](http://amdocs.com), who have kindly given us permission to work
+on Matrix as an independent non-profit initiative.  Other contributors
+are funded by their own employers or donate their own time to the project.
 
 ##### Who is building Matrix?
 
-We're a team of ~10 people with decades of experience building custom
+The core team is ~10 people with extensive experience in building custom
 VoIP and Messaging apps for mobile network operators. Most of us have
-day jobs at Amdocs or OpenMarket, but we are supported by a mix of
-freelancers and volunteers.
+day jobs at [Amdocs](http://amdocs.com) or [OpenMarket](http://openmarket.com),
+but there are an increasing number of contributors from other companies and
+folks all over the internet.
 
 ##### Why are you called Matrix?
 
 We are called Matrix because we provide a structure in which all
 communication can be matrixed together.
 
+|
+
+No, it's nothing to do with the film (although you could go and build virtual
+worlds on top of Matrix if you wanted :)
+
 ##### Why have you released this as open source?
 
 We believe that any open standard defining interoperable communication
@@ -102,16 +135,18 @@ and build on top of it.
 ##### What do you mean by open?
 
 Matrix is an open standard, meaning that we have freely published the
-details for how to interface with Matrix compliant servers and clients,
-and encourage anyone and everyone to interface with them.  We also
+details for how to communicate interoperably using the Matrix set of
+HTTP APIs.  We encourage anyone and everyone to use the APIs and build
+their own projects which implement them and so benefit from
+interoperability with the rest of the Matrix ecosystem.  We also
 ensure the standard is not encumbered by any known patent licensing
 requirements.
 
-| 
+|
 
 Matrix is also open source, meaning that we have released the source
-code of the reference servers and clients to the public domain under the
-[Apache Licence v2](http://www.apache.org/licenses/LICENSE-2.0.html), to
+code of the reference servers, clients and services to the public domain
+under the [Apache Licence v2](http://www.apache.org/licenses/LICENSE-2.0.html), to
 encourage anyone and everyone to run their own servers and clients, and
 enhance them and contribute their enhancements as they see fit.
 
@@ -120,7 +155,7 @@ enhance them and contribute their enhancements as they see fit.
 Federation allows separate deployments of a communication service to
 communicate with each other - for instance a mail server run by Google
 federates with a mail server run by Microsoft when you send email from
-@gmail.com to @outlook.com.
+@gmail.com to @hotmail.com.
 
 |
 
@@ -145,16 +180,18 @@ VoIP and IM.
 ##### Why has no-one done this before?
 
 There have been several attempts before including SIP, XMPP and RCS.
- All of these have had some level of success, but
+ All of these have had some level of success, but many different
 technological/usability/economic factors have ended up limiting their
-success in providing true open federation.
+success.  Unfortunately, we've not ended up in a world where everyone
+has a SIP URI or Jabber ID on their business card, or a phone that
+actually uses RCS.
 
 ##### What is the difference between Matrix and IRC?
 
 We love IRC.  In fact, as of today the core Matrix team still uses it as
 our primary communication tool. Between us we've written IRCds, IRC bots
 and admined dreamforge, UnrealIRCd, epona, ircservices and several
-others.  That said, it has some limitations that Matrix seeks to improve
+others.  That said, it has some limitations that Matrix seeks to improve
 on:
 
 -   Text only
@@ -163,18 +200,24 @@ on:
 -   No presence support
 -   Fragmented identity model
 -   No open federation
--   No standard APIs, just an archaic TCP line protocol
+-   No standard APIs, just a rather limited TCP line protocol
 -   Non-standardised federation protocol
 -   No built-in end-to-end encryption
 -   Disruptive net-splits
 -   Non-extensible
 
+[IRCv3](http://ircv3.net) exists and is addressing some of issues;
+this is great news and we wish them well.  It's almost a contradiction
+in terms to get competitive between openly interoperable communication
+projects - we look forward to increasing the richness of Matrix\<-\>IRC
+bridges as the project progresses.
+
 ##### What is the difference between Matrix and XMPP?
 
 The Matrix team used XMPP (Openfire, ejabberd, spectrum, asmack,
 XMPPFramework) for IM before starting to experiment with open HTTP APIs
-as an alternative.   The main issues with XMPP that drove us in this
-direction were:
+as an alternative in around 2012.  The main issues with XMPP that
+drove us in this direction were:
 
 -   Not particularly web-friendly - you can't easily speak XMPP from a
     web browser. (N.B. Nowadays you have options like XMPP-FTW and
@@ -182,8 +225,8 @@ direction were:
 -   Single logical server per MUC is a single point of control and
     availability. (MUCs can be distributed over multiple physical
     servers, but they still sit behind a single logical JID and domain.
-    FMUC improves this with a similar approach to Matrix, but at time of
-    writing there are no open implementations.)
+    FMUC improves this with a similar approach to Matrix, but as of Oct
+    2015 there are no open source implementations.)
 -   History synchronisation is very much a second class citizen feature
 -   Stanzas aren't framed or reliably delivered without extensions. (See
     [wiki.xmpp.org](http://wiki.xmpp.org/web/Myths#Myth_Four:_XMPP_is_unreliable_without_a_bunch_of_extensions.)
@@ -191,7 +234,8 @@ direction were:
 -   Multiple device support is limited. (Apparently Carbons and MAM help
     with this)
 -   Baseline feature set is so minimal that fragmentation of features
-    between clients and servers is common
+    between clients and servers is common, especially as interoperability
+    profiles for features have fallen behind (as of July 2015)
 -   No strong identity system (i.e. no standard E2E PKI, unless you
     count X.509 certs, which [are
     questionable](http://www.thoughtcrime.org/blog/ssl-and-the-future-of-authenticity/))
@@ -199,31 +243,42 @@ direction were:
     bandwidth-efficient transports. (Since the time of writing a [Push
     XEP has appeared](http://xmpp.org/extensions/xep-0357.html), and
     [wiki.xmpp.org](http://wiki.xmpp.org/web/Myths#Myth_Three:_It.27s_too_bandwidth-inefficient_for_mobile.)
-    claims that XMPP runs fine over a 9600bps + 30s latency link.)
+    claims that XMPP runs "fine" over a 9600bps + 30s latency link.)
 
 The whole subject of XMPP vs Matrix seems to bring out the worst in
-people. We think of the standards as being quite different; at its core
+people.  Rather than fighting over which open interoperable communication
+standard works the best, we should just collaborate and bridge everything
+together.  The more federation and interoperability the better.
+
+|
+
+We think of Matrix and XMPP as being quite different; at its core
 Matrix can be thought of as an eventually consistent global JSON db with
 an HTTP API and pubsub semantics - whilst XMPP can be thought of as a
 message passing protocol. You can use them both to build chat systems;
 you can use them both to build pubsub systems; each comes with different
-tradeoffs. Matrix has a 'kitchen sink' baseline of functionality; XMPP
-has a deliberately minimal baseline set of functionality. If XMPP does
-what you need it to do, then we're genuinely happy for you :) Meanwhile,
-rather than competing, an XMPP Bridge like [Skaverat's xmpptrix
-beta](https://github.com/SkaveRat/xmpptrix) has potential to let both
-environments coexist and make the most of each other's benefits.
+tradeoffs. Matrix has a deliberately extensive 'kitchen sink' baseline
+of functionality; XMPP has a deliberately minimal baseline set of
+functionality. If XMPP does what you need it to do, then we're genuinely
+happy for you :) Meanwhile, rather than competing, an XMPP Bridge like
+[Skaverat's xmpptrix beta](https://github.com/SkaveRat/xmpptrix) or
+[jfred's matrix-xmpp-bridge](https://github.com/jfrederickson/matrix-xmpp-bridge)
+or Matrix.org's own [matrix-appservice-purple](https://github.com/matrix-org/matrix-appservice-purple)
+has potential to let both environments coexist and make the most of each
+other's benefits.
 
 ##### What is the difference between Matrix and PSYC?
 
 PSYC is a open federated messaging protocol loosely inspired by IRC.  In
 version 1 it was a standalone protocol, and in version 2 it is being
-reutilised as the messaging layer on top of GNUnet.  We honestly don't
+reutilised as a messaging layer on top of GNUnet.  We honestly don't
 know that much about it, beyond trying to use psycd as an XMPP\<-\>IRC
 bridge in 2010. Matrix differentiates primarily by providing simple HTTP
 APIs rather than the more exotic compact line protocol in PSYC v1 or the
-complicated GNUnet stack in v2.  Meanwhile, Matrix doesn't provide of
-the metadata protection guarantees that GNUnet/PSYC aims for.
+comprehensive GNUnet stack in v2, and Matrix focuses more on decentralised
+conversation history rather than just decentralised chat servers.
+On the other hand, Matrix doesn't provide the metadata protection
+guarantees that GNUnet/PSYC aims for.
 
 | 
 
@@ -233,29 +288,56 @@ PSYC's views on Matrix.
 ##### What is the difference between Matrix and Tox?
 
 Tox.im looks to be a very cool clone of Skype - a fully decentralised
-peer-to-peer network.  Matrix is deliberately not peer-to-peer; instead
-each user has a well-defined homeserver which stores his data and that
-he can depend upon.  Matrix provides HTTP APIs; Tox.im provides C APIs.
- We haven't actually played with Tox at all yet.
+peer-to-peer network.  Matrix is deliberately not a 'pure' peer-to-peer
+system; instead each user has a well-defined homeserver which stores
+his data and that he can depend upon.  Matrix provides HTTP APIs;
+Tox.im provides C APIs.  As of October 2015 Tox doesn't seem to have an
+answer yet for decentralised conversation history storage.
 
 ##### How does Matrix compare with something like Trillian or Pidgin?
 
 Trillian and Pidgin and similar aggregating IM clients merge all your IM
-activity into a single user experience.  However, your history and
+activity into a single app.  However, your history and
 identity is still fragmented across the networks.  People can't find you
 easily, and your history is fragmented (other than on the device
 where the client runs).   And rather than being able to chose the right
 app for the job when communicating with people, you are pushed towards
 relying on a specific aggregation app.
 
+Matrix lets you get the best of both worlds by linking to all the
+different networks (XMPP, AIM, ICQ, Lync, Skype etc) on the serverside,
+using bridges which can be run by anyone.  Matrix then provides a simple
+standard HTTP API to access any of these networks, and lets you choose
+whichever client you prefer (either as a 'native' Matrix client or using
+a non-Matrix client from one of the networks which has been bridged in).
+
 ##### What Matrix compliant apps are there?
 
-None yet, other than our examples.  It's early days :)
+Quite a few, ranging from the glossy mass-market to the geeky command-line.  There's even an emacs macro.  Check out [https://matrix.org/blog/try-matrix-now] for the current
+list of Matrix enabled projects.
 
-##### Why do you think existing apps will ever join this?
+##### What bridges to other networks are available?
+
+The number of 'bridges' which integrate existing communication networks into
+Matrix are growing on a daily basis - both written by the Matrix core team
+and contributed by the wider community.  The full list can be seen at
+https://matrix.org/blog/try-matrix-now, but the core ones as of Oct 2015 include:
+
+ * [matrix-appservice-irc](https://github.com/matrix-org/matrix-appservice-irc) - an increasingly comprehensive Matrix\<-\>IRC bridge
+ * [matrix-appservice-verto](https://github.com/matrix-org/matrix-appservice-verto) - links from Matrix to FreeSWITCH via the Verto protocol
+ * [matrix-appservice-slack](https://github.com/matrix-org/matrix-appservice-slack) - a basic bridge to Slack
+ * [matrix-appservice-purple](https://github.com/matrix-org/matrix-appservice-purple) - lets you access any of the 20+ protocols supported by
+                                                                                        [libpurple](https://developer.pidgin.im/wiki/WhatIsLibpurple), including
+                                                                                        Skype, Lync, 
+ * [matrix-appservice-bridge](https://github.com/matrix-org/matrix-appservice-bridge) - a general NodeJS framework for writing bridges
+
+Writing new bridges is incredibly fun and easy - see the [matrix-appservice-bridge HOWTO](https://github.com/matrix-org/matrix-appservice-bridge/blob/master/HOWTO.md)
+for an example of how to write a fully functional Slack bridge in less than 100 lines of code!
+
+##### Why do you think existing apps will ever join this officially?
 
 We firmly believe it is what is right for the consumer. As people begin
-to use interoperable communications tools service providers will see the
+to use interoperable communications tools, service providers will see the
 benefit and compete on quality of service, security and features rather
 than relying on locking people into their walled garden. We believe as
 soon as users see the availability and benefits of interoperable
@@ -264,9 +346,9 @@ services they will demand it.
 ##### Why aren't you doing this through the IETF? or W3C? or 3GPP?
 
 We do recognise the advantages of working with existing standards
-bodies. We have been focused on writing code and getting it out. As
-Matrix matures it may well be appropriate to work with an official
-standard body.
+bodies. We have been focused on writing code and getting it out, and the standard has been evolving rapidly since initial release in September 2014.
+Once the standard has matured sufficiently it may well be appropriate to work with an official
+standard body to maintain it going forwards.
 
 |
 
@@ -274,18 +356,19 @@ standard body.
 
 ##### How do I get an account and get started?
 
-The quickest way is to just jump to the demo webclient at
-[http://matrix.org/beta](http://matrix.org/beta) and sign up.  Please note that you can point the
-webclient to access any homeserver - you don't have to use matrix.org,
+The quickest way is to pick a client from https://matrix.org/blog/try-matrix-now and sign up.
+Please note that you can point clients to access any homeserver - you don't have to use matrix.org,
 although as of day 1, matrix.org is the only communal homeserver
 available.
 
 ##### What can I actually do with this?
 
-The demo webclient provides a simple chatroom interface to Matrix -
+A typical client provides a simple chatroom interface to Matrix -
 letting the user interact with users and rooms anywhere within the
 Matrix federation.  Text and image messages are supported, and basic
 voice-only VoIP calling via WebRTC is supported in one-to-one rooms.
+(As of October 2015, experimental multi-way calling is also available
+on Vector.im).
 
 ##### How do I connect my homeserver to the public Matrix network?
 
@@ -295,11 +378,21 @@ for details
 
 ##### How do I Matrix-enable my existing app?
 
-See the [Client-Server API
-HOWTO](http://matrix.org/docs/howtos/client-server.html) for an example
-of how to use Matrix's client-server API to let your app communicate
-with users via Matrix.  We're currently working out the best way to
-integrate your application's existing identity system with Matrix.
+If your app doesn't have any communication capability already, you'll want
+to use one of the Matrix client SDKs to add it in.  These come in different
+levels of sophistication - ranging from a simple HTTP API wrapper (like matrix-js-sdk, matrix-ios-sdk or matrix-android-sdk)
+through to reusable UI components (like matrix-react-sdk and matrix-ios-kit).  Pick
+the one for your platform, or a 3rd party one if none of the above work for you,
+and get plugging it in.  You'll probably also want to read the [Client-Server API
+HOWTO](http://matrix.org/docs/howtos/client-server.html) too.
+
+If you already have communication infrastructure set up (XMPP, custom HTTP, or whatever),
+then you'll want to run a bridge to expose it to the wider Matrix ecosystem.
+See [matrix-appservice-bridge HOWTO](https://github.com/matrix-org/matrix-appservice-bridge/blob/master/HOWTO.md) for a
+guide of how to write bridges using the matrix-appservice-bridge framework, or co-opt one
+from the list at https://matrix.org/blog/try-matrix-now.
+[Application Service API](/docs/spec/#application-service-api) gives the details of the API
+that bridges have to implement.
 
 ##### How can I write a client on Matrix?
 
@@ -308,16 +401,20 @@ HOWTO](http://matrix.org/docs/howtos/client-server.html) and the [API
 docs](/docs/api) and [the Spec](/docs/spec) for all the details you need
 to write a client.
 
-##### *How can I help out with this?*
+##### How can I help out with this?
+
+Come say hi on #matrix:matrix.org!  Install synapse and tell us how you get on. Critique the spec.  Write
+clients.  Write bridges!  Run bridges!  Nose around in [Jira](https://matrix.org/jira) and
+send us some pull requests on github to fix some bugs or add some features!  You could even
+try to write a homeserver (but be warned, Matrix's architecture makes homeservers orders of
+magnitude harder than clients or bridges.)
 
-Install synapse and tell us how you get on. Critique the spec.  Write
-clients.  Just come say hi on [\#matrix:matrix.org](/alpha) or the
-[mailing lists](/mailman/listinfo/matrix-users)!
+See [CONTRIBUTING.rst](http://github.com/matrix-org/synapse/tree/master/CONTRIBUTING.rst) for
+full details on how to contribute to the project.  All are welcome!
 
 ##### Where can I get support?
 
-[\#matrix:matrix.org](/alpha), \#matrix on irc.freenode.net or
-the [mailing lists](/mailman/listinfo/matrix-users) are your best bets.
+\#matrix:matrix.org aka \#matrix on irc.freenode.is your best bet.
 
 ##### How do I register custom matrix event types?
 
@@ -328,7 +425,7 @@ use the [mailing list](/mailman/listinfo/matrix-users) for now.
 ##### How mature is this?
 
 We started working on Matrix in July 2014, and have opened it to the
-public in September 2014.  It's early days, and under no circumstances
+public in September 2014.  It's early days, and under no circumstances
 should you use Matrix or Synapse for anything other than experimentation
 and learning at this point.  Obviously the spec and apps are maturing
 rapidly, but as of the time of writing APIs are not frozen and the apps

From 6e6585cebca0d8c6e72b6d0964c6e429f253284b Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Tue, 6 Oct 2015 20:21:49 +0100
Subject: [PATCH 261/467] FAQ draft

---
 supporting-docs/_posts/2015-08-19-faq.md | 72 ++++++++++++++++++------
 1 file changed, 55 insertions(+), 17 deletions(-)

diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index 517b2ccf..dc7dc723 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -31,17 +31,13 @@ FAQ Content
 
 
 * TOC
-{:toc}
+{:toc .toc}
 
 ### General
 
 ##### What is Matrix?
 
-Matrix is an ambitious new open standard for open, distributed,
-real-time communication over IP. It defines interoperable Instant
-Messaging and VoIP, providing pragmatic HTTP APIs and open source
-reference implementations for creating and running your own real-time
-communication infrastructure.
+Matrix is an ambitious new open standard for open, distributed, real-time communication over IP. It defines interoperable Instant Messaging and VoIP, providing pragmatic HTTP APIs and open source reference implementations for creating and running your own real-time communication infrastructure.
 
 ##### What is Matrix's Mission?
 
@@ -268,8 +264,6 @@ bodies. We have been focused on writing code and getting it out. As
 Matrix matures it may well be appropriate to work with an official
 standard body.
 
-|
-
 ### Quick Start
 
 ##### How do I get an account and get started?
@@ -308,7 +302,7 @@ HOWTO](http://matrix.org/docs/howtos/client-server.html) and the [API
 docs](/docs/api) and [the Spec](/docs/spec) for all the details you need
 to write a client.
 
-##### *How can I help out with this?*
+##### How can I help out with this?
 
 Install synapse and tell us how you get on. Critique the spec.  Write
 clients.  Just come say hi on [\#matrix:matrix.org](/alpha) or the
@@ -341,45 +335,75 @@ soon!* *In the mean time, don't hesitate to get in touch on
 [\#matrix:matrix.org](/alpha) or the [mailing
 lists](/mailman/listinfo/matrix-users)!*
 
-|
-
 ### Standard
 
 ##### What is a home server?
 
-##### What is an identity sever?
+**Users in Matrix use one or more clients to communicate. This could be a web client, a command line client, a mobile client - or multiple of these being used simultaneously by the same user. The clients are registered to a single homeserver, which stores the communication history and account information, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers.**
+
+##### What is an identity server?
+
+**Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix users. A Matrix "Identity" describes both the user ID and any other existing IDs from third party namespaces linked to their account.**
+
+|
+
+**Matrix users can link third-party IDs (3PIDs) such as email addresses, social network accounts and phone numbers to their user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping can then be used by Matrix users in order to discover the MXIDs of their contacts.**
+
+|
+
+**In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID and persist and replicate the mappings.
+Usage of an IS is not required in order for a client application to be part of the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs.**
 
 ##### Where do my conversations get stored?
 
+**Each homeserver stores the communication history and account information for all of its clients, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers and their clients. Clients typically communicate with each other by emitting events in the context of a virtual room. Room data is replicated across all of the homeservers *whose users are participating in a given room*.**
+
 ##### What is a 3PID?
 
+**Third-party IDs (3PIDs) are IDs from other systems or contexts, such as email addresses, social network accounts and phone numbers.**
+
 ##### How do you do VoIP calls on Matrix?
 
+**Voice (and video) over Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like any other event. This means that clients must only send call events to rooms with exactly two participants as currently the WebRTC standard is based around two-party communication. Group calls are on the to-do list, though!**
+
 ##### Can I log into other homeservers with my username and password?
 
+**Currently, no. We are looking at options for enabling multi-server access for users, and might add this feature at a later stage.**
+
 ##### Why Apache Licence?
 
+**The Apache Licence is a permissive licence. We want the Matrix protocol itself to be free and open, but people are free to create both free and commercial apps and services that uses the protocol. In our opinion, any Matrix-service only enhances the Matrix ecosystem.**
+
 ##### Can I write a Matrix homeserver?
 
+**Yes. Matrix is just a spec, so implementations of the spec are very welcome! It should be noted that at the moment, changes are still being made to the spec, so if you want to write a Matrix homeserver, it is strongly recommended that you chat to the Matrix.org devs in [\#matrix:matrix.org](https://matrix.org/beta/#/room/%23matrix:matrix.org) first! You can also read about the [Federation API here]( https://github.com/matrix-org/matrix-doc/blob/master/specification/30_server_server_api.rst).**
+
 ##### How secure is this?
 
+**Server-server traffic is mandatorily TLS from the outset.  Server-client traffic mandates transport layer encryption other than for tinkering. Clients that support PKI publish their public keys, and may encrypt
+and sign their messages for E2E security. "Well behaved" clients should participate in key escrow servers to allow private key submission for law enforcement. End-to-end encryption for group chat is supported through a per-room encryption key which is shared 1:1 between participating members.**
+
 ##### Why aren't you using an ORM layer like SqlAlchemy?  
 
-|
 
 ### APIs
 
 ##### How do I join the global Matrix federation?
 
+**You can download and run one of the available Matrix servers - please see [this guide](http://matrix.org/docs/guides/getting_involved.html#run) for details!**
+
 ##### What ports do I have to open up to join the global Matrix federation?
 
-|
+**That is up to you! Look at ["Setting up Federation"](https://github.com/matrix-org/synapse#setting-up-federation) in the Synapse readme file for details.**
 
 ### Reference Implementations
 
 ##### What is Matrix built on - and why?
 
-##### How do I run my own home server?
+
+##### How do I run my own homeserver?
+
+**Follow the instructions for the homeserver you want to run. If you want to run Synapse, the homeserver created by Matrix.org, follow [these instructions](https://github.com/matrix-org/synapse#synapse-installation).**
 
 ##### Can I run my own identity server?
 
@@ -396,18 +420,32 @@ addresses) to matrix IDs to aid user discovery**.
 
 ##### What is Synapse?
 
+**Synapse is a reference "homeserver" implementation of Matrix from the core development team at matrix.org, written in Python/Twisted for clarity and simplicity. It is intended to showcase the concept of Matrix and let folks see the spec in the context of a codebase and let you run your own homeserver and generally help bootstrap the ecosystem.**
+
 ##### Why is Synapse in Python/Twisted?
 
 ##### What are Synapse's platform requirements?
 
+
+
 ##### What are the Synapse webclient's requirements?
 
 ##### Where is the mobile app?
 
+**The mobile apps can be downloaded from the [Google Play store](https://play.google.com/store/apps/details?id=org.matrix.androidsdk.alpha)
+ and [Apple store](https://itunes.apple.com/gb/app/matrix-console/id970074271).**
+
+|
+
+**For the Android app, you can also install the latest development version 
+built by [Jenkins](http://www.matrix.org/jenkins/job/AndroidConsoleDevelop/lastBuild/artifact/console/build/outputs/apk/console-alpha-debug.apk).**
+
 ##### What decides the room member order on the webclient?
 
+**The members are ordered by their *last active time*.**
+
 |
 
-Any other question? Please contact us on
-[\#matrix:matrix.org](/alpha) or the [mailing
+Any other questions? Please contact us in
+[\#matrix:matrix.org](https://matrix.org/beta/#/room/%23matrix:matrix.org) or the [mailing
 lists](/mailman/listinfo/matrix-users)!

From 2621c22956e300c32ab1f274805121dbc42a3fff Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 13:36:59 -0500
Subject: [PATCH 262/467] fix barefaced lie regarding the passivity of ASes

---
 specification/3-application_service_api.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index bdde7789..a6e82137 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -14,7 +14,8 @@ irrespective of the underlying homeserver implementation.
 
 Passive Application Services
 ----------------------------
-"Passive" application services can only observe events from a given home server.
+"Passive" application services can only observe events from a given home server,
+and inject events into a room they are participating in.
 They cannot prevent events from being sent, nor can they modify the content of
 the event being sent. In order to observe events from a homeserver, the
 homeserver needs to be configured to pass certain types of traffic to the

From ed9028f0fcad707cc51446fcbc510d28d7fb9346 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 14:46:13 -0500
Subject: [PATCH 263/467] merge again...

---
 supporting-docs/_posts/2015-08-19-faq.md | 241 ++++++++++++++++-------
 1 file changed, 171 insertions(+), 70 deletions(-)

diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index dc7dc723..87e10f2c 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -37,55 +37,92 @@ FAQ Content
 
 ##### What is Matrix?
 
-Matrix is an ambitious new open standard for open, distributed, real-time communication over IP. It defines interoperable Instant Messaging and VoIP, providing pragmatic HTTP APIs and open source reference implementations for creating and running your own real-time communication infrastructure.
+Matrix is an open standard for interoperable, decentralised,
+real-time communication over IP. It can be used to power Instant
+Messaging, VoIP/WebRTC signalling, Internet of Things communication - or anywhere
+you need a standard HTTP API for publishing and subscribing to 
+data whilst tracking the conversation history.
+
+|
+
+Matrix defines the standard, and provides open source reference implementations
+of Matrix-compatible Servers, Clients, Client SDKs and Application Services
+to help you create new communication solutions or extend the capabilities
+and reach of existing ones.
 
 ##### What is Matrix's Mission?
 
-Matrix.org's initial inspiration and goal has been to fix the problem of
-fragmented IP communications. But Matrix's real potential and ultimate
-mission is to be a generic messaging and data synchronisation system for
-the web - allowing people, services and devices to easily communicate
-with each other with full history.
+Matrix's initial goal is to fix the problem of fragmented IP communications:
+letting users message and call each other without having to care what app
+the other user is on - making it as easy as sending an email.
+
+|
+
+The longer term goal is for Matrix to act as a generic HTTP messaging and data
+synchronisation system for the whole web - allowing people, services and devices
+to easily communicate with each other, empowering users to own and control their
+data and select the services and vendors they want to use.
 
 ##### What does Matrix provide?
 
-Today Matrix provides a new [open standard](/docs/spec),
-[APIs](/docs/api) to integrate a service to the Matrix ecosystem and
-reference [open source
-implementations](http://github.com/matrix-org/synapse) of the standard.
+Matrix provides:
+
+-   [Open Standard](/docs/spec) HTTP APIs for transferring JSON messages (e.g. instant messages, WebRTC signalling), including:
+  -   [Client\<-\>Server API](/docs/spec#client-server-api-v1) - defines how Matrix compatible clients communicate with Matrix homeservers.
+  -   [Server\<-\>Server API](/docs/spec#federation-api) - defines how Matrix homeservers exchange messages and synchronise history with each other.
+  -   [Application Service API](/docs/spec/#application-service-api) - defines how to extend the functionality of Matrix with 'integrations' and bridge to other networks.
+  -   [Modules](/docs/spec/#modules) - specifies features that must be implemented by particular classes of clients.
+-   Open source reference implementations of:
+  -   Clients (Web (React), iOS, Android)
+  -   Client SDKs (Javascript, Web (React), iOS, Android)
+  -   Homeservers (Synapse)
+  -   Application Services (bridges to IRC, Slack, Skype, Lync and more...)
+-   The actual ecosystem and community of everyone running Matrix servers and services
+-   Loads of 3rd party contributions of clients, SDKs, servers and services.
+
+You can find the full list of Matrix enabled projects at https://matrix.org/blog/try-matrix-now.
 
 ##### What does this mean for users?
 
 The aim is to provide an analogous ecosystem to email - one where you
 can communicate with pretty much anyone, without caring what app or
 server they are using, using whichever app & server you chose to use,
-and a nice neutral identity system like an e-mail address or phone
+and use a neutral identity system like an e-mail address or phone
 number to discover people to talk to.
 
 ##### What kind of company is Matrix.org?
 
 Matrix is an open initiative which acts as a neutral custodian of the
 Matrix standard. It's not actually incorporated anywhere at the moment
-but we are looking at the best legal structure for the future. We are
-committed to keeping the Matrix project open.
+but we are looking at the best legal structure for the future (and as
+of October 2015 we have hopefully found one). Whatever the legal
+structure, we are committed to keeping the Matrix project open.
 
 ##### Who is funding Matrix.org?
 
-We have been given permission by our employers, Amdocs, to work on
-Matrix as an independent non-profit initiative.
+Most of the current core contributors to Matrix work at
+[Amdocs](http://amdocs.com), who have kindly given us permission to work
+on Matrix as an independent non-profit initiative.  Other contributors
+are funded by their own employers or donate their own time to the project.
 
 ##### Who is building Matrix?
 
-We're a team of ~10 people with decades of experience building custom
+The core team is ~10 people with extensive experience in building custom
 VoIP and Messaging apps for mobile network operators. Most of us have
-day jobs at Amdocs or OpenMarket, but we are supported by a mix of
-freelancers and volunteers.
+day jobs at [Amdocs](http://amdocs.com) or [OpenMarket](http://openmarket.com),
+but there are an increasing number of contributors from other companies and
+folks all over the internet.
 
 ##### Why are you called Matrix?
 
 We are called Matrix because we provide a structure in which all
 communication can be matrixed together.
 
+|
+
+No, it's nothing to do with the film (although you could go and build virtual
+worlds on top of Matrix if you wanted :)
+
 ##### Why have you released this as open source?
 
 We believe that any open standard defining interoperable communication
@@ -98,16 +135,18 @@ and build on top of it.
 ##### What do you mean by open?
 
 Matrix is an open standard, meaning that we have freely published the
-details for how to interface with Matrix compliant servers and clients,
-and encourage anyone and everyone to interface with them.  We also
+details for how to communicate interoperably using the Matrix set of
+HTTP APIs.  We encourage anyone and everyone to use the APIs and build
+their own projects which implement them and so benefit from
+interoperability with the rest of the Matrix ecosystem.  We also
 ensure the standard is not encumbered by any known patent licensing
 requirements.
 
-| 
+|
 
 Matrix is also open source, meaning that we have released the source
-code of the reference servers and clients to the public domain under the
-[Apache Licence v2](http://www.apache.org/licenses/LICENSE-2.0.html), to
+code of the reference servers, clients and services to the public domain
+under the [Apache Licence v2](http://www.apache.org/licenses/LICENSE-2.0.html), to
 encourage anyone and everyone to run their own servers and clients, and
 enhance them and contribute their enhancements as they see fit.
 
@@ -116,7 +155,7 @@ enhance them and contribute their enhancements as they see fit.
 Federation allows separate deployments of a communication service to
 communicate with each other - for instance a mail server run by Google
 federates with a mail server run by Microsoft when you send email from
-@gmail.com to @outlook.com.
+@gmail.com to @hotmail.com.
 
 |
 
@@ -141,16 +180,18 @@ VoIP and IM.
 ##### Why has no-one done this before?
 
 There have been several attempts before including SIP, XMPP and RCS.
- All of these have had some level of success, but
+ All of these have had some level of success, but many different
 technological/usability/economic factors have ended up limiting their
-success in providing true open federation.
+success.  Unfortunately, we've not ended up in a world where everyone
+has a SIP URI or Jabber ID on their business card, or a phone that
+actually uses RCS.
 
 ##### What is the difference between Matrix and IRC?
 
 We love IRC.  In fact, as of today the core Matrix team still uses it as
 our primary communication tool. Between us we've written IRCds, IRC bots
 and admined dreamforge, UnrealIRCd, epona, ircservices and several
-others.  That said, it has some limitations that Matrix seeks to improve
+others.  That said, it has some limitations that Matrix seeks to improve
 on:
 
 -   Text only
@@ -159,18 +200,24 @@ on:
 -   No presence support
 -   Fragmented identity model
 -   No open federation
--   No standard APIs, just an archaic TCP line protocol
+-   No standard APIs, just a rather limited TCP line protocol
 -   Non-standardised federation protocol
 -   No built-in end-to-end encryption
 -   Disruptive net-splits
 -   Non-extensible
 
+[IRCv3](http://ircv3.net) exists and is addressing some of issues;
+this is great news and we wish them well.  It's almost a contradiction
+in terms to get competitive between openly interoperable communication
+projects - we look forward to increasing the richness of Matrix\<-\>IRC
+bridges as the project progresses.
+
 ##### What is the difference between Matrix and XMPP?
 
 The Matrix team used XMPP (Openfire, ejabberd, spectrum, asmack,
 XMPPFramework) for IM before starting to experiment with open HTTP APIs
-as an alternative.   The main issues with XMPP that drove us in this
-direction were:
+as an alternative in around 2012.  The main issues with XMPP that
+drove us in this direction were:
 
 -   Not particularly web-friendly - you can't easily speak XMPP from a
     web browser. (N.B. Nowadays you have options like XMPP-FTW and
@@ -178,8 +225,8 @@ direction were:
 -   Single logical server per MUC is a single point of control and
     availability. (MUCs can be distributed over multiple physical
     servers, but they still sit behind a single logical JID and domain.
-    FMUC improves this with a similar approach to Matrix, but at time of
-    writing there are no open implementations.)
+    FMUC improves this with a similar approach to Matrix, but as of Oct
+    2015 there are no open source implementations.)
 -   History synchronisation is very much a second class citizen feature
 -   Stanzas aren't framed or reliably delivered without extensions. (See
     [wiki.xmpp.org](http://wiki.xmpp.org/web/Myths#Myth_Four:_XMPP_is_unreliable_without_a_bunch_of_extensions.)
@@ -187,7 +234,8 @@ direction were:
 -   Multiple device support is limited. (Apparently Carbons and MAM help
     with this)
 -   Baseline feature set is so minimal that fragmentation of features
-    between clients and servers is common
+    between clients and servers is common, especially as interoperability
+    profiles for features have fallen behind (as of July 2015)
 -   No strong identity system (i.e. no standard E2E PKI, unless you
     count X.509 certs, which [are
     questionable](http://www.thoughtcrime.org/blog/ssl-and-the-future-of-authenticity/))
@@ -195,31 +243,42 @@ direction were:
     bandwidth-efficient transports. (Since the time of writing a [Push
     XEP has appeared](http://xmpp.org/extensions/xep-0357.html), and
     [wiki.xmpp.org](http://wiki.xmpp.org/web/Myths#Myth_Three:_It.27s_too_bandwidth-inefficient_for_mobile.)
-    claims that XMPP runs fine over a 9600bps + 30s latency link.)
+    claims that XMPP runs "fine" over a 9600bps + 30s latency link.)
 
 The whole subject of XMPP vs Matrix seems to bring out the worst in
-people. We think of the standards as being quite different; at its core
+people.  Rather than fighting over which open interoperable communication
+standard works the best, we should just collaborate and bridge everything
+together.  The more federation and interoperability the better.
+
+|
+
+We think of Matrix and XMPP as being quite different; at its core
 Matrix can be thought of as an eventually consistent global JSON db with
 an HTTP API and pubsub semantics - whilst XMPP can be thought of as a
 message passing protocol. You can use them both to build chat systems;
 you can use them both to build pubsub systems; each comes with different
-tradeoffs. Matrix has a 'kitchen sink' baseline of functionality; XMPP
-has a deliberately minimal baseline set of functionality. If XMPP does
-what you need it to do, then we're genuinely happy for you :) Meanwhile,
-rather than competing, an XMPP Bridge like [Skaverat's xmpptrix
-beta](https://github.com/SkaveRat/xmpptrix) has potential to let both
-environments coexist and make the most of each other's benefits.
+tradeoffs. Matrix has a deliberately extensive 'kitchen sink' baseline
+of functionality; XMPP has a deliberately minimal baseline set of
+functionality. If XMPP does what you need it to do, then we're genuinely
+happy for you :) Meanwhile, rather than competing, an XMPP Bridge like
+[Skaverat's xmpptrix beta](https://github.com/SkaveRat/xmpptrix) or
+[jfred's matrix-xmpp-bridge](https://github.com/jfrederickson/matrix-xmpp-bridge)
+or Matrix.org's own [matrix-appservice-purple](https://github.com/matrix-org/matrix-appservice-purple)
+has potential to let both environments coexist and make the most of each
+other's benefits.
 
 ##### What is the difference between Matrix and PSYC?
 
 PSYC is a open federated messaging protocol loosely inspired by IRC.  In
 version 1 it was a standalone protocol, and in version 2 it is being
-reutilised as the messaging layer on top of GNUnet.  We honestly don't
+reutilised as a messaging layer on top of GNUnet.  We honestly don't
 know that much about it, beyond trying to use psycd as an XMPP\<-\>IRC
 bridge in 2010. Matrix differentiates primarily by providing simple HTTP
 APIs rather than the more exotic compact line protocol in PSYC v1 or the
-complicated GNUnet stack in v2.  Meanwhile, Matrix doesn't provide of
-the metadata protection guarantees that GNUnet/PSYC aims for.
+comprehensive GNUnet stack in v2, and Matrix focuses more on decentralised
+conversation history rather than just decentralised chat servers.
+On the other hand, Matrix doesn't provide the metadata protection
+guarantees that GNUnet/PSYC aims for.
 
 | 
 
@@ -229,29 +288,56 @@ PSYC's views on Matrix.
 ##### What is the difference between Matrix and Tox?
 
 Tox.im looks to be a very cool clone of Skype - a fully decentralised
-peer-to-peer network.  Matrix is deliberately not peer-to-peer; instead
-each user has a well-defined homeserver which stores his data and that
-he can depend upon.  Matrix provides HTTP APIs; Tox.im provides C APIs.
- We haven't actually played with Tox at all yet.
+peer-to-peer network.  Matrix is deliberately not a 'pure' peer-to-peer
+system; instead each user has a well-defined homeserver which stores
+his data and that he can depend upon.  Matrix provides HTTP APIs;
+Tox.im provides C APIs.  As of October 2015 Tox doesn't seem to have an
+answer yet for decentralised conversation history storage.
 
 ##### How does Matrix compare with something like Trillian or Pidgin?
 
 Trillian and Pidgin and similar aggregating IM clients merge all your IM
-activity into a single user experience.  However, your history and
+activity into a single app.  However, your history and
 identity is still fragmented across the networks.  People can't find you
 easily, and your history is fragmented (other than on the device
 where the client runs).   And rather than being able to chose the right
 app for the job when communicating with people, you are pushed towards
 relying on a specific aggregation app.
 
+Matrix lets you get the best of both worlds by linking to all the
+different networks (XMPP, AIM, ICQ, Lync, Skype etc) on the serverside,
+using bridges which can be run by anyone.  Matrix then provides a simple
+standard HTTP API to access any of these networks, and lets you choose
+whichever client you prefer (either as a 'native' Matrix client or using
+a non-Matrix client from one of the networks which has been bridged in).
+
 ##### What Matrix compliant apps are there?
 
-None yet, other than our examples.  It's early days :)
+Quite a few, ranging from the glossy mass-market to the geeky command-line.  There's even an emacs macro.  Check out [https://matrix.org/blog/try-matrix-now] for the current
+list of Matrix enabled projects.
 
-##### Why do you think existing apps will ever join this?
+##### What bridges to other networks are available?
+
+The number of 'bridges' which integrate existing communication networks into
+Matrix are growing on a daily basis - both written by the Matrix core team
+and contributed by the wider community.  The full list can be seen at
+https://matrix.org/blog/try-matrix-now, but the core ones as of Oct 2015 include:
+
+ * [matrix-appservice-irc](https://github.com/matrix-org/matrix-appservice-irc) - an increasingly comprehensive Matrix\<-\>IRC bridge
+ * [matrix-appservice-verto](https://github.com/matrix-org/matrix-appservice-verto) - links from Matrix to FreeSWITCH via the Verto protocol
+ * [matrix-appservice-slack](https://github.com/matrix-org/matrix-appservice-slack) - a basic bridge to Slack
+ * [matrix-appservice-purple](https://github.com/matrix-org/matrix-appservice-purple) - lets you access any of the 20+ protocols supported by
+                                                                                        [libpurple](https://developer.pidgin.im/wiki/WhatIsLibpurple), including
+                                                                                        Skype, Lync, 
+ * [matrix-appservice-bridge](https://github.com/matrix-org/matrix-appservice-bridge) - a general NodeJS framework for writing bridges
+
+Writing new bridges is incredibly fun and easy - see the [matrix-appservice-bridge HOWTO](https://github.com/matrix-org/matrix-appservice-bridge/blob/master/HOWTO.md)
+for an example of how to write a fully functional Slack bridge in less than 100 lines of code!
+
+##### Why do you think existing apps will ever join this officially?
 
 We firmly believe it is what is right for the consumer. As people begin
-to use interoperable communications tools service providers will see the
+to use interoperable communications tools, service providers will see the
 benefit and compete on quality of service, security and features rather
 than relying on locking people into their walled garden. We believe as
 soon as users see the availability and benefits of interoperable
@@ -260,26 +346,27 @@ services they will demand it.
 ##### Why aren't you doing this through the IETF? or W3C? or 3GPP?
 
 We do recognise the advantages of working with existing standards
-bodies. We have been focused on writing code and getting it out. As
-Matrix matures it may well be appropriate to work with an official
-standard body.
+bodies. We have been focused on writing code and getting it out, and the standard has been evolving rapidly since initial release in September 2014.
+Once the standard has matured sufficiently it may well be appropriate to work with an official
+standard body to maintain it going forwards.
 
 ### Quick Start
 
 ##### How do I get an account and get started?
 
-The quickest way is to just jump to the demo webclient at
-[http://matrix.org/beta](http://matrix.org/beta) and sign up.  Please note that you can point the
-webclient to access any homeserver - you don't have to use matrix.org,
+The quickest way is to pick a client from https://matrix.org/blog/try-matrix-now and sign up.
+Please note that you can point clients to access any homeserver - you don't have to use matrix.org,
 although as of day 1, matrix.org is the only communal homeserver
 available.
 
 ##### What can I actually do with this?
 
-The demo webclient provides a simple chatroom interface to Matrix -
+A typical client provides a simple chatroom interface to Matrix -
 letting the user interact with users and rooms anywhere within the
 Matrix federation.  Text and image messages are supported, and basic
 voice-only VoIP calling via WebRTC is supported in one-to-one rooms.
+(As of October 2015, experimental multi-way calling is also available
+on Vector.im).
 
 ##### How do I connect my homeserver to the public Matrix network?
 
@@ -289,11 +376,21 @@ for details
 
 ##### How do I Matrix-enable my existing app?
 
-See the [Client-Server API
-HOWTO](http://matrix.org/docs/howtos/client-server.html) for an example
-of how to use Matrix's client-server API to let your app communicate
-with users via Matrix.  We're currently working out the best way to
-integrate your application's existing identity system with Matrix.
+If your app doesn't have any communication capability already, you'll want
+to use one of the Matrix client SDKs to add it in.  These come in different
+levels of sophistication - ranging from a simple HTTP API wrapper (like matrix-js-sdk, matrix-ios-sdk or matrix-android-sdk)
+through to reusable UI components (like matrix-react-sdk and matrix-ios-kit).  Pick
+the one for your platform, or a 3rd party one if none of the above work for you,
+and get plugging it in.  You'll probably also want to read the [Client-Server API
+HOWTO](http://matrix.org/docs/howtos/client-server.html) too.
+
+If you already have communication infrastructure set up (XMPP, custom HTTP, or whatever),
+then you'll want to run a bridge to expose it to the wider Matrix ecosystem.
+See [matrix-appservice-bridge HOWTO](https://github.com/matrix-org/matrix-appservice-bridge/blob/master/HOWTO.md) for a
+guide of how to write bridges using the matrix-appservice-bridge framework, or co-opt one
+from the list at https://matrix.org/blog/try-matrix-now.
+[Application Service API](/docs/spec/#application-service-api) gives the details of the API
+that bridges have to implement.
 
 ##### How can I write a client on Matrix?
 
@@ -304,14 +401,18 @@ to write a client.
 
 ##### How can I help out with this?
 
-Install synapse and tell us how you get on. Critique the spec.  Write
-clients.  Just come say hi on [\#matrix:matrix.org](/alpha) or the
-[mailing lists](/mailman/listinfo/matrix-users)!
+Come say hi on #matrix:matrix.org!  Install synapse and tell us how you get on. Critique the spec.  Write
+clients.  Write bridges!  Run bridges!  Nose around in [Jira](https://matrix.org/jira) and
+send us some pull requests on github to fix some bugs or add some features!  You could even
+try to write a homeserver (but be warned, Matrix's architecture makes homeservers orders of
+magnitude harder than clients or bridges.)
+
+See [CONTRIBUTING.rst](http://github.com/matrix-org/synapse/tree/master/CONTRIBUTING.rst) for
+full details on how to contribute to the project.  All are welcome!
 
 ##### Where can I get support?
 
-[\#matrix:matrix.org](/alpha), \#matrix on irc.freenode.net or
-the [mailing lists](/mailman/listinfo/matrix-users) are your best bets.
+\#matrix:matrix.org aka \#matrix on irc.freenode.is your best bet.
 
 ##### How do I register custom matrix event types?
 
@@ -322,7 +423,7 @@ use the [mailing list](/mailman/listinfo/matrix-users) for now.
 ##### How mature is this?
 
 We started working on Matrix in July 2014, and have opened it to the
-public in September 2014.  It's early days, and under no circumstances
+public in September 2014.  It's early days, and under no circumstances
 should you use Matrix or Synapse for anything other than experimentation
 and learning at this point.  Obviously the spec and apps are maturing
 rapidly, but as of the time of writing APIs are not frozen and the apps

From 91fb2ae723b3b0b17008f970dd8b1654ef64bfd9 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 16:38:29 -0500
Subject: [PATCH 264/467] edit oddvar's FAQ entries and add a bunch more
 questions

---
 supporting-docs/_posts/2015-08-19-faq.md | 117 ++++++++++++++++-------
 1 file changed, 81 insertions(+), 36 deletions(-)

diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index df981421..6bc6db92 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -328,7 +328,7 @@ https://matrix.org/blog/try-matrix-now, but the core ones as of Oct 2015 include
  * [matrix-appservice-slack](https://github.com/matrix-org/matrix-appservice-slack) - a basic bridge to Slack
  * [matrix-appservice-purple](https://github.com/matrix-org/matrix-appservice-purple) - lets you access any of the 20+ protocols supported by
                                                                                         [libpurple](https://developer.pidgin.im/wiki/WhatIsLibpurple), including
-                                                                                        Skype, Lync, 
+                                                                                        Skype, Lync, XMPP, etc)
  * [matrix-appservice-bridge](https://github.com/matrix-org/matrix-appservice-bridge) - a general NodeJS framework for writing bridges
 
 Writing new bridges is incredibly fun and easy - see the [matrix-appservice-bridge HOWTO](https://github.com/matrix-org/matrix-appservice-bridge/blob/master/HOWTO.md)
@@ -435,80 +435,95 @@ nice features before we really declare it ready for production.
 
 ### Standard
 
+##### What is a client?
+
+Users in Matrix use one or more clients to communicate. This could be any combination of a web client, a command line client, a mobile client - or embedded clients built into existing apps.  It could even be a piece of hardware (e.g. a drone) that is Matrix enabled.
+
+##### Can I use Matrix without installing a Matrix client?
+
+Sure. An ever increasing number of protocols are being bridged into Matrix, so if you use something like IRC on Freenode you may well be indirectly benefiting from Matrix, as others may be connected into the IRC channel via Matrix.
+
 ##### What is a home server?
 
-**Users in Matrix use one or more clients to communicate. This could be a web client, a command line client, a mobile client - or multiple of these being used simultaneously by the same user. The clients are registered to a single homeserver, which stores the communication history and account information, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers.**
+A user's clients connect to a single homeserver, which stores the communication history and account information for that user, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers.
 
 ##### What is an identity server?
 
-**Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix users. A Matrix "Identity" describes both the user ID and any other existing IDs from third party namespaces linked to their account.**
+Users in Matrix are identified internally via their matrix user ID (MXID). However, existing 3rd party ID (3PID) namespaces such as email addresses or phone numbers should be used publically to identify Matrix users, at least for invitation purposes.  A Matrix "Identity" describes both the user ID and any other existing IDs from third party namespaces linked to their account.
 
 |
 
-**Matrix users can link third-party IDs (3PIDs) such as email addresses, social network accounts and phone numbers to their user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping can then be used by Matrix users in order to discover the MXIDs of their contacts.**
+Matrix users can link third-party IDs (3PIDs) to their user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping can then be used by Matrix users in order to discover the MXIDs of their contacts.
 
 |
 
-**In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID and persist and replicate the mappings.
-Usage of an IS is not required in order for a client application to be part of the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs.**
+In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID and persist and replicate the mappings.
+Usage of an IS is not required in order for a client application to be part of the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs.
+
+|
+
+The precise architecture of identity servers is currently in flux and subject to change as we work to fully decentralise them.
 
 ##### Where do my conversations get stored?
 
-**Each homeserver stores the communication history and account information for all of its clients, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers and their clients. Clients typically communicate with each other by emitting events in the context of a virtual room. Room data is replicated across all of the homeservers *whose users are participating in a given room*.**
+Each homeserver stores the communication history and account information for all of its clients, and shares data with the wider Matrix ecosystem by synchronising communication history with other homeservers and their clients. Clients typically communicate with each other by emitting events in the context of a virtual room. Room data is replicated across all of the homeservers *whose users are participating in a given room*.
 
 ##### What is a 3PID?
 
-**Third-party IDs (3PIDs) are IDs from other systems or contexts, such as email addresses, social network accounts and phone numbers.**
+Third-party IDs (3PIDs) are IDs from other systems or contexts, such as email addresses, social network accounts and phone numbers.
 
 ##### How do you do VoIP calls on Matrix?
 
-**Voice (and video) over Matrix is built on the WebRTC 1.0 standard. Call events are sent to a room, like any other event. This means that clients must only send call events to rooms with exactly two participants as currently the WebRTC standard is based around two-party communication. Group calls are on the to-do list, though!**
+Voice (and video) over Matrix uses the WebRTC 1.0 standard to transfer call media (i.e. the actual voice and video traffic).  Matrix is used to signal the establishment and termination of the call by sending call events, like any other event.  Currently calls are only supported in rooms with exactly two participants - however, one of those participants may be a conferencing bridge.  We're looking at better ways to do group calling.
 
 ##### Can I log into other homeservers with my username and password?
 
-**Currently, no. We are looking at options for enabling multi-server access for users, and might add this feature at a later stage.**
+Currently, no. We are looking at options for decentralising or migrating user accounts between multiple servers, and might add this feature at a later stage.
 
 ##### Why Apache Licence?
 
-**The Apache Licence is a permissive licence. We want the Matrix protocol itself to be free and open, but people are free to create both free and commercial apps and services that uses the protocol. In our opinion, any Matrix-service only enhances the Matrix ecosystem.**
+The Apache Licence is a permissive licence. We want the Matrix protocol itself to be free and open, but people are free to create both free and commercial apps and services that uses the protocol. In our opinion, any Matrix-service only enhances the Matrix ecosystem.
 
 ##### Can I write a Matrix homeserver?
 
-**Yes. Matrix is just a spec, so implementations of the spec are very welcome! It should be noted that at the moment, changes are still being made to the spec, so if you want to write a Matrix homeserver, it is strongly recommended that you chat to the Matrix.org devs in [\#matrix:matrix.org](https://matrix.org/beta/#/room/%23matrix:matrix.org) first! You can also read about the [Federation API here]( https://github.com/matrix-org/matrix-doc/blob/master/specification/30_server_server_api.rst).**
+Yes. Matrix is just a spec, so implementations of the spec are very welcome! It should be noted that as of October 2015 the server, changes are still being made to the spec, so if you want to write a Matrix homeserver, it is strongly recommended that you chat to the Matrix.org devs in #matrix:matrix.org first! You can also read about the [Federation API here]( https://github.com/matrix-org/matrix-doc/blob/master/specification/30_server_server_api.rst).
 
 ##### How secure is this?
 
-**Server-server traffic is mandatorily TLS from the outset.  Server-client traffic mandates transport layer encryption other than for tinkering. Clients that support PKI publish their public keys, and may encrypt
-and sign their messages for E2E security. "Well behaved" clients should participate in key escrow servers to allow private key submission for law enforcement. End-to-end encryption for group chat is supported through a per-room encryption key which is shared 1:1 between participating members.**
+Server-server traffic is mandatorily TLS from the outset.  Server-client traffic mandates transport layer encryption other than for tinkering.  Servers maintain a public/private key pair, and sign the integrity of all messages in the context of the historical conversation, preventing tampering.  Server keys are distributed using a PERSPECTIVES-style system.
+
+End-to-end encryption is coming shortly to clients for both 1:1 and group chats to protect user data stored on servers, using the [Olm](https://matrix.org/git/olm) cryptographic ratchet implementation.  As of October 2015 this is blocked on implementing the necessary key distribution and fingerprint management.
 
-##### Why aren't you using an ORM layer like SqlAlchemy?  
+Privacy of metadata is not currently protected from server administrators - a malicious homeserver administrator can see who is talking to who and when, but not what is being said (once E2E encryption is enabled).  See [this presentation from Jardin Entropique](http://matrix.org/~matthew/2015-06-26%20Matrix%20Jardin%20Entropique.pdf) for a more comprehensive discussion of privacy in Matrix.
 
+### Implementations
 
-### APIs
+##### What is Synapse?
+
+Synapse is a reference "homeserver" implementation of Matrix from the core development team at matrix.org, written in Python 2/Twisted.  It is intended to showcase the concept of Matrix and let folks see the spec in the context of a codebase and let you run your own homeserver and generally help bootstrap the ecosystem.
 
 ##### How do I join the global Matrix federation?
 
-**You can download and run one of the available Matrix servers - please see [this guide](http://matrix.org/docs/guides/getting_involved.html#run) for details!**
+You can download and run one of the available Matrix servers - please see [this guide](http://matrix.org/docs/guides/getting_involved.html#run) for details!
 
 ##### What ports do I have to open up to join the global Matrix federation?
 
-**That is up to you! Look at ["Setting up Federation"](https://github.com/matrix-org/synapse#setting-up-federation) in the Synapse readme file for details.**
-
-### Reference Implementations
+We recommend servers use port 8448 for server\<-\>server HTTPS traffic.  Look at ["Setting up Federation"](https://github.com/matrix-org/synapse#setting-up-federation) in the Synapse readme file for details.
 
-##### What is Matrix built on - and why?
+Client\<-\>Server traffic can talk directly to Synapse via port 8448, but as by default Synapse creates a self-signed TLS certificate this can cause problems for clients which can't easily trust self-signed certificates (e.g. most web browsers).  Instead, you can proxy access to Synapse's HTTP listener on port 8008 via an existing HTTPS proxy with a valid certificate (e.g. an nginx listening on port 443), or you can point Synapse at a valid X.509 signed TLS certificate.  In future, Synapse will probably use letsencrypt to autogenerate valid certificates rather than self-signed ones during installation, simplifying this process enormously.
 
+You can also put Synapse entirely behind an existing TLS load balancer and not expose port 8448 at all.  In this situation, Synapse will need to be configured to share the same *public* TLS certificate as the load balancer (as Synapse uses the public certificate for identity in other areas too, and it has to match the certificate that other servers see when they connect).
 
 ##### How do I run my own homeserver?
 
-**Follow the instructions for the homeserver you want to run. If you want to run Synapse, the homeserver created by Matrix.org, follow [these instructions](https://github.com/matrix-org/synapse#synapse-installation).**
+Follow the instructions for the homeserver you want to run. If you want to run Synapse, the reference homeserver from Matrix.org, follow [these instructions](https://github.com/matrix-org/synapse#synapse-installation).
 
 ##### Can I run my own identity server?
 
 Yes - the reference implementation is
 [sydent](https://github.com/matrix-org/sydent) and you can run your own
-ID server cluster that tracks 3rd party to Matrix ID mappings. If you
-want your server to participate in the global replicated Matrix ID
+ID server cluster that tracks 3rd party to Matrix ID mappings.  This won't be very useful right now, though, and we don't recommend it.
+If you want your server to participate in the global replicated Matrix ID
 service then please get in touch with us. Meanwhile, we are looking at
 ways of decentralising the 'official' Matrix identity service so that
 identity servers are 100% decentralised and can openly federate with
@@ -516,31 +531,61 @@ each other. **N.B. that you can use Matrix without ever using the
 identity service - it exists only to map 3rd party IDs (e.g. email
 addresses) to matrix IDs to aid user discovery**.
 
-##### What is Synapse?
+##### What are Synapse's platform requirements?
 
-**Synapse is a reference "homeserver" implementation of Matrix from the core development team at matrix.org, written in Python/Twisted for clarity and simplicity. It is intended to showcase the concept of Matrix and let folks see the spec in the context of a codebase and let you run your own homeserver and generally help bootstrap the ecosystem.**
+Synapse will use as much RAM as you give it in order to cache conversations in RAM to avoid hitting the database.  For small deployments (<50 active users) around 512MB of RAM is probably okay.  You can configure the amount of RAM used by synapse with the event_cache_size config parameter - the more events in the cache, the more RAM required.  Synapse itself requires relatively little diskspace other than for logging (which as of October 2015 is quite verbose for debugging purposes), but as it caches the content of all the file attachments (images, videos etc) viewed by its users, you may need to size storage appropriately.  Synapse is currently effectively single threaded, and will never use more than 1 core.
 
-##### Why is Synapse in Python/Twisted?
+|
 
-##### What are Synapse's platform requirements?
+For better performance, one should back Synapse with a Postgres database rather than the default SQLite - see https://github.com/matrix-org/synapse/tree/master/README.rst#using-postgresql for details.
 
+##### Why is Synapse in Python/Twisted?
+
+This is because both provide a mature and well known event-driven async IO framework for writing serverside code.  Whilst this has been okay for our initial experimentation and proof of concept, it's likely that future homeserver work will be written in a more strongly typed language (e.g. Go). 
 
+##### Why aren't you using an ORM layer like SqlAlchemy in Synapse?
 
-##### What are the Synapse webclient's requirements?
+Synapse is *very* database dependent (as of Oct 2015; this is improving in the near future however), and we like having the flexibility to sculpt our own queries.
 
-##### Where is the mobile app?
+##### Where can I find a mobile app?
 
-**The mobile apps can be downloaded from the [Google Play store](https://play.google.com/store/apps/details?id=org.matrix.androidsdk.alpha)
- and [Apple store](https://itunes.apple.com/gb/app/matrix-console/id970074271).**
+The "Matrix Console" reference apps (ugly, geeky and powerful - intended for early adopter powerusers) can be downloaded from the [Google Play store](https://play.google.com/store/apps/details?id=org.matrix.androidsdk.alpha)
+ and [Apple store](https://itunes.apple.com/gb/app/matrix-console/id970074271).
 
 |
 
-**For the Android app, you can also install the latest development version 
-built by [Jenkins](http://www.matrix.org/jenkins/job/AndroidConsoleDevelop/lastBuild/artifact/console/build/outputs/apk/console-alpha-debug.apk).**
+For the Android app, you can also install the latest development version 
+built by [Jenkins](http://www.matrix.org/jenkins/job/AndroidConsoleDevelop/lastBuild/artifact/console/build/outputs/apk/console-alpha-debug.apk).
+
+##### Where can I find a web app?
 
-##### What decides the room member order on the webclient?
+As of Oct 2015, the best web app options are to use https://vector.im - a glossy web client written on top of matrix-react-sdk, or the original AngularJS based client at https://matrix.org/beta, which has serious performance problems and is not currently being maintained.  In future a "Matrix Console" reference web app built on matrix-react-sdk will be released by matrix.org to complement the mobile apps above.
+
+|
 
-**The members are ordered by their *last active time*.**
+### QUESTIONS TO BE ANSWERED!
+
+This FAQ is a constant work in progress - patches and pull requests are *very* welcome to help us improve it.  Some of the frequent questions where we need to write an answer include:
+
+ * How do I rename servers?
+ * How do I change the TLS key of my server?
+ * How do I maintain my synapse's DB (e.g. prune old conversations)?
+ * How do I maintain my synapse's content repository (e.g. prune old content)?
+ * What are redactions?
+ * Why is the spec so big, especially relative to the XMPP baseline spec? 
+ * How do I contribute to the spec? 
+ * What is the privacy policy on Matrix.org?
+ * How precisely does E2E work?
+ * How does Matrix actually work architecturally?
+ * What IOT use cases are there for Matrix?
+ * Why is are the Matrix reference implementations written in so many different languages?
+ * How does push work?
+ * What's on the roadmap?
+ * How can I use Matrix to talk on Freenode or other IRC networks?
+ * Where can I learn more about Matrix?  (link to PDFs of other presentations etc)
+ * WHy HTTP? Doesn't HTTP suck?
+ * Why don't you use websockets?
+ * \[your question goes here...\]
 
 |
 

From dffe9b3e4c2f0c12cae91346ad02b261183b5408 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Tue, 6 Oct 2015 21:33:32 -0500
Subject: [PATCH 265/467] more questions

---
 supporting-docs/_posts/2015-08-19-faq.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index 6bc6db92..402021e1 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -583,8 +583,9 @@ This FAQ is a constant work in progress - patches and pull requests are *very* w
  * What's on the roadmap?
  * How can I use Matrix to talk on Freenode or other IRC networks?
  * Where can I learn more about Matrix?  (link to PDFs of other presentations etc)
- * WHy HTTP? Doesn't HTTP suck?
+ * Why HTTP? Doesn't HTTP suck?
  * Why don't you use websockets?
+ * Why is synapse so resource intensive immediately after federating for the first time?
  * \[your question goes here...\]
 
 |

From 7805ca87dff536d98ec7de16f5dfc94d45680028 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 10:34:29 +0100
Subject: [PATCH 266/467] If build.py throws, actually print stdout. Also run
 jenkins.sh in verbose mode.

---
 jenkins.sh        | 2 +-
 scripts/gendoc.py | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/jenkins.sh b/jenkins.sh
index 0936de9d..0b217e58 100755
--- a/jenkins.sh
+++ b/jenkins.sh
@@ -4,6 +4,6 @@ set -ex
 
 (cd event-schemas/ && ./check_examples.py)
 (cd api && ./check_examples.py)
-(cd scripts && ./gendoc.py)
+(cd scripts && ./gendoc.py -v)
 (cd api && npm install && node validator.js -s "client-server/v1" && node validator.js -s "client-server/v2_alpha")
 (cd event-schemas/ && ./check.sh)
diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 1655d6f0..8db604c4 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -261,6 +261,7 @@ def run_through_template(input, set_verbose):
                 cwd="../templating"
             )
     except subprocess.CalledProcessError as e:
+        print e.output
         with open(tmpfile, 'r') as f:
             sys.stderr.write(f.read() + "\n")
         raise

From 91ca36509b3112a83680a97a0d672a428b4d7fef Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 11:51:49 +0100
Subject: [PATCH 267/467] Flesh out IM module

---
 specification/0-intro.rst                   |  2 +
 specification/modules/instant_messaging.rst | 89 ++++++++++++++++++---
 2 files changed, 80 insertions(+), 11 deletions(-)

diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index 445b32ef..7c27e0d7 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -434,6 +434,8 @@ Some requests have unique error codes:
 :``M_LOGIN_EMAIL_URL_NOT_YET``:
   Encountered when polling for an email link which has not been clicked yet.
 
+.. _sect:txn_ids:
+
 The C-S API typically uses ``HTTP POST`` to submit requests. This means these
 requests are not idempotent. The C-S API also allows ``HTTP PUT`` to make
 requests idempotent. In order to use a ``PUT``, paths should be suffixed with
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 7934b80a..426189b5 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -53,7 +53,14 @@ display the fallback plain text ``body`` key instead.
 Client behaviour
 ----------------
 
-Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be
+Clients SHOULD verify the structure of incoming events to ensure that the
+expected keys exist and that they are of the right type. Clients can discard
+malformed events or display a placeholder message to the user. Redacted
+``m.room.message`` events MUST be removed from the client. This can either be
+replaced with placeholder text (e.g. "[REDACTED]") or the redacted message can
+be removed entirely from the messages view.
+
+Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be
 uploaded using the `content repository module`_ where available. The
 resulting ``mxc://`` URI can then be used in the ``url`` key. The
 attachment SHOULD be uploaded *prior* to sending the event in order to stop a
@@ -63,27 +70,87 @@ race condition where the recipient receives a link to a non-existent attachment.
 
 Recommendations when sending messages
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Advise using idempotent PUTs to send messages (and why)
-- Retries (exp backoff, giveup eventually allowing manual retry)
-- Queueing (bucket per room)
 
-Implementing local echo
-~~~~~~~~~~~~~~~~~~~~~~~
-- Local echo (document bug with races) - sending state. Pairing returned event ID.
+Clients can send messages using ``POST`` or ``PUT`` requests. Clients SHOULD use
+``PUT`` requests with `transaction IDs`_ to make requests idempotent. This
+ensures that messages are sent exactly once even under poor network conditions.
+Clients SHOULD retry requests using an exponential-backoff algorithm for a
+certain amount of time T. It is recommended that T is no longer than 5 minutes.
+After this time, the client should stop retrying and mark the message as "unsent".
+Users should be able to manually resend unsent messages.
+
+Users may type several messages at once and send them all in quick succession.
+Clients SHOULD preserve the order in which they were sent by the user. This
+means that clients should wait for the response to the previous request before
+sending the next request. This can lead to head-of-line blocking. In order to
+reduce the impact of head-of-line blocking, clients should use a queue per room
+rather than a global queue, as ordering is only relevant within a single room
+rather than between rooms.
+
+.. _`transaction IDs`: `sect:txn_ids`_
+
+Local echo
+~~~~~~~~~~
+
+Messages SHOULD appear immediately in the message view when a user presses the
+"send" button. This should occur even if the message is still sending. This is
+referred to as "local echo". Clients SHOULD implement "local echo" of messages.
+ 
+Clients need to be able to pair up the "remote echo" from the server with the
+"local echo" to prevent duplicate messages being displayed. Ideally this pairing
+would occur transparently to the user: the UI would not flicker as it transitions
+from local to remote. Flickering cannot be fully avoided in version 1 of the
+client-server API. Two scenarios need to be considered:
+
+- The client sends a message and the remote echo arrives on the event stream
+  *after* the request to send the message completes.
+- The client sends a message and the remote echo arrives on the event stream
+  *before* the request to send the message completes.
+
+In the first scenario, the client will receive an event ID when the request to
+send the message completes. This ID can be used to identify the duplicate event
+when it arrives on the event stream. However, in the second scenario, the event
+arrives before the client has obtained an event ID. This makes it impossible to
+identify it as a duplicate event. This results in the client displaying the
+message twice for a fraction of a second before the the original request to send
+the message completes. Once it completes, the client can take remedial actions
+to remove the duplicate event by looking for duplicate event IDs. Version 2 of
+the client-server API resolves this by attaching the transaction ID of the
+sending request to the event itself.
+
 
 Displaying membership information with messages
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- Member name linking (incl. pagination aka historical display names)
+Clients may wish to show the display name and avatar URL of the room member who
+sent a message. This can be achieved by inspecting the ``m.room.member`` event
+for that user ID.
+
+When a user paginates the message history, clients may wish to show the
+**historical** display name and avatar URL for a room member. This is possible
+because older ``m.room.member`` events are returned when paginating. This can
+be implemented efficiently by keeping two sets of room state: old and current.
+As new events arrive and/or the user paginates back in time, these two sets of
+state diverge from each other. New events update the current state and paginated
+events update the old state. When paginated events are processed sequentially,
+the old state represents the state of the room *at the time the event was sent*.
+This can then be used to set the historical display name and avatar URL.
 
 Server behaviour
 ----------------
 
-- SHOULD enforce the body/msgtype keys are present (can 400 them)
+Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body``
+and ``msgtype`` keys by 400ing the request to send a message.
 
 Security considerations
 -----------------------
 
-- Not encrypted, link to E2E module.
-- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc)
+Messages sent using this module are not encrypted. Encryption can be layered
+over the top of this module: where the plaintext format is an ``m.room.message``
+conforming to this module. This can be achieved using the `E2E module`_.
+
+Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site
+Scripting (XSS) attacks. This includes room names and topics.
+
+.. _`E2E module`: `module:e2e`_
 

From 319ba8f40489ee8313df80fd2fc89109b5799bd8 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 14:00:16 +0100
Subject: [PATCH 268/467] Combine push files into one

We can fork it out again if it makes sense, but certainly 2/3 of the files
were very small and it just adds more complexity when refactoring.
---
 .../modules/{push_cs_api.rst => push.rst}     | 240 +++++++++++++++++-
 specification/modules/push_overview.rst       |  80 ------
 specification/modules/push_push_gw_api.rst    | 144 -----------
 specification/targets.yaml                    |   4 +-
 4 files changed, 233 insertions(+), 235 deletions(-)
 rename specification/modules/{push_cs_api.rst => push.rst} (64%)
 delete mode 100644 specification/modules/push_overview.rst
 delete mode 100644 specification/modules/push_push_gw_api.rst

diff --git a/specification/modules/push_cs_api.rst b/specification/modules/push.rst
similarity index 64%
rename from specification/modules/push_cs_api.rst
rename to specification/modules/push.rst
index c6301926..7a70c580 100644
--- a/specification/modules/push_cs_api.rst
+++ b/specification/modules/push.rst
@@ -1,3 +1,227 @@
+Push Notifications
+==================
+
+.. _module:push:
+
+Overview
+--------
+
+::
+
+                                   +--------------------+ +-------------------+
+                  Matrix HTTP      |                    | |                   |
+             Notification Protocol |   App Developer    | |   Device Vendor   |
+                                   |                    | |                   |
+           +-------------------+   | +----------------+ | | +---------------+ |
+           |                   |   | |                | | | |               | |
+           | Matrix Home Server+----->  Push Gateway  | +---> Push Provider | |
+           |                   |   | |                | | | |               | |
+           +-^-----------------+   | +----------------+ | | +----+----------+ |
+             |                     |                    | |      |            |
+    Matrix   |                     |                    | |      |            |
+ Client/Server API  +              |                    | |      |            |
+             |      |              +--------------------+ +-------------------+
+             |   +--+-+                                          |             
+             |   |    <------------------------------------------+             
+             +---+    |                                                        
+                 |    |          Provider Push Protocol                        
+                 +----+                                                        
+                                                                               
+         Mobile Device or Client                                               
+
+
+Matrix supports push notifications as a first class citizen. Home Servers send
+notifications of user events to user-configured HTTP endpoints. User may also
+configure a number of rules that determine what events generate notifications.
+These are all stored and managed by the users home server such that settings can
+be reused between client apps as appropriate.
+
+The above diagram shows the flow of push notifications being sent to a handset
+where push notifications are submitted via the handset vendor, such as Apple's
+APNS or Google's GCM. This happens as follows:
+
+ 1. The client app signs in to a Matrix Home Server
+ 2. The client app registers with its vendor's Push Notification provider and
+    obtains a routing token of some kind.
+ 3. The mobile app, uses the Matrix client/server API to add a 'pusher',
+    providing the URL of a specific Push Gateway which is configured for that
+    application. It also provides the routing token it has acquired from the
+    Push Notification Provider.
+ 4. The Home Server starts sending notification HTTP requests to the Push
+    Gateway using the supplied URL. The Push Gateway relays this notification to
+    the Push Notification Provider, passing the routing token along with any
+    necessary private credentials the provider requires to send push
+    notifications.
+ 5. The Push Notification provider sends the notification to the device.
+
+Nomenclature
+~~~~~~~~~~~~
+
+Pusher
+  A 'pusher' is an activity in the Home Server that manages the sending
+  of HTTP notifications for a single device of a single user.
+
+Push Rules
+  A push rule is a single rule, configured by a matrix user, that gives
+  instructions to the Home Server about whether an event should be notified
+  about and how given a set of conditions. Matrix clients allow the user to
+  configure these. They create and view them via the Client to Server REST API.
+
+Push Gateway
+  A push gateway is a server that receives HTTP event notifications from Home
+  Servers and passes them on to a different protocol such as APNS for iOS
+  devices or GCM for Android devices. Matrix.org provides a reference push
+  gateway, 'sygnal'. A client app tells a Home Server what push gateway
+  to send notifications to when it sets up a pusher.
+
+For information on the client-server API for setting pushers and push rules, see
+the Client Server API section. For more information on the format of HTTP
+notifications, see the HTTP Notification Protocol section.
+
+HTTP Notification Protocol
+--------------------------
+
+This describes the format used by "HTTP" pushers to send notifications of
+events.
+
+Notifications are sent as HTTP POST requests to the URL configured when the
+pusher is created, but Matrix strongly recommends that the path should be::
+
+  /_matrix/push/v1/notify
+
+The body of the POST request is a JSON dictionary. The format
+is as follows::
+
+  {
+    "notification": {
+      "id": "$3957tyerfgewrf384",
+      "room_id": "!slw48wfj34rtnrf:example.com",
+      "type": "m.room.message",
+      "sender": "@exampleuser:matrix.org",
+      "sender_display_name": "Major Tom",
+      "room_name": "Mission Control",
+      "room_alias": "#exampleroom:matrix.org",
+      "prio": "high",
+      "content": {
+        "msgtype": "m.text",
+        "body": "I'm floating in a most peculiar way."
+      }
+     },
+     "counts": {
+       "unread" : 2,
+       "missed_calls": 1
+     }
+     "devices": [
+       {
+         "app_id": "org.matrix.matrixConsole.ios",
+         "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/",
+         "pushkey_ts": 12345678,
+         "data" : {
+         },
+         "tweaks": {
+           "sound": "bing"
+          }
+        }
+      ]
+    }
+  }
+
+The contents of this dictionary are defined as follows:
+
+id
+  An identifier for this notification that may be used to detect duplicate
+  notification requests. This is not necessarily the ID of the event that
+  triggered the notification.
+room_id
+  The ID of the room in which this event occurred.
+type
+  The type of the event as in the event's 'type' field.
+sender
+  The sender of the event as in the corresponding event field.
+sender_display_name
+  The current display name of the sender in the room in which the event
+  occurred.
+room_name
+  The name of the room in which the event occurred.
+room_alias
+  An alias to display for the room in which the event occurred.
+prio
+  The priority of the notification. Acceptable values are 'high' or 'low. If
+  omitted, 'high' is assumed. This may be used by push gateways to deliver less
+  time-sensitive notifications in a way that will preserve battery power on
+  mobile devices.
+content
+  The 'content' field from the event, if present. If the event had no content
+  field, this field is omitted.
+counts
+  This is a dictionary of the current number of unacknowledged communications
+  for the recipient user. Counts whose value is zero are omitted.
+unread
+  The number of unread messages a user has across all of the rooms they are a
+  member of.
+missed_calls
+  The number of unacknowledged missed calls a user has across all rooms of
+  which they are a member.
+device
+  This is an array of devices that the notification should be sent to.
+app_id
+  The app_id given when the pusher was created.
+pushkey
+  The pushkey given when the pusher was created.
+pushkey_ts
+  The unix timestamp (in seconds) when the pushkey was last updated.
+data
+  A dictionary of additional pusher-specific data. For 'http' pushers, this is
+  the data dictionary passed in at pusher creation minus the 'url' key.
+tweaks
+  A dictionary of customisations made to the way this notification is to be
+  presented. These are added by push rules.
+sound
+  Sets the sound file that should be played. 'default' means that a default
+  sound should be played.
+
+And additional key is defined but only present on member events:
+
+user_is_target
+  This is true if the user receiving the notification is the subject of a member
+  event (i.e. the state_key of the member event is equal to the user's Matrix
+  ID).
+
+The recipient of an HTTP notification should respond with an HTTP 2xx response
+when the notification has been processed. If the endpoint returns an HTTP error
+code, the Home Server should retry for a reasonable amount of time with a
+reasonable back-off scheme.
+
+The endpoint should return a JSON dictionary as follows::
+
+  {
+    "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ]
+  }
+
+Whose keys are:
+
+rejected
+  A list of all pushkeys given in the notification request that are not valid.
+  These could have been rejected by an upstream gateway because they have
+  expired or have never been valid. Home Servers must cease sending notification
+  requests for these pushkeys and remove the associated pushers. It may not
+  necessarily be the notification in the request that failed: it could be that
+  a previous notification to the same pushkey failed.
+
+Push: Recommendations for APNS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+For sending APNS notifications, the exact format is flexible and up to the
+client app and its push gateway to agree on (since APNS requires that the sender
+have a private key owned by the app developer, each app must have its own push
+gateway). However, Matrix strongly recommends:
+
+ * That the APNS token be base64 encoded and used as the pushkey.
+ * That a different app_id be used for apps on the production and sandbox
+   APS environments.
+ * That APNS push gateways do not attempt to wait for errors from the APNS
+   gateway before returning and instead to store failures and return
+   'rejected' responses next time that pushkey is used.
+
 Pushers HTTP API
 ----------------
 
@@ -5,7 +229,7 @@ To receive any notification pokes at all, it is necessary to configure a
 'pusher' on the Home Server that you wish to receive notifications from. There
 is a single API endpoint for this::
 
-	POST $PREFIX/pushers/set
+  POST $PREFIX/pushers/set
 
 This takes a JSON object with the following keys:
 
@@ -51,7 +275,7 @@ be empty).
 
 
 Push Rules
-----------
+~~~~~~~~~~
 Home Servers have an interface to configure what events trigger notifications.
 This behaviour is configured through 'Push Rules'. Push Rules come in a variety
 of different kinds and each kind of rule has an associated priority. The
@@ -112,7 +336,7 @@ If no rules match an event, the Home Server should not notify for the message
 themselves are never alerted for.
 
 Predefined Rules
-----------------
+~~~~~~~~~~~~~~~~
 Matrix specifies the following rule IDs for server default rules. Home Servers
 may define rules as follows with the given IDs. If Home Servers provide rules
 with these IDs, their semantics should match those given below:
@@ -215,7 +439,7 @@ with these IDs, their semantics should match those given below:
     }
 
 Push Rules: Actions:
---------------------
+~~~~~~~~~~~~~~~~~~~~
 All rules have an associated list of 'actions'. An action affects if and how a
 notification is delivered for a matching event. This standard defines the
 following actions, although if Home servers wish to support more, they are free
@@ -241,7 +465,7 @@ represented as a dictionary with a key equal to their name and other keys as
 their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
 
 Push Rules: Actions: Tweaks
----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The ``set_tweak`` key action is used to add an entry to the 'tweaks' dictionary
 that is sent in the notification poke. The following tweaks are defined:
 
@@ -263,7 +487,7 @@ If a kind of tweak that a client understands is not specified in an action, the
 client may choose a sensible behaviour for the tweak.
 
 Push Rules: Conditions
-----------------------
+~~~~~~~~~~~~~~~~~~~~~~
 Override, Underride and Default rules have a list of 'conditions'. All
 conditions must hold true for an event in order for a rule to be applied to an
 event. A rule with no conditions always matches. Matrix specifies the following
@@ -300,7 +524,7 @@ using parameters named as described above. In the cases of room and sender
 rules, the rule_id of the rule determines its behaviour.
 
 Push Rules: API
----------------
+~~~~~~~~~~~~~~~
 Rules live under a hierarchy in the REST API that resembles::
 
   $PREFIX/pushrules/<scope>/<kind>/<rule_id>
@@ -411,7 +635,7 @@ Adding patch components to the request drills down into this structure to filter
 to only the requested set of rules.
 
 Enabling and Disabling Rules
-----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Rules can be enabled or disabled with a PUT operation to the 'enabled' component
 beneath the rule's URI with a content of 'true' or 'false'::
 
diff --git a/specification/modules/push_overview.rst b/specification/modules/push_overview.rst
deleted file mode 100644
index 46028283..00000000
--- a/specification/modules/push_overview.rst
+++ /dev/null
@@ -1,80 +0,0 @@
-Push Notifications
-==================
-
-.. _module:push:
-
-Overview
---------
-
-::
-
-                                   +--------------------+ +-------------------+
-                  Matrix HTTP      |                    | |                   |
-             Notification Protocol |   App Developer    | |   Device Vendor   |
-                                   |                    | |                   |
-           +-------------------+   | +----------------+ | | +---------------+ |
-           |                   |   | |                | | | |               | |
-           | Matrix Home Server+----->  Push Gateway  | +---> Push Provider | |
-           |                   |   | |                | | | |               | |
-           +-^-----------------+   | +----------------+ | | +----+----------+ |
-             |                     |                    | |      |            |
-    Matrix   |                     |                    | |      |            |
- Client/Server API  +              |                    | |      |            |
-             |      |              +--------------------+ +-------------------+
-             |   +--+-+                                          |             
-             |   |    <------------------------------------------+             
-             +---+    |                                                        
-                 |    |          Provider Push Protocol                        
-                 +----+                                                        
-                                                                               
-         Mobile Device or Client                                               
-
-
-Matrix supports push notifications as a first class citizen. Home Servers send
-notifications of user events to user-configured HTTP endpoints. User may also
-configure a number of rules that determine what events generate notifications.
-These are all stored and managed by the users home server such that settings can
-be reused between client apps as appropriate.
-
-The above diagram shows the flow of push notifications being sent to a handset
-where push notifications are submitted via the handset vendor, such as Apple's
-APNS or Google's GCM. This happens as follows:
-
- 1. The client app signs in to a Matrix Home Server
- 2. The client app registers with its vendor's Push Notification provider and
-    obtains a routing token of some kind.
- 3. The mobile app, uses the Matrix client/server API to add a 'pusher',
-    providing the URL of a specific Push Gateway which is configured for that
-    application. It also provides the routing token it has acquired from the
-    Push Notification Provider.
- 4. The Home Server starts sending notification HTTP requests to the Push
-    Gateway using the supplied URL. The Push Gateway relays this notification to
-    the Push Notification Provider, passing the routing token along with any
-    necessary private credentials the provider requires to send push
-    notifications.
- 5. The Push Notification provider sends the notification to the device.
-
-Nomenclature
-------------
-
-Pusher
-  A 'pusher' is an activity in the Home Server that manages the sending
-  of HTTP notifications for a single device of a single user.
-
-Push Rules
-  A push rule is a single rule, configured by a matrix user, that gives
-  instructions to the Home Server about whether an event should be notified
-  about and how given a set of conditions. Matrix clients allow the user to
-  configure these. They create and view them via the Client to Server REST API.
-
-Push Gateway
-  A push gateway is a server that receives HTTP event notifications from Home
-  Servers and passes them on to a different protocol such as APNS for iOS
-  devices or GCM for Android devices. Matrix.org provides a reference push
-  gateway, 'sygnal'. A client app tells a Home Server what push gateway
-  to send notifications to when it sets up a pusher.
-
-For information on the client-server API for setting pushers and push rules, see
-the Client Server API section. For more information on the format of HTTP
-notifications, see the HTTP Notification Protocol section.
-
diff --git a/specification/modules/push_push_gw_api.rst b/specification/modules/push_push_gw_api.rst
deleted file mode 100644
index c970e93e..00000000
--- a/specification/modules/push_push_gw_api.rst
+++ /dev/null
@@ -1,144 +0,0 @@
-HTTP Notification Protocol
---------------------------
-
-This describes the format used by "HTTP" pushers to send notifications of
-events.
-
-Notifications are sent as HTTP POST requests to the URL configured when the
-pusher is created, but Matrix strongly recommends that the path should be::
-
-  /_matrix/push/v1/notify
-
-The body of the POST request is a JSON dictionary. The format
-is as follows::
-
-  {
-    "notification": {
-      "id": "$3957tyerfgewrf384",
-      "room_id": "!slw48wfj34rtnrf:example.com",
-      "type": "m.room.message",
-      "sender": "@exampleuser:matrix.org",
-      "sender_display_name": "Major Tom",
-      "room_name": "Mission Control",
-      "room_alias": "#exampleroom:matrix.org",
-      "prio": "high",
-      "content": {
-        "msgtype": "m.text",
-        "body": "I'm floating in a most peculiar way."
-      }
-     },
-     "counts": {
-       "unread" : 2,
-       "missed_calls": 1
-     }
-     "devices": [
-       {
-         "app_id": "org.matrix.matrixConsole.ios",
-         "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/",
-         "pushkey_ts": 12345678,
-         "data" : {
-         },
-         "tweaks": {
-           "sound": "bing"
-          }
-        }
-      ]
-    }
-  }
-
-The contents of this dictionary are defined as follows:
-
-id
-  An identifier for this notification that may be used to detect duplicate
-  notification requests. This is not necessarily the ID of the event that
-  triggered the notification.
-room_id
-  The ID of the room in which this event occurred.
-type
-  The type of the event as in the event's 'type' field.
-sender
-  The sender of the event as in the corresponding event field.
-sender_display_name
-  The current display name of the sender in the room in which the event
-  occurred.
-room_name
-  The name of the room in which the event occurred.
-room_alias
-  An alias to display for the room in which the event occurred.
-prio
-  The priority of the notification. Acceptable values are 'high' or 'low. If
-  omitted, 'high' is assumed. This may be used by push gateways to deliver less
-  time-sensitive notifications in a way that will preserve battery power on
-  mobile devices.
-content
-  The 'content' field from the event, if present. If the event had no content
-  field, this field is omitted.
-counts
-  This is a dictionary of the current number of unacknowledged communications
-  for the recipient user. Counts whose value is zero are omitted.
-unread
-  The number of unread messages a user has across all of the rooms they are a
-  member of.
-missed_calls
-  The number of unacknowledged missed calls a user has across all rooms of
-  which they are a member.
-device
-  This is an array of devices that the notification should be sent to.
-app_id
-  The app_id given when the pusher was created.
-pushkey
-  The pushkey given when the pusher was created.
-pushkey_ts
-  The unix timestamp (in seconds) when the pushkey was last updated.
-data
-  A dictionary of additional pusher-specific data. For 'http' pushers, this is
-  the data dictionary passed in at pusher creation minus the 'url' key.
-tweaks
-  A dictionary of customisations made to the way this notification is to be
-  presented. These are added by push rules.
-sound
-  Sets the sound file that should be played. 'default' means that a default
-  sound should be played.
-
-And additional key is defined but only present on member events:
-
-user_is_target
-  This is true if the user receiving the notification is the subject of a member
-  event (i.e. the state_key of the member event is equal to the user's Matrix
-  ID).
-
-The recipient of an HTTP notification should respond with an HTTP 2xx response
-when the notification has been processed. If the endpoint returns an HTTP error
-code, the Home Server should retry for a reasonable amount of time with a
-reasonable back-off scheme.
-
-The endpoint should return a JSON dictionary as follows::
-
-  {
-    "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ]
-  }
-
-Whose keys are:
-
-rejected
-  A list of all pushkeys given in the notification request that are not valid.
-  These could have been rejected by an upstream gateway because they have
-  expired or have never been valid. Home Servers must cease sending notification
-  requests for these pushkeys and remove the associated pushers. It may not
-  necessarily be the notification in the request that failed: it could be that
-  a previous notification to the same pushkey failed.
-
-Push: Recommendations for APNS
-------------------------------
-For sending APNS notifications, the exact format is flexible and up to the
-client app and its push gateway to agree on (since APNS requires that the sender
-have a private key owned by the app developer, each app must have its own push
-gateway). However, Matrix strongly recommends:
-
- * That the APNS token be base64 encoded and used as the pushkey.
- * That a different app_id be used for apps on the production and sandbox
-   APS environments.
- * That APNS push gateways do not attempt to wait for errors from the APNS
-   gateway before returning and instead to store failures and return
-   'rejected' responses next time that pushkey is used.
-
diff --git a/specification/targets.yaml b/specification/targets.yaml
index d77bf8b5..8106b7cf 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -22,9 +22,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/content_repo.rst
     - modules/end_to_end_encryption.rst
     - modules/history_visibility.rst
-    - modules/push_overview.rst
-      # relative depth
-    - { 1: [modules/push_cs_api.rst , modules/push_push_gw_api.rst] }
+    - modules/push.rst
 
 title_styles: ["=", "-", "~", "+", "^"]
 

From a7400508c6aec74ad564e2e42fed8a6e6954932e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 14:07:36 +0100
Subject: [PATCH 269/467] Define module titles from the template

---
 specification/modules/push.rst | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index 7a70c580..d68e9ef0 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -3,9 +3,6 @@ Push Notifications
 
 .. _module:push:
 
-Overview
---------
-
 ::
 
                                    +--------------------+ +-------------------+
@@ -54,8 +51,7 @@ APNS or Google's GCM. This happens as follows:
     notifications.
  5. The Push Notification provider sends the notification to the device.
 
-Nomenclature
-~~~~~~~~~~~~
+Definitions for terms used in this section are below:
 
 Pusher
   A 'pusher' is an activity in the Home Server that manages the sending
@@ -222,7 +218,7 @@ gateway). However, Matrix strongly recommends:
    gateway before returning and instead to store failures and return
    'rejected' responses next time that pushkey is used.
 
-Pushers HTTP API
+Client behaviour
 ----------------
 
 To receive any notification pokes at all, it is necessary to configure a
@@ -641,4 +637,15 @@ beneath the rule's URI with a content of 'true' or 'false'::
 
   curl -X PUT -H "Content-Type: application/json" -d 'false' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org/enabled?access_token=123456"
 
+Server behaviour
+----------------
+
+- HS to sygnal?
+
+Security considerations
+-----------------------
 
+- Message content shouldn't be sent in the push itself as it will pass through
+  the Push Provider (Google/Apple). Instead, send a ping to tell the client to
+  sync.
+- HTTPS should be used on the Matrix HTTP Notification Protocol.

From d3ed517ebfc272d7aa0274882a5b647620ef72ac Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 14:14:15 +0100
Subject: [PATCH 270/467] Finish laying out rough sections

---
 specification/modules/push.rst | 292 +++++++++++++++++----------------
 1 file changed, 147 insertions(+), 145 deletions(-)

diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index d68e9ef0..addd92b6 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -74,150 +74,6 @@ For information on the client-server API for setting pushers and push rules, see
 the Client Server API section. For more information on the format of HTTP
 notifications, see the HTTP Notification Protocol section.
 
-HTTP Notification Protocol
---------------------------
-
-This describes the format used by "HTTP" pushers to send notifications of
-events.
-
-Notifications are sent as HTTP POST requests to the URL configured when the
-pusher is created, but Matrix strongly recommends that the path should be::
-
-  /_matrix/push/v1/notify
-
-The body of the POST request is a JSON dictionary. The format
-is as follows::
-
-  {
-    "notification": {
-      "id": "$3957tyerfgewrf384",
-      "room_id": "!slw48wfj34rtnrf:example.com",
-      "type": "m.room.message",
-      "sender": "@exampleuser:matrix.org",
-      "sender_display_name": "Major Tom",
-      "room_name": "Mission Control",
-      "room_alias": "#exampleroom:matrix.org",
-      "prio": "high",
-      "content": {
-        "msgtype": "m.text",
-        "body": "I'm floating in a most peculiar way."
-      }
-     },
-     "counts": {
-       "unread" : 2,
-       "missed_calls": 1
-     }
-     "devices": [
-       {
-         "app_id": "org.matrix.matrixConsole.ios",
-         "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/",
-         "pushkey_ts": 12345678,
-         "data" : {
-         },
-         "tweaks": {
-           "sound": "bing"
-          }
-        }
-      ]
-    }
-  }
-
-The contents of this dictionary are defined as follows:
-
-id
-  An identifier for this notification that may be used to detect duplicate
-  notification requests. This is not necessarily the ID of the event that
-  triggered the notification.
-room_id
-  The ID of the room in which this event occurred.
-type
-  The type of the event as in the event's 'type' field.
-sender
-  The sender of the event as in the corresponding event field.
-sender_display_name
-  The current display name of the sender in the room in which the event
-  occurred.
-room_name
-  The name of the room in which the event occurred.
-room_alias
-  An alias to display for the room in which the event occurred.
-prio
-  The priority of the notification. Acceptable values are 'high' or 'low. If
-  omitted, 'high' is assumed. This may be used by push gateways to deliver less
-  time-sensitive notifications in a way that will preserve battery power on
-  mobile devices.
-content
-  The 'content' field from the event, if present. If the event had no content
-  field, this field is omitted.
-counts
-  This is a dictionary of the current number of unacknowledged communications
-  for the recipient user. Counts whose value is zero are omitted.
-unread
-  The number of unread messages a user has across all of the rooms they are a
-  member of.
-missed_calls
-  The number of unacknowledged missed calls a user has across all rooms of
-  which they are a member.
-device
-  This is an array of devices that the notification should be sent to.
-app_id
-  The app_id given when the pusher was created.
-pushkey
-  The pushkey given when the pusher was created.
-pushkey_ts
-  The unix timestamp (in seconds) when the pushkey was last updated.
-data
-  A dictionary of additional pusher-specific data. For 'http' pushers, this is
-  the data dictionary passed in at pusher creation minus the 'url' key.
-tweaks
-  A dictionary of customisations made to the way this notification is to be
-  presented. These are added by push rules.
-sound
-  Sets the sound file that should be played. 'default' means that a default
-  sound should be played.
-
-And additional key is defined but only present on member events:
-
-user_is_target
-  This is true if the user receiving the notification is the subject of a member
-  event (i.e. the state_key of the member event is equal to the user's Matrix
-  ID).
-
-The recipient of an HTTP notification should respond with an HTTP 2xx response
-when the notification has been processed. If the endpoint returns an HTTP error
-code, the Home Server should retry for a reasonable amount of time with a
-reasonable back-off scheme.
-
-The endpoint should return a JSON dictionary as follows::
-
-  {
-    "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ]
-  }
-
-Whose keys are:
-
-rejected
-  A list of all pushkeys given in the notification request that are not valid.
-  These could have been rejected by an upstream gateway because they have
-  expired or have never been valid. Home Servers must cease sending notification
-  requests for these pushkeys and remove the associated pushers. It may not
-  necessarily be the notification in the request that failed: it could be that
-  a previous notification to the same pushkey failed.
-
-Push: Recommendations for APNS
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For sending APNS notifications, the exact format is flexible and up to the
-client app and its push gateway to agree on (since APNS requires that the sender
-have a private key owned by the app developer, each app must have its own push
-gateway). However, Matrix strongly recommends:
-
- * That the APNS token be base64 encoded and used as the pushkey.
- * That a different app_id be used for apps on the production and sandbox
-   APS environments.
- * That APNS push gateways do not attempt to wait for errors from the APNS
-   gateway before returning and instead to store failures and return
-   'rejected' responses next time that pushkey is used.
-
 Client behaviour
 ----------------
 
@@ -640,7 +496,152 @@ beneath the rule's URI with a content of 'true' or 'false'::
 Server behaviour
 ----------------
 
-- HS to sygnal?
+HTTP Notification Protocol
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This describes the format used by "HTTP" pushers to send notifications of
+events.
+
+Notifications are sent as HTTP POST requests to the URL configured when the
+pusher is created, but Matrix strongly recommends that the path should be::
+
+  /_matrix/push/v1/notify
+
+The body of the POST request is a JSON dictionary. The format
+is as follows::
+
+  {
+    "notification": {
+      "id": "$3957tyerfgewrf384",
+      "room_id": "!slw48wfj34rtnrf:example.com",
+      "type": "m.room.message",
+      "sender": "@exampleuser:matrix.org",
+      "sender_display_name": "Major Tom",
+      "room_name": "Mission Control",
+      "room_alias": "#exampleroom:matrix.org",
+      "prio": "high",
+      "content": {
+        "msgtype": "m.text",
+        "body": "I'm floating in a most peculiar way."
+      }
+     },
+     "counts": {
+       "unread" : 2,
+       "missed_calls": 1
+     }
+     "devices": [
+       {
+         "app_id": "org.matrix.matrixConsole.ios",
+         "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/",
+         "pushkey_ts": 12345678,
+         "data" : {
+         },
+         "tweaks": {
+           "sound": "bing"
+          }
+        }
+      ]
+    }
+  }
+
+The contents of this dictionary are defined as follows:
+
+id
+  An identifier for this notification that may be used to detect duplicate
+  notification requests. This is not necessarily the ID of the event that
+  triggered the notification.
+room_id
+  The ID of the room in which this event occurred.
+type
+  The type of the event as in the event's 'type' field.
+sender
+  The sender of the event as in the corresponding event field.
+sender_display_name
+  The current display name of the sender in the room in which the event
+  occurred.
+room_name
+  The name of the room in which the event occurred.
+room_alias
+  An alias to display for the room in which the event occurred.
+prio
+  The priority of the notification. Acceptable values are 'high' or 'low. If
+  omitted, 'high' is assumed. This may be used by push gateways to deliver less
+  time-sensitive notifications in a way that will preserve battery power on
+  mobile devices.
+content
+  The 'content' field from the event, if present. If the event had no content
+  field, this field is omitted.
+counts
+  This is a dictionary of the current number of unacknowledged communications
+  for the recipient user. Counts whose value is zero are omitted.
+unread
+  The number of unread messages a user has across all of the rooms they are a
+  member of.
+missed_calls
+  The number of unacknowledged missed calls a user has across all rooms of
+  which they are a member.
+device
+  This is an array of devices that the notification should be sent to.
+app_id
+  The app_id given when the pusher was created.
+pushkey
+  The pushkey given when the pusher was created.
+pushkey_ts
+  The unix timestamp (in seconds) when the pushkey was last updated.
+data
+  A dictionary of additional pusher-specific data. For 'http' pushers, this is
+  the data dictionary passed in at pusher creation minus the 'url' key.
+tweaks
+  A dictionary of customisations made to the way this notification is to be
+  presented. These are added by push rules.
+sound
+  Sets the sound file that should be played. 'default' means that a default
+  sound should be played.
+
+And additional key is defined but only present on member events:
+
+user_is_target
+  This is true if the user receiving the notification is the subject of a member
+  event (i.e. the state_key of the member event is equal to the user's Matrix
+  ID).
+
+The recipient of an HTTP notification should respond with an HTTP 2xx response
+when the notification has been processed. If the endpoint returns an HTTP error
+code, the Home Server should retry for a reasonable amount of time with a
+reasonable back-off scheme.
+
+The endpoint should return a JSON dictionary as follows::
+
+  {
+    "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ]
+  }
+
+Whose keys are:
+
+rejected
+  A list of all pushkeys given in the notification request that are not valid.
+  These could have been rejected by an upstream gateway because they have
+  expired or have never been valid. Home Servers must cease sending notification
+  requests for these pushkeys and remove the associated pushers. It may not
+  necessarily be the notification in the request that failed: it could be that
+  a previous notification to the same pushkey failed.
+
+Push: Recommendations for APNS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+For sending APNS notifications, the exact format is flexible and up to the
+client app and its push gateway to agree on (since APNS requires that the sender
+have a private key owned by the app developer, each app must have its own push
+gateway). However, Matrix strongly recommends:
+
+ * That the APNS token be base64 encoded and used as the pushkey.
+ * That a different app_id be used for apps on the production and sandbox
+   APS environments.
+ * That APNS push gateways do not attempt to wait for errors from the APNS
+   gateway before returning and instead to store failures and return
+   'rejected' responses next time that pushkey is used.
+
+Push Gateway behaviour
+----------------------
 
 Security considerations
 -----------------------
@@ -649,3 +650,4 @@ Security considerations
   the Push Provider (Google/Apple). Instead, send a ping to tell the client to
   sync.
 - HTTPS should be used on the Matrix HTTP Notification Protocol.
+

From c5edc60c4cf2d082384e4437a798464e6a380348 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 16:01:36 +0100
Subject: [PATCH 271/467] Add push YAML for pushers endpoint.

Also display "required" text on required JSON body request params. Also
increase the size of the request param column to support longer param names
present in the pushers API.
---
 api/client-server/v1/push.yaml                | 144 ++++++++++++++++++
 specification/modules/push.rst                |  47 +-----
 .../matrix_templates/templates/http-api.tmpl  |  18 +--
 templating/matrix_templates/units.py          |   8 +-
 4 files changed, 162 insertions(+), 55 deletions(-)
 create mode 100644 api/client-server/v1/push.yaml

diff --git a/api/client-server/v1/push.yaml b/api/client-server/v1/push.yaml
new file mode 100644
index 00000000..fd179fe6
--- /dev/null
+++ b/api/client-server/v1/push.yaml
@@ -0,0 +1,144 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Push API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/pushers/set":
+    post:
+      summary: Modify a pusher for this user on the homeserver.
+      description: |-
+        This endpoint allows the creation, modification and deletion of pushers
+        for this user ID. The behaviour of this endpoint varies depending on the
+        values in the JSON body.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: pusher
+          description: The pusher information
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "lang": "en",
+                "kind": "http",
+                "app_display_name": "Mat Rix",
+                "device_display_name": "iPhone 9",
+                "app_id": "com.example.app.ios",
+                "profile_tag": "4bea66906d0111e59d70feff819cdc9f",
+                "pushkey": "APA91bHPRgkF3JUikC4ENAHEeMrd41Zxv3hVZjC9KtT8OvPVGJ-hQMRKRrZuJAEcl7B338qju59zJMjw2DELjzEvxwYv7hH5Ynpc1ODQ0aT4U4OFEeco8ohsN5PjL1iC2dNtk2BAokeMCg2ZXKqpc8FXKmhX94kIxQ"
+                "data": {
+                  "url": "https://push-gateway.location.here"
+                }
+                "append": false
+              }
+            properties:
+              pushkey:
+                type: string
+                description: |-
+                  This is a unique identifier for this pusher. The value you
+                  should use for this is the routing or destination address
+                  information for the notification, for example, the APNS token
+                  for APNS or the Registration ID for GCM. If your notification
+                  client has no such concept, use any unique identifier.
+                  Max length, 512 bytes.
+              kind:
+                type: string
+                enum: ["http", null]
+                description: |-
+                  The kind of pusher to configure. 'http' makes a pusher that
+                  sends HTTP pokes. null deletes the pusher.
+              profile_tag:
+                type: string
+                description: |-
+                  This is a string that determines what set of device rules will
+                  be matched when evaluating push rules for this pusher. It is
+                  an arbitrary string. Multiple devices maybe use the same
+                  ``profile_tag``. It is advised that when an app's data is
+                  copied or restored to a different device, this value remain
+                  the same. Client apps should offer ways to change the
+                  ``profile_tag``, optionally copying rules from the old
+                  profile tag. Max length, 32 bytes.
+              app_id:
+                type: string
+                description: |-
+                  This is a reverse-DNS style identifier for the application.
+                  It is recommended that this end with the platform, such that
+                  different platform versions get different app identifiers.
+                  Max length, 64 chars.
+              app_display_name:
+                type: string
+                description: |-
+                  A string that will allow the user to identify what application
+                  owns this pusher.
+              device_display_name:
+                type: string
+                description: |-
+                  A string that will allow the user to identify what device owns
+                  this pusher.
+              lang:
+                type: string
+                description: |-
+                  The preferred language for receiving notifications (e.g. 'en'
+                  or 'en-US')
+              data:
+                type: object
+                description: |-
+                  A dictionary of information for the pusher implementation
+                  itself. If ``kind`` is ``http``, this should contain ``url``
+                  which is the URL to use to send notifications to.
+                properties:
+                  url:
+                    type: string
+                    description: |-
+                      Required if ``kind`` is ``http``. The URL to use to send
+                      notifications to.
+              append:
+                type: boolean
+                description: |-
+                  If true, the homeserver should add another pusher with the
+                  given pushkey and App ID in addition to any others with
+                  different user IDs. Otherwise, the Home Server must remove any
+                  other pushers with the same App ID and pushkey for different
+                  users. The default is ``false``.
+            required: ['profile_tag', 'kind', 'app_id', 'app_display_name',
+                'device_display_name', 'pushkey', 'lang', 'data']
+      responses:
+        200:
+          description: The pusher was set.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
+        400:
+          description: One or more of the pusher values were invalid.
+          examples:
+            application/json: |-
+              {
+                "error": "Missing parameters: lang, data",
+                "errcode": "M_MISSING_PARAM"
+              }
+          schema:
+            type: object
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index addd92b6..bd08dc9e 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -79,52 +79,9 @@ Client behaviour
 
 To receive any notification pokes at all, it is necessary to configure a
 'pusher' on the Home Server that you wish to receive notifications from. There
-is a single API endpoint for this::
-
-  POST $PREFIX/pushers/set
-
-This takes a JSON object with the following keys:
-
-pushkey
-  This is a unique identifier for this pusher. The value you should use for this
-  is the routing or destination address information for the notification, for
-  example, the APNS token for APNS or the Registration ID for GCM. If your
-  notification client has no such concept, use any unique identifier. Max length,
-  512 bytes.
-kind
-  The kind of pusher to configure. 'http' makes a pusher that sends HTTP pokes.
-  null deletes the pusher.
-profile_tag
-  This is a string that determines what set of device rules will be matched when
-  evaluating push rules for this pusher. It is an arbitrary string. Multiple
-  devices maybe use the same profile_tag. It is advised that when an app's
-  data is copied or restored to a different device, this value remain the same.
-  Client apps should offer ways to change the profile_tag, optionally copying
-  rules from the old profile tag. Max length, 32 bytes.
-app_id
-  appId is a reverse-DNS style identifier for the application. It is recommended
-  that this end with the platform, such that different platform versions get
-  different app identifiers. Max length, 64 chars.
-app_display_name
-  A string that will allow the user to identify what application owns this
-  pusher.
-device_display_name
-  A string that will allow the user to identify what device owns this pusher.
-lang
-  The preferred language for receiving notifications (eg, 'en' or 'en-US')
-data
-  A dictionary of information for the pusher implementation itself. For HTTP
-  pushers, this must contain a 'url' key which is a string of the URL that
-  should be used to send notifications.
-append
-  If this is set to boolean true, the Home Server should add another pusher
-  with the given pushkey and App ID in addition to any others with different
-  user IDs. Otherwise, the Home Server must remove any other pushers with the
-  same App ID and pushkey for different users. The default is false.
-
-If the pusher was created successfully, a JSON dictionary is returned (which may
-be empty).
+is a single API endpoint for this, as described below.
 
+{{push_http_api}}
 
 Push Rules
 ~~~~~~~~~~
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 472c9d7a..f2c3d0b6 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -13,17 +13,17 @@
 
 Request format:
 
-================== ================= ===========================================
+====================== ================= ===========================================
    Parameter            Value                        Description
-================== ================= ===========================================
+====================== ================= ===========================================
 {% for loc in endpoint.req_param_by_loc -%}
 *{{loc}} parameters*
--------------------------------------------------------------------------------- 
+------------------------------------------------------------------------------------ 
 {% for param in endpoint.req_param_by_loc[loc] -%}
-{{param.key}}{{param.type|indent(19-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(37)}}
+{{param.key}}{{param.type|indent(23-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(41)}}
 {% endfor -%}
 {% endfor -%}
-================== ================= ===========================================
+====================== ================= ===========================================
 
 {% if endpoint.res_tables|length > 0 -%}
 Response format:
@@ -31,18 +31,18 @@ Response format:
 {% for table in endpoint.res_tables -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-=================== ================= ==========================================
+======================= ================= ==========================================
       Param              Type                 Description
-=================== ================= ==========================================
+======================= ================= ==========================================
 {% for row in table.rows -%}
 {# -#}
 {# Row type needs to prepend spaces to line up with the type column (20 ch) -#}
 {# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
 {#      It also needs to then wrap inside the desc col (42 ch width) -#}
 {# -#}
-{{row.key}}{{row.type|indent(20-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(18 - (row.type|length))) |indent_block(38)}}
+{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(40,row.req_str | indent(18 - (row.type|length))) |indent_block(42)}}
 {% endfor -%}
-=================== ================= ==========================================
+======================= ================= ==========================================
 
 {% endfor %}
 {% endif -%}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index eebfba84..1b4b8c7d 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -207,12 +207,18 @@ class MatrixUnits(Units):
                         )
                     # loop top-level json keys
                     json_body = Units.prop(param, "schema/properties")
+                    required_params = []
+                    if Units.prop(param, "schema/required"):
+                        required_params = Units.prop(param, "schema/required")
                     for key in json_body:
+                        pdesc = json_body[key]["description"]
+                        if key in required_params:
+                            pdesc = "**Required.** " + pdesc
                         endpoint["req_params"].append({
                             "key": key,
                             "loc": "JSON body",
                             "type": json_body[key]["type"],
-                            "desc": json_body[key]["description"]
+                            "desc": pdesc
                         })
                 # endfor[param]
                 for row in endpoint["req_params"]:

From 9540069acd3106dacdfdcd85c5f1f225239ea4ce Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 16:06:11 +0100
Subject: [PATCH 272/467] Use valid JSON

---
 api/client-server/v1/push.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/push.yaml b/api/client-server/v1/push.yaml
index fd179fe6..50bf318d 100644
--- a/api/client-server/v1/push.yaml
+++ b/api/client-server/v1/push.yaml
@@ -42,10 +42,10 @@ paths:
                 "device_display_name": "iPhone 9",
                 "app_id": "com.example.app.ios",
                 "profile_tag": "4bea66906d0111e59d70feff819cdc9f",
-                "pushkey": "APA91bHPRgkF3JUikC4ENAHEeMrd41Zxv3hVZjC9KtT8OvPVGJ-hQMRKRrZuJAEcl7B338qju59zJMjw2DELjzEvxwYv7hH5Ynpc1ODQ0aT4U4OFEeco8ohsN5PjL1iC2dNtk2BAokeMCg2ZXKqpc8FXKmhX94kIxQ"
+                "pushkey": "APA91bHPRgkF3JUikC4ENAHEeMrd41Zxv3hVZjC9KtT8OvPVGJ-hQMRKRrZuJAEcl7B338qju59zJMjw2DELjzEvxwYv7hH5Ynpc1ODQ0aT4U4OFEeco8ohsN5PjL1iC2dNtk2BAokeMCg2ZXKqpc8FXKmhX94kIxQ",
                 "data": {
                   "url": "https://push-gateway.location.here"
-                }
+                },
                 "append": false
               }
             properties:

From db25276856b1933cf9fc06707cecfaed5bd73d37 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 7 Oct 2015 17:44:05 +0100
Subject: [PATCH 273/467] Start adding in push definitions

This is going to be painful to represent due to how the push API allows
mixed types (strings or objects) and mixed top-level keys ("content" rule kind
allowing "pattern" as a top-level key). We may wish to re-visit the design
of this API for v2.
---
 .../v1/definitions/push_rule.yaml             |  12 +
 .../v1/definitions/push_ruleset.yaml          |  27 ++
 .../v1/{push.yaml => pusher.yaml}             |   4 +-
 api/client-server/v1/pushrules.yaml           | 434 ++++++++++++++++++
 specification/modules/push.rst                |   8 +-
 5 files changed, 480 insertions(+), 5 deletions(-)
 create mode 100644 api/client-server/v1/definitions/push_rule.yaml
 create mode 100644 api/client-server/v1/definitions/push_ruleset.yaml
 rename api/client-server/v1/{push.yaml => pusher.yaml} (97%)
 create mode 100644 api/client-server/v1/pushrules.yaml

diff --git a/api/client-server/v1/definitions/push_rule.yaml b/api/client-server/v1/definitions/push_rule.yaml
new file mode 100644
index 00000000..be4791c8
--- /dev/null
+++ b/api/client-server/v1/definitions/push_rule.yaml
@@ -0,0 +1,12 @@
+type: "object"
+properties:
+  rule_id:
+    type: string
+  actions:
+    type: array
+    items:
+      type: string
+  default:
+    type: boolean
+  enabled:
+    type: boolean
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/push_ruleset.yaml b/api/client-server/v1/definitions/push_ruleset.yaml
new file mode 100644
index 00000000..77653b98
--- /dev/null
+++ b/api/client-server/v1/definitions/push_ruleset.yaml
@@ -0,0 +1,27 @@
+type: "object"
+properties:
+  content:
+    type: array
+    items:
+      type: object
+      allOf: [ "$ref": "definitions/push_rule.yaml"]
+  override:
+    type: array
+    items:
+      type: object
+      allOf: [ "$ref": "definitions/push_rule.yaml"]
+  room:
+    type: array
+    items:
+      type: object
+      allOf: [ "$ref": "definitions/push_rule.yaml"]
+  sender:
+    type: array
+    items:
+      type: object
+      allOf: [ "$ref": "definitions/push_rule.yaml"]
+  underride:
+    type: array
+    items:
+      type: object
+      allOf: [ "$ref": "definitions/push_rule.yaml"]
diff --git a/api/client-server/v1/push.yaml b/api/client-server/v1/pusher.yaml
similarity index 97%
rename from api/client-server/v1/push.yaml
rename to api/client-server/v1/pusher.yaml
index 50bf318d..3c9ffe8c 100644
--- a/api/client-server/v1/push.yaml
+++ b/api/client-server/v1/pusher.yaml
@@ -62,8 +62,8 @@ paths:
                 type: string
                 enum: ["http", null]
                 description: |-
-                  The kind of pusher to configure. 'http' makes a pusher that
-                  sends HTTP pokes. null deletes the pusher.
+                  The kind of pusher to configure. ``"http"`` makes a pusher that
+                  sends HTTP pokes. ``null`` deletes the pusher.
               profile_tag:
                 type: string
                 description: |-
diff --git a/api/client-server/v1/pushrules.yaml b/api/client-server/v1/pushrules.yaml
new file mode 100644
index 00000000..29a8b0cc
--- /dev/null
+++ b/api/client-server/v1/pushrules.yaml
@@ -0,0 +1,434 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Push Rules API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/pushrules/":
+    get:
+      summary: Retrieve all push rulesets.
+      description: |-
+        Retrieve all push rulesets for this user. Clients can "drill-down" on
+        the rulesets by suffixing a ``scope`` to this path e.g.
+        ``/pushrules/global/``. This will return a subset of this data under the
+        specified key: in this case the ``global`` key.
+      security:
+        - accessToken: []
+      responses:
+        200:
+          schema:
+            type: object
+            required: ["device", "global"]
+            properties:
+              device:
+                type: object
+                title: Devices
+                description: A dictionary of devices.
+                patternProperties:
+                  ".*":
+                    x-pattern: "$PROFILE_TAG"
+                    type: object
+                    description: The ruleset for this profile tag.
+                    title: Ruleset
+                    allOf: [
+                      "$ref": "definitions/push_ruleset.yaml"
+                    ]
+              global:
+                type: object
+                description: The global ruleset.
+                title: Ruleset
+                allOf: [
+                  "$ref": "definitions/push_ruleset.yaml"
+                ]
+          examples:
+            application/json: |-
+              {
+                "device": {},
+                "global": {
+                    "content": [
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "sound",
+                                    "value": "default"
+                                },
+                                {
+                                    "set_tweak": "highlight"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "pattern": "alice",
+                            "rule_id": ".m.rule.contains_user_name"
+                        }
+                    ],
+                    "override": [
+                        {
+                            "actions": [
+                                "dont_notify"
+                            ],
+                            "conditions": [],
+                            "default": true,
+                            "enabled": false,
+                            "rule_id": ".m.rule.master"
+                        },
+                        {
+                            "actions": [
+                                "dont_notify"
+                            ],
+                            "conditions": [
+                                {
+                                    "key": "content.msgtype",
+                                    "kind": "event_match",
+                                    "pattern": "m.notice"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.suppress_notices"
+                        }
+                    ],
+                    "room": [],
+                    "sender": [],
+                    "underride": [
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "sound",
+                                    "value": "ring"
+                                },
+                                {
+                                    "set_tweak": "highlight",
+                                    "value": false
+                                }
+                            ],
+                            "conditions": [
+                                {
+                                    "key": "type",
+                                    "kind": "event_match",
+                                    "pattern": "m.call.invite"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.call"
+                        },
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "sound",
+                                    "value": "default"
+                                },
+                                {
+                                    "set_tweak": "highlight"
+                                }
+                            ],
+                            "conditions": [
+                                {
+                                    "kind": "contains_display_name"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.contains_display_name"
+                        },
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "sound",
+                                    "value": "default"
+                                },
+                                {
+                                    "set_tweak": "highlight",
+                                    "value": false
+                                }
+                            ],
+                            "conditions": [
+                                {
+                                    "is": "2",
+                                    "kind": "room_member_count"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.room_one_to_one"
+                        },
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "sound",
+                                    "value": "default"
+                                },
+                                {
+                                    "set_tweak": "highlight",
+                                    "value": false
+                                }
+                            ],
+                            "conditions": [
+                                {
+                                    "key": "type",
+                                    "kind": "event_match",
+                                    "pattern": "m.room.member"
+                                },
+                                {
+                                    "key": "content.membership",
+                                    "kind": "event_match",
+                                    "pattern": "invite"
+                                },
+                                {
+                                    "key": "state_key",
+                                    "kind": "event_match",
+                                    "pattern": "@alice:example.com"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.invite_for_me"
+                        },
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "highlight",
+                                    "value": false
+                                }
+                            ],
+                            "conditions": [
+                                {
+                                    "key": "type",
+                                    "kind": "event_match",
+                                    "pattern": "m.room.member"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.member_event"
+                        },
+                        {
+                            "actions": [
+                                "notify",
+                                {
+                                    "set_tweak": "highlight",
+                                    "value": false
+                                }
+                            ],
+                            "conditions": [
+                                {
+                                    "key": "type",
+                                    "kind": "event_match",
+                                    "pattern": "m.room.message"
+                                }
+                            ],
+                            "default": true,
+                            "enabled": true,
+                            "rule_id": ".m.rule.message"
+                        }
+                    ]
+                }
+              }
+  "/pushrules/{scope}/{kind}/{ruleId}":
+    get:
+      summary: Retrieve a push rule.
+      description: |-
+        Retrieve a single specified push rule.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: scope
+          required: true
+          x-example: "global"
+          description: |-
+            Either ``global`` or ``device/<profile_tag>`` to specify global
+            rules or device rules for the given ``profile_tag``.
+        - in: path
+          type: string
+          name: kind
+          required: true
+          x-example: room
+          enum: ["override", "underride", "sender", "room", "content"]
+          description: |
+            The kind of rule
+        - in: path
+          type: string
+          name: ruleId
+          required: true
+          x-example: "#spam:example.com"
+          description: |
+            The identifier for the rule.
+      responses:
+        200:
+          description: |-
+            The specific push rule. This will also include keys specific to the
+            rule itself such as the rule's ``actions`` and ``conditions`` if set.
+          examples:
+            application/json: |-
+              {
+                "actions": [
+                    "dont_notify"
+                ],
+                "rule_id": "#spam:matrix.org",
+                "enabled": true
+              }
+          schema:
+            type: object
+            required: ["enabled", "rule_id", "actions"]
+            properties:
+              enabled:
+                type: boolean
+                description: Whether this rule is active or not.
+              rule_id:
+                type: string
+                description: The rule ID from the path.
+    delete:
+      summary: Delete a push rule.
+      description: |-
+        This endpoint removes the push rule defined in the path.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: scope
+          required: true
+          x-example: "global"
+          description: |-
+            Either ``global`` or ``device/<profile_tag>`` to specify global
+            rules or device rules for the given ``profile_tag``.
+        - in: path
+          type: string
+          name: kind
+          required: true
+          x-example: room
+          enum: ["override", "underride", "sender", "room", "content"]
+          description: |
+            The kind of rule
+        - in: path
+          type: string
+          name: ruleId
+          required: true
+          x-example: "#spam:example.com"
+          description: |
+            The identifier for the rule.
+    put:
+      summary: Add or change a push rule.
+      description: |-
+        This endpoint allows the creation, modification and deletion of pushers
+        for this user ID. The behaviour of this endpoint varies depending on the
+        values in the JSON body.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: scope
+          required: true
+          x-example: "global"
+          description: |-
+            Either ``global`` or ``device/<profile_tag>`` to specify global
+            rules or device rules for the given ``profile_tag``.
+        - in: path
+          type: string
+          name: kind
+          required: true
+          x-example: room
+          enum: ["override", "underride", "sender", "room", "content"]
+          description: |
+            The kind of rule
+        - in: path
+          type: string
+          name: ruleId
+          required: true
+          x-example: "#spam:example.com"
+          description: |
+            The identifier for the rule.
+        - in: query
+          type: string
+          name: before
+          required: false
+          x-example: someRuleId
+          description: |-
+            Use 'before' with a ``rule_id`` as its value to make the new rule the
+            next-most important rule with respect to the given rule.
+        - in: query
+          type: string
+          name: after
+          required: false
+          x-example: anotherRuleId
+          description: |-
+            This makes the new rule the next-less important rule relative to the
+            given rule.
+        - in: body
+          name: pushrule
+          description: |-
+            The push rule data. Additional top-level keys may be present depending
+            on the parameters for the rule ``kind``.
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "pattern": "cake*lie",
+                "actions": ["notify"]
+              }
+            properties:
+              actions:
+                type: array
+                description: |-
+                  The action(s) to perform when the conditions for this rule are met.
+                items:
+                  type: string
+                  enum: ["notify", "dont_notify", "coalesce", "set_tweak"]
+                  # TODO: type: object e.g. {"set_sound":"beeroclock.wav"} :/
+              conditions:
+                type: array
+                description: |-
+                  The conditions that must hold true for an event in order for a
+                  rule to be applied to an event. A rule with no conditions
+                  always matches.
+                items:
+                  type: object
+            required: ["actions"]
+      responses:
+        200:
+          description: The pusher was set.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
+        400:
+          description: There was a problem configuring this push rule.
+          examples:
+            application/json: |-
+              {
+                "error": "before/after rule not found: someRuleId",
+                "errcode": "M_UNKNOWN"
+              }
+          schema:
+            type: object
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index bd08dc9e..72991565 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -81,12 +81,12 @@ To receive any notification pokes at all, it is necessary to configure a
 'pusher' on the Home Server that you wish to receive notifications from. There
 is a single API endpoint for this, as described below.
 
-{{push_http_api}}
+{{pusher_http_api}}
 
 Push Rules
 ~~~~~~~~~~
-Home Servers have an interface to configure what events trigger notifications.
-This behaviour is configured through 'Push Rules'. Push Rules come in a variety
+Homeservers have an interface to configure what events trigger notifications.
+This behaviour is configured through "Push Rules". Push Rules come in a variety
 of different kinds and each kind of rule has an associated priority. The
 different kinds of rule, in descending order of priority, are:
 
@@ -332,6 +332,8 @@ but instead have predefined conditions, the behaviour of which can be configured
 using parameters named as described above. In the cases of room and sender
 rules, the rule_id of the rule determines its behaviour.
 
+{{pushrules_http_api}}
+
 Push Rules: API
 ~~~~~~~~~~~~~~~
 Rules live under a hierarchy in the REST API that resembles::

From 2ab9372b167f50046a16f856a5ab62ddc2529cac Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Wed, 7 Oct 2015 18:05:50 +0100
Subject: [PATCH 274/467] new nav bar. todo: get animation working.

---
 supporting-docs/_includes/footer.html         | 24 ++++++++
 supporting-docs/_includes/head.html           | 15 +++++
 supporting-docs/_includes/nav.html            | 55 +++++++++++--------
 supporting-docs/_layouts/default.html         |  7 ++-
 .../_posts/2015-08-14-getting_involved.md     |  5 ++
 supporting-docs/_posts/2015-08-19-faq.md      |  8 +--
 supporting-docs/css/site_overrides.css        |  5 ++
 7 files changed, 90 insertions(+), 29 deletions(-)

diff --git a/supporting-docs/_includes/footer.html b/supporting-docs/_includes/footer.html
index fc43ec97..3e020c64 100644
--- a/supporting-docs/_includes/footer.html
+++ b/supporting-docs/_includes/footer.html
@@ -1 +1,25 @@
   <div id="footer"><div id="footerContent">&copy 2014-2015 Matrix.org</div></div>
+	<script>
+  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+  ga('create', 'UA-54779209-1', 'auto');
+  ga('send', 'pageview');
+</script>
+<script type='text/javascript'>
+/* <![CDATA[ */
+var tocplus = {"smooth_scroll":"1"};
+/* ]]> */
+</script>
+<script type='text/javascript' src='https://matrix.org/blog/wp-content/plugins/table-of-contents-plus/front.min.js?ver=1509'></script>
+<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/jquery.fitvids.js?ver=2.2'></script>
+<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/waypoints.min.js?ver=2.2'></script>
+<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/jquery.magnific-popup.js?ver=2.2'></script>
+<script type='text/javascript'>
+/* <![CDATA[ */
+var et_custom = {"ajaxurl":"https:\/\/matrix.org\/blog\/wp-admin\/admin-ajax.php","images_uri":"https:\/\/matrix.org\/blog\/wp-content\/themes\/Divi-child\/images","et_load_nonce":"dc61fa2f94","subscription_failed":"Please, check the fields below to make sure you entered the correct information.","fill":"Fill","field":"field","invalid":"Invalid email","captcha":"Captcha","prev":"Prev","next":"Next"};
+/* ]]> */
+</script>
+<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/custom.js?ver=2.2'></script>
diff --git a/supporting-docs/_includes/head.html b/supporting-docs/_includes/head.html
index ddb4d988..213e1796 100644
--- a/supporting-docs/_includes/head.html
+++ b/supporting-docs/_includes/head.html
@@ -12,5 +12,20 @@
   <link rel="stylesheet" href="{{ "/css/basic.css" | prepend: site.baseurl }}">
   <link rel="stylesheet" href="{{ "/css/nature.css" | prepend: site.baseurl }}">
 
+  <link rel='stylesheet' id='toc-screen-css'  href='https://matrix.org/blog/wp-content/plugins/table-of-contents-plus/screen.min.css?ver=1509' type='text/css' media='all' />
+  <link rel='stylesheet' id='divi-fonts-css'  href='https://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,800italic,400,300,700,800&#038;subset=latin,latin-ext' type='text/css' media='all' />
+  <link rel='stylesheet' id='divi-style-css'  href='https://matrix.org/blog/wp-content/themes/Divi-child/style.css?ver=2.2' type='text/css' media='all' />
+
   <link rel="canonical" href="{{ page.url | replace:'index.html','' | prepend: site.baseurl | prepend: site.url }}">
   <link rel="alternate" type="application/rss+xml" title="{{ site.title }}" href="{{ "/feed.xml" | prepend: site.baseurl | prepend: site.url }}" />
+  
+<script>
+  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+  ga('create', 'UA-54779209-1', 'auto');
+  ga('send', 'pageview');
+</script>
+
diff --git a/supporting-docs/_includes/nav.html b/supporting-docs/_includes/nav.html
index b9f13762..b7e2d629 100644
--- a/supporting-docs/_includes/nav.html
+++ b/supporting-docs/_includes/nav.html
@@ -1,22 +1,33 @@
-  <div id="header">
-    <div id="headerContent">
-
-        <a href="/#about2" class="navButton">About</a>
-        <a href="/docs/howtos" class="navButton">HOWTOs</a>
-        <a href="/docs/spec" class="navButton">Spec</a>
-        <a href="/docs/api" class="navButton">APIs</a>
-        <a href="/code" class="navButton">Code</a>
-        <a href="/jira" class="navButton">JIRA</a>
-        <a href="/blog/faq" class="navButton">FAQ</a>
-        <a href="/blog" class="navButton">Blog</a>
-<script>
-  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
-  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
-  ga('create', 'UA-54779209-1', 'auto');
-  ga('send', 'pageview');
-</script>
-    </div>
-  </div>
+<header id="main-header" class="et_nav_text_color_dark et-fixed-header">
+  <div class="container clearfix" >
+    <a href="https://matrix.org/blog/">
+      <img src="https://matrix.org/blog/wp-content/uploads/2015/01/logo1.png" alt="Matrix.org" id="logo" />
+    </a>
+    <div id="et-top-navigation">
+      <nav id="top-menu-nav">
+        <ul id="top-menu" class="nav"><li id="menu-item-17" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-17"><a href="http://matrix.org/">Home</a></li>
+<li id="menu-item-794" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-794"><a href="https://matrix.org/blog/try-matrix-now">Try Matrix Now!</a></li>
+<li id="menu-item-348" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-348"><a href="/docs/howtos">HOWTOs</a></li>
+<li id="menu-item-349" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-349"><a href="/docs/spec">Spec</a></li>
+<li id="menu-item-350" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-350"><a href="/docs/api">APIs</a></li>
+<li id="menu-item-351" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-351"><a href="/code">Code</a></li>
+<li id="menu-item-352" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-352"><a href="/jira">JIRA</a></li>
+<li id="menu-item-347" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-347"><a href="https://matrix.org/blog/faq/">FAQ</a></li>
+<li id="menu-item-353" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-353"><a href="/blog">Blog</a></li>
+        </ul>
+      </nav>
+      <div id="et_top_search">
+        <span id="et_search_icon"></span>
+        <form role="search" method="get" class="et-search-form et-hidden" action="https://matrix.org/blog/">
+          <input type="search" class="et-search-field" placeholder="Search &hellip;" value="" name="s" title="Search for:" />
+        </form>
+      </div>
+      <div id="et_mobile_nav_menu">
+        <a href="#" class="mobile_nav closed">
+          <span class="select_page">Select Page</span>
+          <span class="mobile_menu_bar"></span>
+        </a>
+      </div>
+    </div> <!-- #et-top-navigation -->
+  </div> <!-- .container -->
+</header> <!-- #main-header -->
diff --git a/supporting-docs/_layouts/default.html b/supporting-docs/_layouts/default.html
index 00ea25ce..da18483e 100644
--- a/supporting-docs/_layouts/default.html
+++ b/supporting-docs/_layouts/default.html
@@ -4,9 +4,12 @@
   {% include head.html %}
 
   </head>
-  <body>
-    <div class="page-content" id="page">
+  <body class="blog et_fixed_nav et_cover_background linux et_right_sidebar chrome">
+
+    <div class="page-content" id="page-container" style="padding-top: 78px;">
       {% include nav.html %}
+      <div id="main-content">
+
       <div class="wrapper" id="wrapper">
         <div class="document_foo" id="document">
       {{ content }}
diff --git a/supporting-docs/_posts/2015-08-14-getting_involved.md b/supporting-docs/_posts/2015-08-14-getting_involved.md
index 9a37a0ea..162ee7b8 100644
--- a/supporting-docs/_posts/2015-08-14-getting_involved.md
+++ b/supporting-docs/_posts/2015-08-14-getting_involved.md
@@ -33,6 +33,8 @@ You can use multiple clients with the same user, so you might want to also look
 
 ## Run a server and/or client yourself
 
+You can clone our open source projects and run clients and servers yourself. Here is how:
+
 ### Running your own client:
 
 You can run your own Matrix client; there are [numerous clients available](https://matrix.org/blog/try-matrix-now/). You can take Matrix.org's [reference client](https://github.com/matrix-org/matrix-angular-sdk) and use it as-is - or modify it any way you want! Since it's written in JavaScript, running a client is [really easy](https://github.com/matrix-org/matrix-angular-sdk#running)!
@@ -98,6 +100,7 @@ curl -XGET "http://localhost:8008/_matrix/client/api/v1/events?access_token=YOUR
             "end": "s39_18_0",
             "start": "s39_18_0"
         }
+
 7. Even if there are no new events (as in the example above), there will be some pagination stream response keys. The client should make subsequent requests using the value of the "end" key (in this case s39_18_0) as the from query parameter e.g.
 ```
 http://localhost:8008/_matrix/client/api/v1/events?access _token=YOUR_ACCESS_TOKEN&from=s39_18_0
@@ -105,6 +108,8 @@ http://localhost:8008/_matrix/client/api/v1/events?access _token=YOUR_ACCESS_TOK
 
 8. This ensures that you only get new events. Now you have initial rooms and presence, and a stream of events - a good client should be able to process all these events and present them to the user. And potentially you might want to add functionality to generate events as well (such as messages from the user, for example) - again please consult the [client-server API spec](http://matrix.org/docs/howtos/client-server.html)!
 
+| 
+
 ### Write your own server:
 
 We are still working on the server-server spec, so the best thing to do if you are interested in writing a server, is to come talk to us in [#matrix:matrix.org](https://matrix.org/beta/#/room/%23matrix:matrix.org).
diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/_posts/2015-08-19-faq.md
index 402021e1..ce58c158 100644
--- a/supporting-docs/_posts/2015-08-19-faq.md
+++ b/supporting-docs/_posts/2015-08-19-faq.md
@@ -19,9 +19,7 @@ Categories
 
 [Standard](#standard)
 
-[APIs](#apis)
-
-[Reference Implementations](#reference-implementations)
+[Implementations](#implementations)
 
 |
 
@@ -486,7 +484,7 @@ The Apache Licence is a permissive licence. We want the Matrix protocol itself t
 
 ##### Can I write a Matrix homeserver?
 
-Yes. Matrix is just a spec, so implementations of the spec are very welcome! It should be noted that as of October 2015 the server, changes are still being made to the spec, so if you want to write a Matrix homeserver, it is strongly recommended that you chat to the Matrix.org devs in #matrix:matrix.org first! You can also read about the [Federation API here]( https://github.com/matrix-org/matrix-doc/blob/master/specification/30_server_server_api.rst).
+Yes. Matrix is just a spec, so implementations of the spec are very welcome! It should be noted that as of October 2015, changes are still being made to the spec, so if you want to write a Matrix homeserver, it is strongly recommended that you chat to the Matrix.org devs in #matrix:matrix.org first! You can also read about the [Federation API here]( https://github.com/matrix-org/matrix-doc/blob/master/specification/30_server_server_api.rst).
 
 ##### How secure is this?
 
@@ -559,7 +557,7 @@ built by [Jenkins](http://www.matrix.org/jenkins/job/AndroidConsoleDevelop/lastB
 
 ##### Where can I find a web app?
 
-As of Oct 2015, the best web app options are to use https://vector.im - a glossy web client written on top of matrix-react-sdk, or the original AngularJS based client at https://matrix.org/beta, which has serious performance problems and is not currently being maintained.  In future a "Matrix Console" reference web app built on matrix-react-sdk will be released by matrix.org to complement the mobile apps above.
+As of Oct 2015, the best web app options are to use [Vector.im](https://vector.im) - a glossy web client written on top of [matrix-react-sdk](https://github.com/matrix-org/matrix-react-sdk), or the original [AngularJS based client](https://matrix.org/beta), which has serious performance problems and is not currently being maintained.  In future a "Matrix Console" reference web app built on matrix-react-sdk will be released by matrix.org to complement the mobile apps above.
 
 |
 
diff --git a/supporting-docs/css/site_overrides.css b/supporting-docs/css/site_overrides.css
index 0f9a2c80..f7640e1c 100644
--- a/supporting-docs/css/site_overrides.css
+++ b/supporting-docs/css/site_overrides.css
@@ -53,5 +53,10 @@ h1,h2,h3,h4,h5 {
 
 #document {
     margin-top: 10px; /* We want a little whitespace before the page content starts */
+    width: 1080px;
+}
+
+#wrapper {
+    max-width: 1080px;
 }
 

From f6da9d7760537459e4facab69a6f03b3223b227e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 09:57:40 +0100
Subject: [PATCH 275/467] Give valid swagger. Split out rule/ruleset to
 separate definitions.

---
 .../v1/definitions/push_rule.json             | 20 +++++++
 .../v1/definitions/push_rule.yaml             | 12 ----
 .../v1/definitions/push_ruleset.json          | 60 +++++++++++++++++++
 .../v1/definitions/push_ruleset.yaml          | 27 ---------
 api/client-server/v1/pushrules.yaml           | 29 +++++----
 5 files changed, 97 insertions(+), 51 deletions(-)
 create mode 100644 api/client-server/v1/definitions/push_rule.json
 delete mode 100644 api/client-server/v1/definitions/push_rule.yaml
 create mode 100644 api/client-server/v1/definitions/push_ruleset.json
 delete mode 100644 api/client-server/v1/definitions/push_ruleset.yaml

diff --git a/api/client-server/v1/definitions/push_rule.json b/api/client-server/v1/definitions/push_rule.json
new file mode 100644
index 00000000..4df93f67
--- /dev/null
+++ b/api/client-server/v1/definitions/push_rule.json
@@ -0,0 +1,20 @@
+{
+  "type": "object", 
+  "properties": {
+    "default": {
+      "type": "boolean"
+    }, 
+    "enabled": {
+      "type": "boolean"
+    }, 
+    "rule_id": {
+      "type": "string"
+    }, 
+    "actions": {
+      "items": {
+        "type": ["object", "string"]
+      }, 
+      "type": "array"
+    }
+  }
+}
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/push_rule.yaml b/api/client-server/v1/definitions/push_rule.yaml
deleted file mode 100644
index be4791c8..00000000
--- a/api/client-server/v1/definitions/push_rule.yaml
+++ /dev/null
@@ -1,12 +0,0 @@
-type: "object"
-properties:
-  rule_id:
-    type: string
-  actions:
-    type: array
-    items:
-      type: string
-  default:
-    type: boolean
-  enabled:
-    type: boolean
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/push_ruleset.json b/api/client-server/v1/definitions/push_ruleset.json
new file mode 100644
index 00000000..e0372701
--- /dev/null
+++ b/api/client-server/v1/definitions/push_ruleset.json
@@ -0,0 +1,60 @@
+{
+  "type": "object", 
+  "properties": {
+    "content": {
+      "items": {
+        "type": "object", 
+        "allOf": [
+          {
+            "$ref": "push_rule.json"
+          }
+        ]
+      }, 
+      "type": "array"
+    }, 
+    "override": {
+      "items": {
+        "type": "object", 
+        "allOf": [
+          {
+            "$ref": "push_rule.json"
+          }
+        ]
+      }, 
+      "type": "array"
+    }, 
+    "sender": {
+      "items": {
+        "type": "object", 
+        "allOf": [
+          {
+            "$ref": "push_rule.json"
+          }
+        ]
+      }, 
+      "type": "array"
+    }, 
+    "underride": {
+      "items": {
+        "type": "object", 
+        "allOf": [
+          {
+            "$ref": "push_rule.json"
+          }
+        ]
+      }, 
+      "type": "array"
+    }, 
+    "room": {
+      "items": {
+        "type": "object", 
+        "allOf": [
+          {
+            "$ref": "push_rule.json"
+          }
+        ]
+      }, 
+      "type": "array"
+    }
+  }
+}
\ No newline at end of file
diff --git a/api/client-server/v1/definitions/push_ruleset.yaml b/api/client-server/v1/definitions/push_ruleset.yaml
deleted file mode 100644
index 77653b98..00000000
--- a/api/client-server/v1/definitions/push_ruleset.yaml
+++ /dev/null
@@ -1,27 +0,0 @@
-type: "object"
-properties:
-  content:
-    type: array
-    items:
-      type: object
-      allOf: [ "$ref": "definitions/push_rule.yaml"]
-  override:
-    type: array
-    items:
-      type: object
-      allOf: [ "$ref": "definitions/push_rule.yaml"]
-  room:
-    type: array
-    items:
-      type: object
-      allOf: [ "$ref": "definitions/push_rule.yaml"]
-  sender:
-    type: array
-    items:
-      type: object
-      allOf: [ "$ref": "definitions/push_rule.yaml"]
-  underride:
-    type: array
-    items:
-      type: object
-      allOf: [ "$ref": "definitions/push_rule.yaml"]
diff --git a/api/client-server/v1/pushrules.yaml b/api/client-server/v1/pushrules.yaml
index 29a8b0cc..17061b2d 100644
--- a/api/client-server/v1/pushrules.yaml
+++ b/api/client-server/v1/pushrules.yaml
@@ -30,6 +30,7 @@ paths:
         - accessToken: []
       responses:
         200:
+          description: All the push rulesets for this user.
           schema:
             type: object
             required: ["device", "global"]
@@ -38,21 +39,20 @@ paths:
                 type: object
                 title: Devices
                 description: A dictionary of devices.
-                patternProperties:
-                  ".*":
+                additionalProperties:
                     x-pattern: "$PROFILE_TAG"
                     type: object
                     description: The ruleset for this profile tag.
                     title: Ruleset
                     allOf: [
-                      "$ref": "definitions/push_ruleset.yaml"
+                      "$ref": "definitions/push_ruleset.json"
                     ]
               global:
                 type: object
                 description: The global ruleset.
                 title: Ruleset
                 allOf: [
-                  "$ref": "definitions/push_ruleset.yaml"
+                  "$ref": "definitions/push_ruleset.json"
                 ]
           examples:
             application/json: |-
@@ -292,14 +292,11 @@ paths:
               }
           schema:
             type: object
-            required: ["enabled", "rule_id", "actions"]
-            properties:
-              enabled:
-                type: boolean
-                description: Whether this rule is active or not.
-              rule_id:
-                type: string
-                description: The rule ID from the path.
+            description: The push rule.
+            title: PushRule
+            allOf: [
+              "$ref": "definitions/push_rule.json"
+            ]
     delete:
       summary: Delete a push rule.
       description: |-
@@ -330,6 +327,14 @@ paths:
           x-example: "#spam:example.com"
           description: |
             The identifier for the rule.
+      responses:
+        200:
+          description: The push rule was deleted.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
     put:
       summary: Add or change a push rule.
       description: |-

From 56ce432399f3dd72614664d8d9450da41afb46f0 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 10:14:05 +0100
Subject: [PATCH 276/467] Get profile tag keys displaying correctly.

---
 api/client-server/v1/pushrules.yaml           |  4 +-
 specification/modules/push.rst                | 81 -------------------
 .../matrix_templates/templates/http-api.tmpl  | 10 +--
 templating/matrix_templates/units.py          |  5 +-
 4 files changed, 11 insertions(+), 89 deletions(-)

diff --git a/api/client-server/v1/pushrules.yaml b/api/client-server/v1/pushrules.yaml
index 17061b2d..c9dd777e 100644
--- a/api/client-server/v1/pushrules.yaml
+++ b/api/client-server/v1/pushrules.yaml
@@ -25,7 +25,7 @@ paths:
         Retrieve all push rulesets for this user. Clients can "drill-down" on
         the rulesets by suffixing a ``scope`` to this path e.g.
         ``/pushrules/global/``. This will return a subset of this data under the
-        specified key: in this case the ``global`` key.
+        specified key e.g. the ``global`` key.
       security:
         - accessToken: []
       responses:
@@ -38,7 +38,7 @@ paths:
               device:
                 type: object
                 title: Devices
-                description: A dictionary of devices.
+                description: A dictionary of profile tags to rulesets.
                 additionalProperties:
                     x-pattern: "$PROFILE_TAG"
                     type: object
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index 72991565..6411f3df 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -336,33 +336,6 @@ rules, the rule_id of the rule determines its behaviour.
 
 Push Rules: API
 ~~~~~~~~~~~~~~~
-Rules live under a hierarchy in the REST API that resembles::
-
-  $PREFIX/pushrules/<scope>/<kind>/<rule_id>
-
-The component parts are as follows:
-
-scope
-  Either 'global' or 'device/<profile_tag>' to specify global rules or
-  device rules for the given profile_tag.
-kind
-  The kind of rule, i.e. 'override', 'underride', 'sender', 'room', 'content'.
-rule_id
-  The identifier for the rule.
-
-To add or change a rule, a client performs a PUT request to the appropriate URL.
-When adding rules of a type that has an ordering, the client can add parameters
-that define the priority of the rule:
-
-before
-  Use 'before' with a rule_id as its value to make the new rule the next-more
-  important rule with respect to the given rule.
-after
-  This makes the new rule the next-less important rule relative to the given
-  rule.
-
-All requests to the push rules API also require an access_token as a query
-parameter.
 
 The content of the PUT request is a JSON object with a list of actions under the
 'actions' key and either conditions (under the 'conditions' key) or the
@@ -391,60 +364,6 @@ To add a custom sound for notifications messages containing the word 'beer' in a
   curl -X PUT -H "Content-Type: application/json" -d '{ "conditions": [{"kind": "event_match", "key": "content.body", "pattern": "beer" }, {"kind": "room_member_count", "is": "<=10"}], "actions" : ["notify", {"set_sound":"beeroclock.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456
 
 
-To delete rules, a client would just make a DELETE request to the same URL::
-
-  curl -X DELETE "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456"
-
-
-Retrieving the current ruleset can be done either by fetching individual rules
-using the scheme as specified above. This returns the rule in the same format as
-would be given in the PUT API with the addition of a rule_id::
-
-  curl "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%23spam%3Amatrix.org?access_token=123456"
-
-Returns::
-
-  {
-    "actions": [
-        "dont_notify"
-    ],
-    "rule_id": "#spam:matrix.org",
-    "enabled": true
-  }
-
-Clients can also fetch broader sets of rules by removing path components.
-Requesting the root level returns a structure as follows::
-
-  {
-      "device": {
-          "exampledevice": {
-              "content": [],
-              "override": [],
-              "room": [
-                  {
-                      "actions": [
-                          "dont_notify"
-                      ],
-                      "rule_id": "#spam:matrix.org",
-                      "enabled", true
-                  }
-              ],
-              "sender": [],
-              "underride": []
-          }
-      },
-      "global": {
-          "content": [],
-          "override": [],
-          "room": [],
-          "sender": [],
-          "underride": []
-      }
-  }
-
-Adding patch components to the request drills down into this structure to filter
-to only the requested set of rules.
-
 Enabling and Disabling Rules
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Rules can be enabled or disabled with a PUT operation to the 'enabled' component
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index f2c3d0b6..33b2cc98 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -31,18 +31,18 @@ Response format:
 {% for table in endpoint.res_tables -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-======================= ================= ==========================================
-      Param              Type                 Description
-======================= ================= ==========================================
+======================= ========================= ==========================================
+      Param                        Type                           Description
+======================= ========================= ==========================================
 {% for row in table.rows -%}
 {# -#}
 {# Row type needs to prepend spaces to line up with the type column (20 ch) -#}
 {# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
 {#      It also needs to then wrap inside the desc col (42 ch width) -#}
 {# -#}
-{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(40,row.req_str | indent(18 - (row.type|length))) |indent_block(42)}}
+{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(40,row.req_str | indent(26 - (row.type|length))) |indent_block(50)}}
 {% endfor -%}
-======================= ================= ==========================================
+======================= ========================= ==========================================
 
 {% endfor %}
 {% endif -%}
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 1b4b8c7d..24e92f5a 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -86,7 +86,10 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                         props[key_name]["additionalProperties"],
                         enforce_title=True
                     )
-                    value_type = "{string: %s}" % nested_object[0]["title"]
+                    key = props[key_name]["additionalProperties"].get(
+                        "x-pattern", "string"
+                    )
+                    value_type = "{%s: %s}" % (key, nested_object[0]["title"])
                     if not nested_object[0].get("no-table"):
                         tables += nested_object
                 else:

From a9618a981b2a861b59acb304c8fc37b3f985cc91 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 11:16:42 +0100
Subject: [PATCH 277/467] Swaggerify the /enabled endpoint

---
 api/check_examples.py                |   2 +-
 api/client-server/v1/pushrules.yaml  |  49 +++++++++++-
 specification/modules/push.rst       | 113 ++++++++++++++++-----------
 specification/targets.yaml           |   2 +-
 templating/matrix_templates/units.py |   5 ++
 5 files changed, 122 insertions(+), 49 deletions(-)

diff --git a/api/check_examples.py b/api/check_examples.py
index f08b2dc1..0622bfeb 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -52,7 +52,7 @@ def check_parameter(filepath, request, parameter):
             jsonschema.validate(example, schema)
         except Exception as e:
             raise ValueError("Error validating JSON schema for %r %r" % (
-                request, code
+                request
             ), e)
 
 
diff --git a/api/client-server/v1/pushrules.yaml b/api/client-server/v1/pushrules.yaml
index c9dd777e..eef0b2c7 100644
--- a/api/client-server/v1/pushrules.yaml
+++ b/api/client-server/v1/pushrules.yaml
@@ -436,4 +436,51 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
-
+  "/pushrules/{scope}/{kind}/{ruleId}/enabled":
+    put:
+      summary: "Enable or disable a push rule."
+      description: |-
+        This endpoint allows clients to enable or disable the specified push rule.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: scope
+          required: true
+          x-example: "global"
+          description: |-
+            Either ``global`` or ``device/<profile_tag>`` to specify global
+            rules or device rules for the given ``profile_tag``.
+        - in: path
+          type: string
+          name: kind
+          required: true
+          x-example: room
+          enum: ["override", "underride", "sender", "room", "content"]
+          description: |
+            The kind of rule
+        - in: path
+          type: string
+          name: ruleId
+          required: true
+          x-example: "#spam:example.com"
+          description: |
+            The identifier for the rule.
+        - in: body
+          name: <body>
+          description: |
+            Whether the push rule is enabled or not.
+          required: true
+          schema:
+            type: boolean
+            example: |-
+              true
+      responses:
+        200:
+          description: The push rule was enabled or disabled.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object  # empty json object
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index 6411f3df..ab3f0bf0 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -110,7 +110,7 @@ Underride
 
 In addition, each kind of rule may be either global or device-specific. Device
 specific rules only affect delivery of notifications via pushers with a matching
-profile_tag. All device-specific rules are higher priority than all global
+``profile_tag``. All device-specific rules are higher priority than all global
 rules. Thusly, the full list of rule kinds, in descending priority order, is as
 follows:
 
@@ -145,7 +145,7 @@ If no rules match an event, the Home Server should not notify for the message
 themselves are never alerted for.
 
 Predefined Rules
-~~~~~~~~~~~~~~~~
+++++++++++++++++
 Matrix specifies the following rule IDs for server default rules. Home Servers
 may define rules as follows with the given IDs. If Home Servers provide rules
 with these IDs, their semantics should match those given below:
@@ -247,8 +247,8 @@ with these IDs, their semantics should match those given below:
         ],
     }
 
-Push Rules: Actions:
-~~~~~~~~~~~~~~~~~~~~
+Actions
++++++++
 All rules have an associated list of 'actions'. An action affects if and how a
 notification is delivered for a matching event. This standard defines the
 following actions, although if Home servers wish to support more, they are free
@@ -273,8 +273,8 @@ Actions that have no parameters are represented as a string. Otherwise, they are
 represented as a dictionary with a key equal to their name and other keys as
 their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
 
-Push Rules: Actions: Tweaks
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Tweaks
+^^^^^^
 The ``set_tweak`` key action is used to add an entry to the 'tweaks' dictionary
 that is sent in the notification poke. The following tweaks are defined:
 
@@ -295,8 +295,8 @@ notification light on a mobile device.
 If a kind of tweak that a client understands is not specified in an action, the
 client may choose a sensible behaviour for the tweak.
 
-Push Rules: Conditions
-~~~~~~~~~~~~~~~~~~~~~~
+Conditions
+++++++++++
 Override, Underride and Default rules have a list of 'conditions'. All
 conditions must hold true for an event in order for a rule to be applied to an
 event. A rule with no conditions always matches. Matrix specifies the following
@@ -332,44 +332,65 @@ but instead have predefined conditions, the behaviour of which can be configured
 using parameters named as described above. In the cases of room and sender
 rules, the rule_id of the rule determines its behaviour.
 
-{{pushrules_http_api}}
-
 Push Rules: API
 ~~~~~~~~~~~~~~~
 
-The content of the PUT request is a JSON object with a list of actions under the
-'actions' key and either conditions (under the 'conditions' key) or the
-appropriate parameters for the rule (under the appropriate key name).
-
-Examples:
-
-To create a rule that suppresses notifications for the room with ID '!dj234r78wl45Gh4D:matrix.org'::
-
-  curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456"
-
-To suppress notifications for the user '@spambot:matrix.org'::
-
-  curl -X PUT -H "Content-Type: application/json" -d '{ "actions" : ["dont_notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456"
-
-To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of 'SSByZWFsbHkgbGlrZSBjYWtl')::
-
-  curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456"
-
-To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseeding the previous rule::
-
-  curl -X PUT -H "Content-Type: application/json" -d '{ "pattern": "cake*lie", "actions" : ["notify"] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl"
+Clients can retrieve, add, modify and remove push rules globally or per-device
+using the APIs below.
 
-To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules)::
-
-  curl -X PUT -H "Content-Type: application/json" -d '{ "conditions": [{"kind": "event_match", "key": "content.body", "pattern": "beer" }, {"kind": "room_member_count", "is": "<=10"}], "actions" : ["notify", {"set_sound":"beeroclock.wav"}] }' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456
-
-
-Enabling and Disabling Rules
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Rules can be enabled or disabled with a PUT operation to the 'enabled' component
-beneath the rule's URI with a content of 'true' or 'false'::
+{{pushrules_http_api}}
 
-  curl -X PUT -H "Content-Type: application/json" -d 'false' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org/enabled?access_token=123456"
+Examples
+++++++++
+
+To create a rule that suppresses notifications for the room with ID
+``!dj234r78wl45Gh4D:matrix.org``::
+
+  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \
+  '{
+     "actions" : ["dont_notify"]
+   }' 
+
+To suppress notifications for the user ``@spambot:matrix.org``::
+
+  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \
+  '{
+     "actions" : ["dont_notify"]
+   }' 
+
+To always notify for messages that contain the work 'cake' and set a specific
+sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``)::
+
+  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \
+  '{
+     "pattern": "cake",
+     "actions" : ["notify", {"set_sound":"cakealarm.wav"}]
+   }' 
+
+To add a rule suppressing notifications for messages starting with 'cake' but
+ending with 'lie', superseding the previous rule::
+
+  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \
+  '{
+     "pattern": "cake*lie",
+     "actions" : ["notify"]
+   }' 
+
+To add a custom sound for notifications messages containing the word 'beer' in
+any rooms with 10 members or fewer (with greater importance than the room,
+sender and content rules)::
+
+  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \
+  '{
+     "conditions": [
+       {"kind": "event_match", "key": "content.body", "pattern": "beer" },
+       {"kind": "room_member_count", "is": "<=10"}
+     ],
+     "actions" : [
+       "notify",
+       {"set_sound":"beeroclock.wav"}
+     ]
+   }' 
 
 Server behaviour
 ----------------
@@ -504,8 +525,11 @@ rejected
   necessarily be the notification in the request that failed: it could be that
   a previous notification to the same pushkey failed.
 
-Push: Recommendations for APNS
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Push Gateway behaviour
+----------------------
+
+Recommendations for APNS
+~~~~~~~~~~~~~~~~~~~~~~~~
 For sending APNS notifications, the exact format is flexible and up to the
 client app and its push gateway to agree on (since APNS requires that the sender
 have a private key owned by the app developer, each app must have its own push
@@ -518,9 +542,6 @@ gateway). However, Matrix strongly recommends:
    gateway before returning and instead to store failures and return
    'rejected' responses next time that pushkey is used.
 
-Push Gateway behaviour
-----------------------
-
 Security considerations
 -----------------------
 
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 8106b7cf..5786ea6b 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -24,7 +24,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/history_visibility.rst
     - modules/push.rst
 
-title_styles: ["=", "-", "~", "+", "^"]
+title_styles: ["=", "-", "~", "+", "^", "`"]
 
 # The templating system doesn't know the right title style to use when generating
 # RST. These symbols are 'relative' to say "make a sub-title" (-1), "make a title
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 24e92f5a..c3be2b60 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -184,6 +184,11 @@ class MatrixUnits(Units):
                         val_type = refType  # TODO: Resolve to human-readable.
                     if not val_type and schemaFmt:
                         val_type = schemaFmt
+                    # handle top-level strings/bools
+                    if not val_type and Units.prop(param, "schema/type") == "string":
+                        val_type = "string"
+                    if not val_type and Units.prop(param, "schema/type") == "boolean":
+                        val_type = "boolean"
                     if val_type:
                         endpoint["req_params"].append({
                             "key": param["name"],

From 31ae4b38597f57c6026de29a7fd9044d6e632a61 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 13:08:21 +0100
Subject: [PATCH 278/467] Swaggerify push notification API

Edit units.py to support nested JSON request keys
---
 api/check_examples.py                         |   2 +-
 api/client-server/v1/push_notifier.yaml       | 194 ++++++++++++++++++
 specification/modules/push.rst                |   7 +-
 .../matrix_templates/templates/http-api.tmpl  |  12 +-
 templating/matrix_templates/units.py          |  13 ++
 5 files changed, 218 insertions(+), 10 deletions(-)
 create mode 100644 api/client-server/v1/push_notifier.yaml

diff --git a/api/check_examples.py b/api/check_examples.py
index 0622bfeb..ee3c773c 100755
--- a/api/check_examples.py
+++ b/api/check_examples.py
@@ -51,7 +51,7 @@ def check_parameter(filepath, request, parameter):
             schema['id'] = fileurl
             jsonschema.validate(example, schema)
         except Exception as e:
-            raise ValueError("Error validating JSON schema for %r %r" % (
+            raise ValueError("Error validating JSON schema for %r" % (
                 request
             ), e)
 
diff --git a/api/client-server/v1/push_notifier.yaml b/api/client-server/v1/push_notifier.yaml
new file mode 100644
index 00000000..54576b6c
--- /dev/null
+++ b/api/client-server/v1/push_notifier.yaml
@@ -0,0 +1,194 @@
+swagger: '2.0'
+info:
+  title: "Matrix Push Notification API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/push/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+paths:
+  "/notify":
+    post:
+      summary: Notify a push gateway about an event.
+      description: |-
+        This endpoint is invoked by HTTP pushers to notify a push gateway about
+        an event.
+        *NB: Notifications are send to the URL configured when the pusher is
+        created. This means that the HTTP path may be different depending on the
+        push gateway.*
+      parameters:
+        - in: body
+          name: notification
+          description: Information about the push notification.
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "notification": {
+                  "id": "$3957tyerfgewrf384",
+                  "room_id": "!slw48wfj34rtnrf:example.com",
+                  "type": "m.room.message",
+                  "sender": "@exampleuser:matrix.org",
+                  "sender_display_name": "Major Tom",
+                  "room_name": "Mission Control",
+                  "room_alias": "#exampleroom:matrix.org",
+                  "prio": "high",
+                  "content": {
+                    "msgtype": "m.text",
+                    "body": "I'm floating in a most peculiar way."
+                  }
+                },
+                "counts": {
+                   "unread" : 2,
+                   "missed_calls": 1
+                },
+                "devices": [
+                  {
+                     "app_id": "org.matrix.matrixConsole.ios",
+                     "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/",
+                     "pushkey_ts": 12345678,
+                     "data" : {},
+                     "tweaks": {
+                       "sound": "bing"
+                      }
+                  }
+                ]
+              }
+            required: ["notification", "counts", "devices"]
+            properties:
+              notification:
+                type: object
+                description: Information about the push notification
+                required: ["id", "room_id", "type", "sender"]
+                properties:
+                  id:
+                    type: string
+                    description: |-
+                      An identifier for this notification that may be used to
+                      detect duplicate notification requests. This is not
+                      necessarily the ID of the event that triggered the
+                      notification.
+                  room_id:
+                    type: string
+                    description: The ID of the room in which this event occurred.
+                  type:
+                    type: string
+                    description: The type of the event as in the event's ``type`` field.
+                  sender:
+                    type: string
+                    description: The sender of the event as in the corresponding event field.
+                  sender_display_name:
+                    type: string
+                    description: |-
+                      The current display name of the sender in the room in which
+                      the event occurred.
+                  room_name:
+                    type: string
+                    description: The name of the room in which the event occurred.
+                  room_alias:
+                    type: string
+                    description: An alias to display for the room in which the event occurred.
+                  prio:
+                    type: string
+                    enum: ["high", "low"]
+                    description: |-
+                      The priority of the notification. If omitted, ``high`` is
+                      assumed. This may be used by push gateways to deliver less
+                      time-sensitive notifications in a way that will preserve
+                      battery power on mobile devices.
+                  content:
+                    type: object
+                    title: EventContent
+                    properties:
+                      body:
+                        type: string
+                        description: Message text
+                    description: |-
+                      The ``content`` field from the event, if present. If the
+                      event had no content field, this field is omitted.
+              counts:
+                type: object
+                description: |-
+                  This is a dictionary of the current number of unacknowledged
+                  communications for the recipient user. Counts whose value is
+                  zero are omitted.
+                properties:
+                  unread:
+                    type: integer
+                    description: |-
+                      The number of unread messages a user has across all of the
+                      rooms they are a member of.
+                  missed_calls:
+                    type: integer
+                    description: |-
+                      The number of unacknowledged missed calls a user has
+                      across all rooms of which they are a member.
+              devices:
+                type: array
+                title: Devices
+                description: |-
+                  This is an array of devices that the notification should be sent to.
+                items:
+                  type: object
+                  properties:
+                    app_id:
+                      type: string
+                      description: |-
+                        The app_id given when the pusher was created.
+                    pushkey:
+                      type: string
+                      description: The pushkey given when the pusher was created.
+                    pushkey_ts:
+                      type: integer
+                      description: |-
+                        The unix timestamp (in seconds) when the
+                        pushkey was last updated.
+                    data:
+                      type: object
+                      description: |-
+                        A dictionary of additional pusher-specific data. For
+                        'http' pushers, this is the data dictionary passed in at
+                        pusher creation minus the ``url`` key.
+                    tweaks:
+                      type: object
+                      description: |-
+                        A dictionary of customisations made to the way this
+                        notification is to be presented. These are added by push rules.
+                      properties:
+                        sound:
+                          type: string
+                          description: |-
+                            Sets the sound file that should be played.
+                            ``default`` means that a default sound should be played.
+      responses:
+        200:
+          description: A list of rejected push keys.
+          examples:
+            application/json: |-
+              {
+                "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ]
+              }
+          schema:
+            type: object  # empty json object
+            properties:
+              rejected:
+                type: array
+                description: |-
+                  A list of all pushkeys given in the notification request that
+                  are not valid. These could have been rejected by an upstream
+                  gateway because they have expired or have never been valid.
+                  Homeservers must cease sending notification requests for these
+                  pushkeys and remove the associated pushers. It may not
+                  necessarily be the notification in the request that failed:
+                  it could be that a previous notification to the same pushkey
+                  failed.
+                items:
+                  type: string
+                  description: A pushkey
+
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index ab3f0bf0..bc76f5d6 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -395,12 +395,13 @@ sender and content rules)::
 Server behaviour
 ----------------
 
-HTTP Notification Protocol
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
 This describes the format used by "HTTP" pushers to send notifications of
 events.
 
+{{push_notifier_http_api}}
+
+
+
 Notifications are sent as HTTP POST requests to the URL configured when the
 pusher is created, but Matrix strongly recommends that the path should be::
 
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 33b2cc98..787f02ca 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -13,17 +13,17 @@
 
 Request format:
 
-====================== ================= ===========================================
-   Parameter            Value                        Description
-====================== ================= ===========================================
+=================================== ================= ===========================================
+   Parameter                             Value                        Description
+=================================== ================= ===========================================
 {% for loc in endpoint.req_param_by_loc -%}
 *{{loc}} parameters*
------------------------------------------------------------------------------------- 
+------------------------------------------------------------------------------------------------- 
 {% for param in endpoint.req_param_by_loc[loc] -%}
-{{param.key}}{{param.type|indent(23-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(41)}}
+{{param.key}}{{param.type|indent(36-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(54)}}
 {% endfor -%}
 {% endfor -%}
-====================== ================= ===========================================
+=================================== ================= ===========================================
 
 {% if endpoint.res_tables|length > 0 -%}
 Response format:
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index c3be2b60..558461b9 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -228,6 +228,19 @@ class MatrixUnits(Units):
                             "type": json_body[key]["type"],
                             "desc": pdesc
                         })
+                        if json_body[key]["type"] in ["object"]:
+                            req_tables = get_json_schema_object_fields(
+                                json_body[key]
+                            )
+                            for table in req_tables:
+                                for row in table["rows"]:
+                                    endpoint["req_params"].append({
+                                        "key": key + "." + row["key"],
+                                        "loc": "JSON body",
+                                        "type": row["type"],
+                                        "desc": row["req_str"] + row["desc"]
+                                    })
+
                 # endfor[param]
                 for row in endpoint["req_params"]:
                     self.log("Request parameter: %s" % row)

From 65504db7bb3c4c17e58cff52583699c69e45b9a9 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 13:40:21 +0100
Subject: [PATCH 279/467] Display nested keys on arrays of objects. Make it
 valid swagger.

---
 api/client-server/v1/push_notifier.yaml |  5 +++++
 api/client-server/v1/pushrules.yaml     |  2 ++
 templating/matrix_templates/units.py    | 30 +++++++++++++++++++------
 3 files changed, 30 insertions(+), 7 deletions(-)

diff --git a/api/client-server/v1/push_notifier.yaml b/api/client-server/v1/push_notifier.yaml
index 54576b6c..0a0204c0 100644
--- a/api/client-server/v1/push_notifier.yaml
+++ b/api/client-server/v1/push_notifier.yaml
@@ -151,12 +151,17 @@ paths:
                         pushkey was last updated.
                     data:
                       type: object
+                      title: PusherData
                       description: |-
                         A dictionary of additional pusher-specific data. For
                         'http' pushers, this is the data dictionary passed in at
                         pusher creation minus the ``url`` key.
+                      properties:
+                        foo:
+                          type: string
                     tweaks:
                       type: object
+                      title: Tweaks
                       description: |-
                         A dictionary of customisations made to the way this
                         notification is to be presented. These are added by push rules.
diff --git a/api/client-server/v1/pushrules.yaml b/api/client-server/v1/pushrules.yaml
index eef0b2c7..31e84f55 100644
--- a/api/client-server/v1/pushrules.yaml
+++ b/api/client-server/v1/pushrules.yaml
@@ -413,6 +413,8 @@ paths:
                   always matches.
                 items:
                   type: object
+                  title: conditions
+                  allOf: [ "$ref": "definitions/push_condition.json" ]
             required: ["actions"]
       responses:
         200:
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 558461b9..83376fba 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -219,23 +219,39 @@ class MatrixUnits(Units):
                     if Units.prop(param, "schema/required"):
                         required_params = Units.prop(param, "schema/required")
                     for key in json_body:
-                        pdesc = json_body[key]["description"]
+                        req_obj = json_body[key]
+                        pdesc = req_obj["description"]
                         if key in required_params:
                             pdesc = "**Required.** " + pdesc
+
+                        is_array = req_obj["type"] == "array"
+                        is_array_of_objects = (
+                            is_array and req_obj["items"]["type"] == "object"
+                        )
                         endpoint["req_params"].append({
                             "key": key,
                             "loc": "JSON body",
-                            "type": json_body[key]["type"],
+                            "type": (
+                                req_obj["type"] if not is_array else
+                                "array[%s]" % req_obj["items"]["type"]
+                            ),
                             "desc": pdesc
                         })
-                        if json_body[key]["type"] in ["object"]:
-                            req_tables = get_json_schema_object_fields(
-                                json_body[key]
-                            )
+                        if not is_array_of_objects and req_obj["type"] == "array":
+                            continue
+                        # Put in request.dot.notation for nested keys
+                        if req_obj["type"] in ["object", "array"]:
+                            if is_array_of_objects:
+                                req_obj = req_obj["items"]
+
+                            req_tables = get_json_schema_object_fields(req_obj)
+                            key_sep = "[0]." if is_array else "."
                             for table in req_tables:
+                                if table.get("no-table"):
+                                    continue
                                 for row in table["rows"]:
                                     endpoint["req_params"].append({
-                                        "key": key + "." + row["key"],
+                                        "key": key + key_sep + row["key"],
                                         "loc": "JSON body",
                                         "type": row["type"],
                                         "desc": row["req_str"] + row["desc"]

From befa23624e0a9d41ba9bfe56acb6da5445728370 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 13:41:47 +0100
Subject: [PATCH 280/467] It's dangerous to go alone, take this.

---
 api/client-server/v1/definitions/push_condition.json | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100644 api/client-server/v1/definitions/push_condition.json

diff --git a/api/client-server/v1/definitions/push_condition.json b/api/client-server/v1/definitions/push_condition.json
new file mode 100644
index 00000000..1d84955c
--- /dev/null
+++ b/api/client-server/v1/definitions/push_condition.json
@@ -0,0 +1,9 @@
+{
+  "type": "object", 
+  "properties": {
+    "kind": {
+      "type": "string",
+      "enum": ["event_match", "profile_tag", "contains_display_name", "room_member_count"]
+    }
+  }
+}
\ No newline at end of file

From 9c95ee5c1f27eb88a414207e4038292ee6dba5bd Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 15:36:03 +0100
Subject: [PATCH 281/467] Modify wording and formatting

---
 api/client-server/v1/push_notifier.yaml |   6 +
 specification/modules/push.rst          | 458 +++++++++---------------
 2 files changed, 174 insertions(+), 290 deletions(-)

diff --git a/api/client-server/v1/push_notifier.yaml b/api/client-server/v1/push_notifier.yaml
index 0a0204c0..07c93f3b 100644
--- a/api/client-server/v1/push_notifier.yaml
+++ b/api/client-server/v1/push_notifier.yaml
@@ -94,6 +94,12 @@ paths:
                   room_alias:
                     type: string
                     description: An alias to display for the room in which the event occurred.
+                  user_is_target:
+                    type: boolean
+                    description: |-
+                      This is true if the user receiving the notification is the
+                      subject of a member event (i.e. the ``state_key`` of the
+                      member event is equal to the user's Matrix ID).
                   prio:
                     type: string
                     enum: ["high", "low"]
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index bc76f5d6..ac156dcc 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -27,92 +27,101 @@ Push Notifications
          Mobile Device or Client                                               
 
 
-Matrix supports push notifications as a first class citizen. Home Servers send
-notifications of user events to user-configured HTTP endpoints. User may also
-configure a number of rules that determine what events generate notifications.
-These are all stored and managed by the users home server such that settings can
-be reused between client apps as appropriate.
+This module adds support for push notifications. Homeservers send notifications
+of user events to user-configured HTTP endpoints. Users may also configure a
+number of rules that determine which events generate notifications. These are
+all stored and managed by the user's homeserver. This allows user-specific push
+settings to be reused between client applications.
 
 The above diagram shows the flow of push notifications being sent to a handset
 where push notifications are submitted via the handset vendor, such as Apple's
 APNS or Google's GCM. This happens as follows:
 
- 1. The client app signs in to a Matrix Home Server
- 2. The client app registers with its vendor's Push Notification provider and
+ 1. The client app signs in to a homeserver.
+ 2. The client app registers with its vendor's Push Provider and
     obtains a routing token of some kind.
- 3. The mobile app, uses the Matrix client/server API to add a 'pusher',
-    providing the URL of a specific Push Gateway which is configured for that
+ 3. The mobile app uses the Client/Server API to add a 'pusher', providing the
+    URL of a specific Push Gateway which is configured for that
     application. It also provides the routing token it has acquired from the
-    Push Notification Provider.
- 4. The Home Server starts sending notification HTTP requests to the Push
-    Gateway using the supplied URL. The Push Gateway relays this notification to
-    the Push Notification Provider, passing the routing token along with any
+    Push Provider.
+ 4. The homeserver starts sending HTTP requests to the Push Gateway using the
+    supplied URL. The Push Gateway relays this notification to
+    the Push Provider, passing the routing token along with any
     necessary private credentials the provider requires to send push
     notifications.
- 5. The Push Notification provider sends the notification to the device.
+ 5. The Push Provider sends the notification to the device.
 
 Definitions for terms used in this section are below:
 
-Pusher
-  A 'pusher' is an activity in the Home Server that manages the sending
-  of HTTP notifications for a single device of a single user.
-
-Push Rules
-  A push rule is a single rule, configured by a matrix user, that gives
-  instructions to the Home Server about whether an event should be notified
-  about and how given a set of conditions. Matrix clients allow the user to
-  configure these. They create and view them via the Client to Server REST API.
+Push Provider
+  A push provider is a service managed by the device vendor which can send
+  notifications directly to the device. Google Cloud Messaging (GCM) and Apple
+  Push Notification Service (APNS) are two examples of push providers.
 
 Push Gateway
-  A push gateway is a server that receives HTTP event notifications from Home
-  Servers and passes them on to a different protocol such as APNS for iOS
-  devices or GCM for Android devices. Matrix.org provides a reference push
-  gateway, 'sygnal'. A client app tells a Home Server what push gateway
-  to send notifications to when it sets up a pusher.
+  A push gateway is a server that receives HTTP event notifications from
+  homeservers and passes them on to a different protocol such as APNS for iOS
+  devices or GCM for Android devices. Clients inform the homeserver which
+  Push Gateway to send notifications to when it sets up a Pusher.
 
-For information on the client-server API for setting pushers and push rules, see
-the Client Server API section. For more information on the format of HTTP
-notifications, see the HTTP Notification Protocol section.
+Pusher
+  A pusher is a worker on the homeserver that manages the sending
+  of HTTP notifications for a user. A user can have multiple pushers: one per
+  device.
+
+Push Rule
+  A push rule is a single rule that states under what *conditions* an event should
+  be passed onto a push gateway and *how* the notification should be presented.
+  These rules are stored on the user's homeserver. They are manually configured
+  by the user, who can create and view them via the Client/Server API.
+
+Push Ruleset
+  A push ruleset *scopes a set of rules according to some criteria*. For example,
+  some rules may only be applied for messages from a particular sender,
+  a particular room, or by default. The push ruleset contains the entire set
+  of scopes and rules.
 
 Client behaviour
 ----------------
 
-To receive any notification pokes at all, it is necessary to configure a
-'pusher' on the Home Server that you wish to receive notifications from. There
-is a single API endpoint for this, as described below.
+Clients MUST configure a Pusher before they will receive push notifications.
+There is a single API endpoint for this, as described below.
 
 {{pusher_http_api}}
 
 Push Rules
 ~~~~~~~~~~
-Homeservers have an interface to configure what events trigger notifications.
-This behaviour is configured through "Push Rules". Push Rules come in a variety
-of different kinds and each kind of rule has an associated priority. The
-different kinds of rule, in descending order of priority, are:
-
-Override Rules
+A push rule is a single rule that states under what *conditions* an event should
+be passed onto a push gateway and *how* the notification should be presented.
+There are different "kinds" of push rules and each rule has an associated
+priority. Every push rule MUST have a ``kind`` and ``rule_id``. The ``rule_id``
+is a unique string within the kind of rule and its' scope: ``rule_ids`` do not
+need to be unique between rules of the same kind on different devices. Rules may
+have extra keys depending on the value of ``kind``.The different kinds of rule
+in descending order of priority are:
+
+Override Rules ``override``
   The highest priority rules are user-configured overrides.
-Content Rules
+Content-specific Rules ``content``
   These configure behaviour for (unencrypted) messages that match certain
-  patterns. Content rules take one parameter, 'pattern', that gives the pattern
-  to match against. This is treated in the same way as pattern for event_match
-  conditions, below.
-Room Rules
-  These change the behaviour of all messages to a given room. The rule_id of a
-  room rule is always the ID of the room that it affects.
-Sender
-  These rules configure notification behaviour for messages from a specific,
-  named Matrix user ID. The rule_id of Sender rules is always the Matrix user
+  patterns. Content rules take one parameter: ``pattern``, that gives the glob
+  pattern to match against. This is treated in the same way as pattern for
+  ``event_match`` conditions, below.
+Room-specific Rules ``room``
+  These rules change the behaviour of all messages for a given room. The
+  ``rule_id`` of a room rule is always the ID of the room that it affects.
+Sender-specific rules ``sender``
+  These rules configure notification behaviour for messages from a specific
+  Matrix user ID. The ``rule_id`` of Sender rules is always the Matrix user
   ID of the user whose messages they'd apply to.
-Underride
-  These are identical to override rules, but have a lower priority than content,
-  room and sender rules.
+Underride rules ``underride``
+  These are identical to ``override`` rules, but have a lower priority than
+  ``content``, ``room`` and ``sender`` rules.
 
-In addition, each kind of rule may be either global or device-specific. Device
-specific rules only affect delivery of notifications via pushers with a matching
-``profile_tag``. All device-specific rules are higher priority than all global
-rules. Thusly, the full list of rule kinds, in descending priority order, is as
-follows:
+Push rules may be either global or device-specific. Device specific rules only
+affect delivery of notifications via pushers with a matching ``profile_tag``.
+All device-specific rules have a higher priority than global rules. This means
+that the full list of rule kinds, in descending priority order, is as follows:
 
  * Device-specific Override
  * Device-specific Content
@@ -125,36 +134,30 @@ follows:
  * Global Sender
  * Global Underride
 
-For some kinds of rule, rules of the same kind also have an ordering with
-respect to one another. The kinds that do not are room and sender rules where
-the rules are mutually exclusive by definition and therefore an ordering would
-be redundant. Actions for the highest priority rule and only that rule apply
-(for example, a set_tweak action in a lower priority rule will not apply if a
-higher priority rule matches, even if that rule does not specify any tweaks).
-
-Rules also have an identifier, ``rule_id``, which is a string. The ``rule_id``
-is unique within the kind of rule and scope: ``rule_ids`` need not be unique
-between rules of the same kind on different devices. A home server may also have
-server default rules of each kind and in each scope. Server default rules are
-lower priority than user-defined rules in each scope. Server default rules (and
-only server default rules) begin with a dot ('.') character. In addition, all
-rules may be enabled or disabled. Disabled rules never match.
-
-If no rules match an event, the Home Server should not notify for the message
-(that is to say, the default action is "dont-notify"). Events that the user sent
-themselves are never alerted for.
+Rules with the same ``kind`` can specify an ordering priority. This determines
+which rule is selected in the event of multiple matches. For example, a rule
+matching "tea" and a separate rule matching "time" would both match the sentence
+"It's time for tea". The ordering of the rules would then resolve the tiebreak
+to determine which rule is executed. Only ``actions`` for highest priority rule
+will be sent to the Push Gateway.
+
+Each rule can be enabled or disabled. Disabled rules never match. If no rules
+match an event, the homeserver MUST NOT notify the Push Gateway for that event.
+Homeservers MUST NOT notify the Push Gateway for events that the user has sent
+themselves. 
 
 Predefined Rules
 ++++++++++++++++
-Matrix specifies the following rule IDs for server default rules. Home Servers
-may define rules as follows with the given IDs. If Home Servers provide rules
-with these IDs, their semantics should match those given below:
+Homeservers can specify "server-default rules" which operate at a lower priority
+than "user-defined rules". The ``rule_id`` for all server-default rules MUST
+start with a dot (".") to identify them as "server-default". The following
+server-default rules are specified:
 
-.m.rule.contains_user_name
+``.m.rule.contains_user_name``
   Matches any message whose content is unencrypted and contains the local part
   of the user's Matrix ID, separated by word boundaries.
 
-  Definition (as a content rule)::
+  Definition (as a ``content`` rule)::
 
     {
         "rule_id": ".m.rule.contains_user_name"
@@ -168,11 +171,11 @@ with these IDs, their semantics should match those given below:
         ],
     }
 
-.m.rule.contains_display_name
+``.m.rule.contains_display_name``
   Matches any message whose content is unencrypted and contains the user's
   current display name in the room in which it was sent.
 
-  Definition (this rule can only be an override or underride rule)::
+  Definition (this rule can only be an ``override`` or ``underride`` rule)::
 
     {
         "rule_id": ".m.rule.contains_display_name"
@@ -190,10 +193,10 @@ with these IDs, their semantics should match those given below:
         ],
     }
 
-.m.rule.room_one_to_one
+``.m.rule.room_one_to_one``
   Matches any message sent in a room with exactly two members.
 
-  Definition (this rule can only be an override or underride rule)::
+  Definition (this rule can only be an ``override`` or ``underride`` rule)::
 
     {
         "rule_id": ".m.rule.room_two_members"
@@ -212,9 +215,10 @@ with these IDs, their semantics should match those given below:
         ],
     }
 
-.m.rule.suppress_notices
-  Matches messages with 'msgtype' of 'notice'. This should be an override rule
-  such that, when enabled, it takes priority over content / sender / room rules.
+``.m.rule.suppress_notices``
+  Matches messages with a ``msgtype`` of ``notice``. This should be an
+  ``override`` rule so that it takes priority over ``content`` / ``sender`` /
+  ``room`` rules.
 
   Definition::
 
@@ -232,10 +236,10 @@ with these IDs, their semantics should match those given below:
         ]
     }
   
-.m.rule.fallback
+``.m.rule.fallback``
   Matches any message. Used to define the behaviour of messages that match no
-  other rules. Therefore, if Home Servers define this, it should be the lowest
-  priority underride rule.
+  other rules. If homeservers define this it should be the lowest priority
+  ``underride`` rule.
 
   Definition::
 
@@ -249,24 +253,22 @@ with these IDs, their semantics should match those given below:
 
 Actions
 +++++++
-All rules have an associated list of 'actions'. An action affects if and how a
-notification is delivered for a matching event. This standard defines the
-following actions, although if Home servers wish to support more, they are free
-to do so:
+All rules have an associated list of ``actions``. An action affects if and how a
+notification is delivered for a matching event. The following actions are defined:
 
-notify
+``notify``
   This causes each matching event to generate a notification.
-dont_notify
-  Prevents this event from generating a notification
-coalesce
-  This enables notifications for matching events but activates Home Server
+``dont_notify``
+  This prevents this event from generating a notification
+``coalesce``
+  This enables notifications for matching events but activates homeserver
   specific behaviour to intelligently coalesce multiple events into a single 
-  notification. Not all Home Servers may support this. Those that do not should
-  treat it as the 'notify' action.
-set_tweak
-  Sets an entry in the 'tweaks' dictionary key that is sent in the notification
-  poke. This takes the form of a dictionary with a 'set_tweak' key whose value
-  is the name of the tweak to set. It may also have a 'value' key which is
+  notification. Not all homeservers may support this. Those that do not support
+  it should treat it as the ``notify`` action.
+``set_tweak``
+  Sets an entry in the ``tweaks`` dictionary key that is sent in the notification
+  request. This takes the form of a dictionary with a ``set_tweak`` key whose value
+  is the name of the tweak to set. It may also have a ``value`` key which is
   the value to which it should be set.
 
 Actions that have no parameters are represented as a string. Otherwise, they are
@@ -275,62 +277,60 @@ their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
 
 Tweaks
 ^^^^^^
-The ``set_tweak`` key action is used to add an entry to the 'tweaks' dictionary
-that is sent in the notification poke. The following tweaks are defined:
-
-sound
-  A sound to be played when this notification arrives. 'default' means to
-  play a default sound.
-highlight
-  Whether or not this message should be highlighted in the UI. This will
-  normally take the form of presenting the message in a different colour and/or
-  weight. The UI might also be adjusted to draw particular attention to the room
-  in which the event occurred. The value may be omitted from the highlight
-  tweak, in which case it should be read as if it had a value of true.
-
-Tweaks are passed transparently through the Home Server so client applications
-and push gateways may agree on additional tweaks, for example, how to flash the
-notification light on a mobile device.
-
-If a kind of tweak that a client understands is not specified in an action, the
-client may choose a sensible behaviour for the tweak.
+The ``set_tweak`` action is used to add an entry to the 'tweaks' dictionary
+that is sent in the notification request. The following tweaks are defined:
+
+``sound``
+  A string representing the sound to be played when this notification arrives.
+  A value of ``default`` means to play a default sound.
+``highlight``
+  A boolean representing whether or not this message should be highlighted in
+  the UI. This will normally take the form of presenting the message in a
+  different colour and/or style. The UI might also be adjusted to draw
+  particular attention to the room in which the event occurred. The ``value``
+  may be omitted from the highlight tweak, in which case it should default to
+  ``true``.
+
+Tweaks are passed transparently through the homeserver so client applications
+and Push Gateways may agree on additional tweaks. For example, a tweak may be
+added to specify how to flash the notification light on a mobile device.
 
 Conditions
 ++++++++++
-Override, Underride and Default rules have a list of 'conditions'. All
-conditions must hold true for an event in order for a rule to be applied to an
-event. A rule with no conditions always matches. Matrix specifies the following
-conditions, although if Home Servers wish to support others, they are free to
-do so:
 
-event_match
+Override, Underride and Default Rules MAY have a list of 'conditions'. 
+All conditions must hold true for an event in order to apply the ``action`` for
+the event. A rule with no conditions always matches. Room, Sender, User and
+Content rules do not have conditions in the same way, but instead have
+predefined conditions. These conditions can be configured using the parameters
+outlined below. In the cases of room and sender rules, the ``rule_id`` of the
+rule determines its behaviour. The following conditions are defined:
+
+``event_match``
   This is a glob pattern match on a field of the event. Parameters:
-   * 'key': The dot-separated field of the event to match, e.g. content.body
-   * 'pattern': The glob-style pattern to match against. Patterns with no
-                special glob characters should be treated as having asterisks
-                prepended and appended when testing the condition.
-profile_tag
-  Matches the profile_tag of the device that the notification would be
+   * ``key``: The dot-separated field of the event to match, e.g. ``content.body``
+   * ``pattern``: The glob-style pattern to match against. Patterns with no
+     special glob characters should be treated as having asterisks
+     prepended and appended when testing the condition.
+
+``profile_tag``
+  Matches the ``profile_tag`` of the device that the notification would be
   delivered to. Parameters:
 
-   * 'profile_tag': The profile_tag to match with.
-contains_display_name
-  This matches unencrypted messages where content.body contains the owner's
+   * ``profile_tag``: The profile_tag to match with.
+
+``contains_display_name``
+  This matches unencrypted messages where ``content.body`` contains the owner's
   display name in that room. This is a separate rule because display names may
   change and as such it would be hard to maintain a rule that matched the user's
   display name. This condition has no parameters.
-room_member_count
-  This matches the current number of members in the room.
-   * 'is': A decimal integer optionally prefixed by one of, '==', '<', '>',
-     '>=' or '<='. A prefix of '<' matches rooms where the member count is
-     strictly less than the given number and so forth. If no prefix is present,
-     this matches rooms where the member count is exactly equal to the given
-     number (i.e. the same as '==').
-
-Room, Sender, User and Content rules do not have conditions in the same way,
-but instead have predefined conditions, the behaviour of which can be configured
-using parameters named as described above. In the cases of room and sender
-rules, the rule_id of the rule determines its behaviour.
+
+``room_member_count``
+  This matches the current number of members in the room. Parameters:
+   * ``is``: A decimal integer optionally prefixed by one of, ``==``, ``<``,
+     ``>``, ``>=`` or ``<=``. A prefix of ``<`` matches rooms where the member
+     count is strictly less than the given number and so forth. If no prefix is
+     present, this parameter defaults to ``==``.
 
 Push Rules: API
 ~~~~~~~~~~~~~~~
@@ -396,158 +396,36 @@ Server behaviour
 ----------------
 
 This describes the format used by "HTTP" pushers to send notifications of
-events.
+events. If the endpoint returns an HTTP error code, the homeserver should retry
+for a reasonable amount of time with a reasonable back-off scheme.
 
 {{push_notifier_http_api}}
 
-
-
-Notifications are sent as HTTP POST requests to the URL configured when the
-pusher is created, but Matrix strongly recommends that the path should be::
-
-  /_matrix/push/v1/notify
-
-The body of the POST request is a JSON dictionary. The format
-is as follows::
-
-  {
-    "notification": {
-      "id": "$3957tyerfgewrf384",
-      "room_id": "!slw48wfj34rtnrf:example.com",
-      "type": "m.room.message",
-      "sender": "@exampleuser:matrix.org",
-      "sender_display_name": "Major Tom",
-      "room_name": "Mission Control",
-      "room_alias": "#exampleroom:matrix.org",
-      "prio": "high",
-      "content": {
-        "msgtype": "m.text",
-        "body": "I'm floating in a most peculiar way."
-      }
-     },
-     "counts": {
-       "unread" : 2,
-       "missed_calls": 1
-     }
-     "devices": [
-       {
-         "app_id": "org.matrix.matrixConsole.ios",
-         "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/",
-         "pushkey_ts": 12345678,
-         "data" : {
-         },
-         "tweaks": {
-           "sound": "bing"
-          }
-        }
-      ]
-    }
-  }
-
-The contents of this dictionary are defined as follows:
-
-id
-  An identifier for this notification that may be used to detect duplicate
-  notification requests. This is not necessarily the ID of the event that
-  triggered the notification.
-room_id
-  The ID of the room in which this event occurred.
-type
-  The type of the event as in the event's 'type' field.
-sender
-  The sender of the event as in the corresponding event field.
-sender_display_name
-  The current display name of the sender in the room in which the event
-  occurred.
-room_name
-  The name of the room in which the event occurred.
-room_alias
-  An alias to display for the room in which the event occurred.
-prio
-  The priority of the notification. Acceptable values are 'high' or 'low. If
-  omitted, 'high' is assumed. This may be used by push gateways to deliver less
-  time-sensitive notifications in a way that will preserve battery power on
-  mobile devices.
-content
-  The 'content' field from the event, if present. If the event had no content
-  field, this field is omitted.
-counts
-  This is a dictionary of the current number of unacknowledged communications
-  for the recipient user. Counts whose value is zero are omitted.
-unread
-  The number of unread messages a user has across all of the rooms they are a
-  member of.
-missed_calls
-  The number of unacknowledged missed calls a user has across all rooms of
-  which they are a member.
-device
-  This is an array of devices that the notification should be sent to.
-app_id
-  The app_id given when the pusher was created.
-pushkey
-  The pushkey given when the pusher was created.
-pushkey_ts
-  The unix timestamp (in seconds) when the pushkey was last updated.
-data
-  A dictionary of additional pusher-specific data. For 'http' pushers, this is
-  the data dictionary passed in at pusher creation minus the 'url' key.
-tweaks
-  A dictionary of customisations made to the way this notification is to be
-  presented. These are added by push rules.
-sound
-  Sets the sound file that should be played. 'default' means that a default
-  sound should be played.
-
-And additional key is defined but only present on member events:
-
-user_is_target
-  This is true if the user receiving the notification is the subject of a member
-  event (i.e. the state_key of the member event is equal to the user's Matrix
-  ID).
-
-The recipient of an HTTP notification should respond with an HTTP 2xx response
-when the notification has been processed. If the endpoint returns an HTTP error
-code, the Home Server should retry for a reasonable amount of time with a
-reasonable back-off scheme.
-
-The endpoint should return a JSON dictionary as follows::
-
-  {
-    "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ]
-  }
-
-Whose keys are:
-
-rejected
-  A list of all pushkeys given in the notification request that are not valid.
-  These could have been rejected by an upstream gateway because they have
-  expired or have never been valid. Home Servers must cease sending notification
-  requests for these pushkeys and remove the associated pushers. It may not
-  necessarily be the notification in the request that failed: it could be that
-  a previous notification to the same pushkey failed.
-
 Push Gateway behaviour
 ----------------------
 
 Recommendations for APNS
 ~~~~~~~~~~~~~~~~~~~~~~~~
-For sending APNS notifications, the exact format is flexible and up to the
-client app and its push gateway to agree on (since APNS requires that the sender
-have a private key owned by the app developer, each app must have its own push
-gateway). However, Matrix strongly recommends:
+The exact format for sending APNS notifications is flexible and up to the
+client app and its' push gateway to agree on. As APNS requires that the sender
+has a private key owned by the app developer, each app must have its own push
+gateway. It is recommended that:
 
- * That the APNS token be base64 encoded and used as the pushkey.
- * That a different app_id be used for apps on the production and sandbox
+ * The APNS token be base64 encoded and used as the pushkey.
+ * A different app_id be used for apps on the production and sandbox
    APS environments.
- * That APNS push gateways do not attempt to wait for errors from the APNS
+ * APNS push gateways do not attempt to wait for errors from the APNS
    gateway before returning and instead to store failures and return
    'rejected' responses next time that pushkey is used.
 
 Security considerations
 -----------------------
 
-- Message content shouldn't be sent in the push itself as it will pass through
-  the Push Provider (Google/Apple). Instead, send a ping to tell the client to
-  sync.
-- HTTPS should be used on the Matrix HTTP Notification Protocol.
+Clients specify the Push Gateway URL to use to send event notifications to. This
+URL should be over HTTPS and *never* over HTTP.
+
+As push notifications will pass through a Push Provider, message content
+shouldn't be sent in the push itself where possible. Instead, Push Gateways
+should send a "sync" command to instruct the client to get new events from the
+homeserver directly.
 

From dc3c02aff5ed34686dd62b39fcce72cae8065f5f Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 8 Oct 2015 10:08:38 -0500
Subject: [PATCH 282/467] Review comments

---
 api/client-server/v1/membership.yaml          | 31 +++++++------------
 event-schemas/schema/v1/m.room.member         |  4 +--
 specification/modules/third_party_invites.rst | 29 ++++++++++++-----
 3 files changed, 35 insertions(+), 29 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index ff35fa35..efc82578 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -80,11 +80,6 @@ paths:
         They do not start participating in the room until they actually join the
         room.
 
-        This serves two purposes; firstly, to notify the user that the room
-        exists (and that their presence is requested). Secondly, some rooms can
-        only be joined if a user is invited to join it; sending the invite gives
-        that user permission to join the room.
-
         Only users currently in a particular room can invite other users to
         join that room.
 
@@ -142,20 +137,21 @@ paths:
       description: |-
         *Note that there are two forms of this API, which are documented separately.
         This version of the API does not require that the inviter know the Matrix
-        identifier of the invitee, and instead relies on third party identifiers.*
+        identifier of the invitee, and instead relies on third party identifiers.
+        The homeserver uses an identity server to perform the mapping from
+        third party identifier to a Matrix identifier.*
 
         This API invites a user to participate in a particular room.
         They do not start participating in the room until they actually join the
         room.
 
-        This serves two purposes; firstly, to notify the user that the room
-        exists (and that their presence is requested). Secondly, some rooms can
-        only be joined if a user is invited to join it; sending the invite gives
-        that user permission to join the room.
-
         Only users currently in a particular room can invite other users to
         join that room.
 
+        If the identity server did know the Matrix user identifier for the
+        third party identifier, the home server will append a ``m.room.member``
+        event to the room.
+
         If the identity server does not know a Matrix user identifier for the
         passed third party identifier, the homeserver will issue an invitation
         which can be accepted upon providing proof of ownership of the third
@@ -164,9 +160,9 @@ paths:
         add an ``m.room.third_party_invite`` event into the graph for the room,
         containing that token.
 
-        When a user binds the invited third party identifier to a Matrix user ID,
-        the identity server will give the user a list of pending invitations,
-        each containing:
+        When the invitee binds the invited third party identifier to a Matrix
+        user ID, the identity server will give the user a list of pending
+        invitations, each containing:
 
           - The room ID to which they were invited
 
@@ -176,10 +172,6 @@ paths:
 
           - The matrix user ID who invited them to the room
 
-        If the identity server did know the Matrix user identifier for the
-        third party identifier, the home server will append a ``m.room.member``
-        event to the room.
-
         If a token is requested from the identity server, the home server will
         append a ``m.room.third_party_invite`` event to the room.
       security:
@@ -209,7 +201,8 @@ paths:
                 description: The hostname+port of the identity server which should be used for third party identifier lookups.
               medium:
                 type: string
-                description: The kind of address being passed in the address field.
+                # TODO: Link to identity service spec when it eixsts
+                description: The kind of address being passed in the address field, for example ``email``.
               address:
                 type: string
                 description: The invitee's third party identifier.
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 56104328..912f6cf3 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "The current membership state of a user in the room.",
-    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. The token, public_key, key_validity_url, signature, and sender properties either must all or none be set - they are set if the invite was an ``m.room.third_party_invite`` event, and absent if the invite was an ``m.room.member`` event.",
+    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. The ``third_party_invite`` property will be set if the invite was an ``m.room.third_party_invite`` event, and absent if the invite was an ``m.room.member`` event.",
     "allOf": [{
         "$ref": "core-event-schema/state_event.json"
     }],
@@ -32,7 +32,7 @@
                         },
                         "key_validity_url": {
                             "type": "string",
-                            "description": "A URL which can be fetched, with querystring public_key=public_key, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
+                            "description": "A URL which can be fetched, with querystring ``public_key=public_key``, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
                         },
                         "public_key": {
                             "type": "string",
diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index 3365501e..9b3b52e1 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -8,7 +8,7 @@ Matrix user ID is not known, instead addressing them by a third party identifier
 such as an email address.
 
 There are two flows here; one if a Matrix user ID is known for the third party
-identifier, and one if not. Either way, the client calls /invite with the
+identifier, and one if not. Either way, the client calls ``/invite`` with the
 details of the third party identifier.
 
 The homeserver asks the identity server whether a Matrix user ID is known for
@@ -16,8 +16,8 @@ that identifier. If it is, an invite is simply issued for that user.
 
 If it is not, the homeserver asks the identity server to record the details of
 the invitation, and to notify the client of this pending invitation if it gets
-a binding for this identifier in the future. The identity server returns to the
-homeserver a token, as well as its public key.
+a binding for this identifier in the future. The identity server returns a token
+and public key to the homeserver.
 
 If a client then tries to join the room in the future, it will be allowed to if
 it presents both the token, and a signature of that token from the identity
@@ -43,6 +43,8 @@ If a client of the current homeserver is joining by an
 key used for signing is still valid, by checking ``key_validity_url``. It does
 this by making an HTTP GET request to ``key_validity_url``:
 
+.. TODO: Link to identity server spec when it exists
+
 Schema::
 
     => GET $key_validity_url?public_key=$public_key
@@ -63,14 +65,16 @@ Example::
     }
 
 with the querystring
-?public_key=``public_key``. A JSON object will be returned, and the key is
-considered valid if the object contains a key named ``valid`` whose value is
-``true``. If this cannot be verified, the invitation must be rejected.
+?public_key=``public_key``. A JSON object will be returned.
+The invitation is valid if the object contains a key named ``valid`` which is
+``true``. Otherwise, the invitation MUST be rejected. This request is
+idempotent, and may be retried by the homeserver.
 
 If a homeserver is joining a room for the first time because of an
 ``m.room.third_party_invite``, the server which is already participating in the
-room MUST validate that the public key used for signing is still valid, by
-checking ``key_validity_url`` in the above described way.
+room (which is chosen as per the standard server-server specification) MUST
+validate that the public key used for signing is still valid, by checking
+``key_validity_url`` in the above described way.
 
 No other homeservers may reject the joining of the room on the basis of
 ``key_validity_url``, this is so that all homeservers have a consistent view of
@@ -98,3 +102,12 @@ sign(``token``, ``public_key``) = ``signature`` *and* check ``key_validity_url``
 Having validated these things, H1 writes the join event to the room, and H3
 begins participating in the room. H2 *must* accept this event.
 
+The reason that no other homeserver may reject the event based on checking
+``key_validity_url`` is that we must ensure event acceptance is deterministic.
+If some other participating server doesn't have a network path to the keyserver,
+or if the keyserver were to go offline, or revoke its keys, that other server
+would reject the event and cause the participating servers' graphs to diverge.
+This relies on participating servers trusting each other, but that trust is
+already implied by the server-server protocol. Also, the public key signature
+verification must still be performed, so the attack surface here is minimized.
+

From f2a6950cc3e32193363360653073cf773a5b9de0 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 16:21:26 +0100
Subject: [PATCH 283/467] Minor tweaks; allow objects without props/parents if
 a title is set

This allows us to do things like {Tweaks} where Tweaks is defined somewhere
else.
---
 api/client-server/v1/push_notifier.yaml |  15 +--
 api/client-server/v1/pusher.yaml        |   2 +-
 specification/modules/push.rst          | 139 ++++++++++++------------
 templating/matrix_templates/units.py    |   7 ++
 4 files changed, 80 insertions(+), 83 deletions(-)

diff --git a/api/client-server/v1/push_notifier.yaml b/api/client-server/v1/push_notifier.yaml
index 07c93f3b..be3b5f74 100644
--- a/api/client-server/v1/push_notifier.yaml
+++ b/api/client-server/v1/push_notifier.yaml
@@ -18,7 +18,7 @@ paths:
       description: |-
         This endpoint is invoked by HTTP pushers to notify a push gateway about
         an event.
-        *NB: Notifications are send to the URL configured when the pusher is
+        *NB: Notifications are sent to the URL configured when the pusher is
         created. This means that the HTTP path may be different depending on the
         push gateway.*
       parameters:
@@ -111,10 +111,6 @@ paths:
                   content:
                     type: object
                     title: EventContent
-                    properties:
-                      body:
-                        type: string
-                        description: Message text
                     description: |-
                       The ``content`` field from the event, if present. If the
                       event had no content field, this field is omitted.
@@ -162,21 +158,12 @@ paths:
                         A dictionary of additional pusher-specific data. For
                         'http' pushers, this is the data dictionary passed in at
                         pusher creation minus the ``url`` key.
-                      properties:
-                        foo:
-                          type: string
                     tweaks:
                       type: object
                       title: Tweaks
                       description: |-
                         A dictionary of customisations made to the way this
                         notification is to be presented. These are added by push rules.
-                      properties:
-                        sound:
-                          type: string
-                          description: |-
-                            Sets the sound file that should be played.
-                            ``default`` means that a default sound should be played.
       responses:
         200:
           description: A list of rejected push keys.
diff --git a/api/client-server/v1/pusher.yaml b/api/client-server/v1/pusher.yaml
index 3c9ffe8c..99a70ee7 100644
--- a/api/client-server/v1/pusher.yaml
+++ b/api/client-server/v1/pusher.yaml
@@ -69,7 +69,7 @@ paths:
                 description: |-
                   This is a string that determines what set of device rules will
                   be matched when evaluating push rules for this pusher. It is
-                  an arbitrary string. Multiple devices maybe use the same
+                  an arbitrary string. Multiple devices may use the same
                   ``profile_tag``. It is advised that when an app's data is
                   copied or restored to a different device, this value remain
                   the same. Client apps should offer ways to change the
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index ac156dcc..7948475c 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -5,21 +5,21 @@ Push Notifications
 
 ::
 
-                                   +--------------------+ +-------------------+
-                  Matrix HTTP      |                    | |                   |
-             Notification Protocol |   App Developer    | |   Device Vendor   |
-                                   |                    | |                   |
-           +-------------------+   | +----------------+ | | +---------------+ |
-           |                   |   | |                | | | |               | |
-           | Matrix Home Server+----->  Push Gateway  | +---> Push Provider | |
-           |                   |   | |                | | | |               | |
-           +-^-----------------+   | +----------------+ | | +----+----------+ |
-             |                     |                    | |      |            |
-    Matrix   |                     |                    | |      |            |
- Client/Server API  +              |                    | |      |            |
-             |      |              +--------------------+ +-------------------+
-             |   +--+-+                                          |             
-             |   |    <------------------------------------------+             
+                                   +--------------------+  +-------------------+
+                  Matrix HTTP      |                    |  |                   |
+             Notification Protocol |   App Developer    |  |   Device Vendor   |
+                                   |                    |  |                   |
+           +-------------------+   | +----------------+ |  | +---------------+ |
+           |                   |   | |                | |  | |               | |
+           | Matrix Home Server+----->  Push Gateway  +------> Push Provider | |
+           |                   |   | |                | |  | |               | |
+           +-^-----------------+   | +----------------+ |  | +----+----------+ |
+             |                     |                    |  |      |            |
+    Matrix   |                     |                    |  |      |            |
+ Client/Server API  +              |                    |  |      |            |
+             |      |              +--------------------+  +-------------------+
+             |   +--+-+                                           |             
+             |   |    <-------------------------------------------+             
              +---+    |                                                        
                  |    |          Provider Push Protocol                        
                  +----+                                                        
@@ -28,7 +28,7 @@ Push Notifications
 
 
 This module adds support for push notifications. Homeservers send notifications
-of user events to user-configured HTTP endpoints. Users may also configure a
+of events to user-configured HTTP endpoints. Users may also configure a
 number of rules that determine which events generate notifications. These are
 all stored and managed by the user's homeserver. This allows user-specific push
 settings to be reused between client applications.
@@ -105,8 +105,8 @@ Override Rules ``override``
 Content-specific Rules ``content``
   These configure behaviour for (unencrypted) messages that match certain
   patterns. Content rules take one parameter: ``pattern``, that gives the glob
-  pattern to match against. This is treated in the same way as pattern for
-  ``event_match`` conditions, below.
+  pattern to match against. This is treated in the same way as ``pattern`` for
+  ``event_match``.
 Room-specific Rules ``room``
   These rules change the behaviour of all messages for a given room. The
   ``rule_id`` of a room rule is always the ID of the room that it affects.
@@ -144,7 +144,52 @@ will be sent to the Push Gateway.
 Each rule can be enabled or disabled. Disabled rules never match. If no rules
 match an event, the homeserver MUST NOT notify the Push Gateway for that event.
 Homeservers MUST NOT notify the Push Gateway for events that the user has sent
-themselves. 
+themselves.
+
+Actions
++++++++
+All rules have an associated list of ``actions``. An action affects if and how a
+notification is delivered for a matching event. The following actions are defined:
+
+``notify``
+  This causes each matching event to generate a notification.
+``dont_notify``
+  This prevents each matching event from generating a notification
+``coalesce``
+  This enables notifications for matching events but activates homeserver
+  specific behaviour to intelligently coalesce multiple events into a single 
+  notification. Not all homeservers may support this. Those that do not support
+  it should treat it as the ``notify`` action.
+``set_tweak``
+  Sets an entry in the ``tweaks`` dictionary key that is sent in the notification
+  request to the Push Gateway. This takes the form of a dictionary with a
+  ``set_tweak`` key whose value is the name of the tweak to set. It may also
+  have a ``value`` key which is the value to which it should be set.
+
+Actions that have no parameters are represented as a string. Otherwise, they are
+represented as a dictionary with a key equal to their name and other keys as
+their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
+
+Tweaks
+^^^^^^
+The ``set_tweak`` action is used to add an entry to the 'tweaks' dictionary
+that is sent in the notification request to the Push Gateway. The following
+tweaks are defined:
+
+``sound``
+  A string representing the sound to be played when this notification arrives.
+  A value of ``default`` means to play a default sound.
+``highlight``
+  A boolean representing whether or not this message should be highlighted in
+  the UI. This will normally take the form of presenting the message in a
+  different colour and/or style. The UI might also be adjusted to draw
+  particular attention to the room in which the event occurred. The ``value``
+  may be omitted from the highlight tweak, in which case it should default to
+  ``true``.
+
+Tweaks are passed transparently through the homeserver so client applications
+and Push Gateways may agree on additional tweaks. For example, a tweak may be
+added to specify how to flash the notification light on a mobile device.
 
 Predefined Rules
 ++++++++++++++++
@@ -251,49 +296,7 @@ server-default rules are specified:
         ],
     }
 
-Actions
-+++++++
-All rules have an associated list of ``actions``. An action affects if and how a
-notification is delivered for a matching event. The following actions are defined:
-
-``notify``
-  This causes each matching event to generate a notification.
-``dont_notify``
-  This prevents this event from generating a notification
-``coalesce``
-  This enables notifications for matching events but activates homeserver
-  specific behaviour to intelligently coalesce multiple events into a single 
-  notification. Not all homeservers may support this. Those that do not support
-  it should treat it as the ``notify`` action.
-``set_tweak``
-  Sets an entry in the ``tweaks`` dictionary key that is sent in the notification
-  request. This takes the form of a dictionary with a ``set_tweak`` key whose value
-  is the name of the tweak to set. It may also have a ``value`` key which is
-  the value to which it should be set.
-
-Actions that have no parameters are represented as a string. Otherwise, they are
-represented as a dictionary with a key equal to their name and other keys as
-their parameters, e.g. ``{ "set_tweak": "sound", "value": "default" }``
-
-Tweaks
-^^^^^^
-The ``set_tweak`` action is used to add an entry to the 'tweaks' dictionary
-that is sent in the notification request. The following tweaks are defined:
 
-``sound``
-  A string representing the sound to be played when this notification arrives.
-  A value of ``default`` means to play a default sound.
-``highlight``
-  A boolean representing whether or not this message should be highlighted in
-  the UI. This will normally take the form of presenting the message in a
-  different colour and/or style. The UI might also be adjusted to draw
-  particular attention to the room in which the event occurred. The ``value``
-  may be omitted from the highlight tweak, in which case it should default to
-  ``true``.
-
-Tweaks are passed transparently through the homeserver so client applications
-and Push Gateways may agree on additional tweaks. For example, a tweak may be
-added to specify how to flash the notification light on a mobile device.
 
 Conditions
 ++++++++++
@@ -349,14 +352,14 @@ To create a rule that suppresses notifications for the room with ID
   curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \
   '{
      "actions" : ["dont_notify"]
-   }' 
+   }'
 
 To suppress notifications for the user ``@spambot:matrix.org``::
 
   curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \
   '{
      "actions" : ["dont_notify"]
-   }' 
+   }'
 
 To always notify for messages that contain the work 'cake' and set a specific
 sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``)::
@@ -365,7 +368,7 @@ sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``)::
   '{
      "pattern": "cake",
      "actions" : ["notify", {"set_sound":"cakealarm.wav"}]
-   }' 
+   }'
 
 To add a rule suppressing notifications for messages starting with 'cake' but
 ending with 'lie', superseding the previous rule::
@@ -374,7 +377,7 @@ ending with 'lie', superseding the previous rule::
   '{
      "pattern": "cake*lie",
      "actions" : ["notify"]
-   }' 
+   }'
 
 To add a custom sound for notifications messages containing the word 'beer' in
 any rooms with 10 members or fewer (with greater importance than the room,
@@ -390,14 +393,14 @@ sender and content rules)::
        "notify",
        {"set_sound":"beeroclock.wav"}
      ]
-   }' 
+   }'
 
 Server behaviour
 ----------------
 
 This describes the format used by "HTTP" pushers to send notifications of
-events. If the endpoint returns an HTTP error code, the homeserver should retry
-for a reasonable amount of time with a reasonable back-off scheme.
+events to Push Gateways. If the endpoint returns an HTTP error code, the
+homeserver SHOULD retry for a reasonable amount of time using exponential-backoff.
 
 {{push_notifier_http_api}}
 
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 83376fba..bbfa8423 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -61,6 +61,13 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                 if pretty_key:
                     props[pretty_key] = props[key_name]
                     del props[key_name]
+    if not props and not parents:
+        # Sometimes you just want to specify that a thing is an object without
+        # doing all the keys. Allow people to do that if they set a 'title'.
+        if obj.get("title"):
+            parents = [{
+                "$ref": obj.get("title")
+            }]
     if not props and not parents:
         raise Exception(
             "Object %s has no properties or parents." % obj

From 213cb252900ea43542890fa87bb6a81d9c6e0c27 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 8 Oct 2015 16:38:41 +0100
Subject: [PATCH 284/467] Wrap correctly

---
 templating/matrix_templates/templates/http-api.tmpl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 787f02ca..d1633513 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -40,7 +40,7 @@ Response format:
 {# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
 {#      It also needs to then wrap inside the desc col (42 ch width) -#}
 {# -#}
-{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(40,row.req_str | indent(26 - (row.type|length))) |indent_block(50)}}
+{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(26 - (row.type|length))) |indent_block(50)}}
 {% endfor -%}
 ======================= ========================= ==========================================
 

From 241096dc811cbbaa544d4a4aac73acdeb548aca4 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 8 Oct 2015 17:25:38 +0100
Subject: [PATCH 285/467] Add a flag to initial sync to indicate we want rooms
 that the user has left

---
 api/client-server/v1/sync.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 8fe30056..de8dfec9 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -39,6 +39,16 @@ paths:
           description: The maximum time in milliseconds to wait for an event.
           required: false
           x-example: "35000"
+        - in: query
+          type: string
+          name: archived
+          description: |-
+            Whether to include rooms that the user has left. If absent then
+            only rooms that the user has been invited to or has joined are
+            included. If set to "1" then rooms that the user has left are
+            included as well.
+          required: false
+          x-example: "1"
       responses:
         200:
           description: "The events received, which may be none."

From d42c3195e6f709b0d27e42367b7e0635f00f626f Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 8 Oct 2015 18:06:52 +0100
Subject: [PATCH 286/467] Use 'true' rather than '1' for archived flag

---
 api/client-server/v1/sync.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index de8dfec9..7c1d43f3 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -45,10 +45,10 @@ paths:
           description: |-
             Whether to include rooms that the user has left. If absent then
             only rooms that the user has been invited to or has joined are
-            included. If set to "1" then rooms that the user has left are
+            included. If set to "true" then rooms that the user has left are
             included as well.
           required: false
-          x-example: "1"
+          x-example: "true"
       responses:
         200:
           description: "The events received, which may be none."

From 51ab51ae26ba9e5e469e4594e7a579f8fe434c09 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Thu, 8 Oct 2015 19:27:08 +0100
Subject: [PATCH 287/467] Only documents in matrix-doc!

---
 supporting-docs/_config.yml                   |  16 -
 supporting-docs/_includes/footer.html         |  25 -
 supporting-docs/_includes/head.html           |  31 --
 supporting-docs/_includes/nav.html            |  33 --
 supporting-docs/_layouts/default.html         |  23 -
 supporting-docs/_layouts/page.html            |  14 -
 supporting-docs/_layouts/post.html            |   6 -
 .../_plugins/jekyll-rst-master/.gitignore     |   1 -
 .../_plugins/jekyll-rst-master/LICENSE.txt    |  20 -
 .../_plugins/jekyll-rst-master/README.rst     |  97 ----
 .../_plugins/jekyll-rst-master/converter.rb   |  30 -
 .../_plugins/jekyll-rst-master/directives.py  |  97 ----
 .../_plugins/jekyll-rst-master/rst2html.py    |  39 --
 .../_plugins/jekyll-rst-master/transform.py   |  42 --
 .../_plugins/project_version_tag.rb           |  51 --
 supporting-docs/_posts/index.html             |  18 -
 supporting-docs/_sass/_base.scss              | 204 -------
 supporting-docs/_sass/_layout.scss            | 236 --------
 .../_sass/_syntax-highlighting.scss           |  67 ---
 supporting-docs/about.md                      |  11 -
 supporting-docs/css/basic.css                 | 512 ------------------
 supporting-docs/css/faq.css                   |  40 --
 supporting-docs/css/main.scss                 |  52 --
 supporting-docs/css/nature.css                | 285 ----------
 supporting-docs/css/site_overrides.css        |  62 ---
 supporting-docs/feed.xml                      |  30 -
 .../2015-08-10-client-server.rst              |   0
 .../2015-08-14-getting_involved.md            |   0
 .../{_posts => guides}/2015-08-19-faq.md      |   0
 .../2015-08-21-application_services.md        |   0
 supporting-docs/index.html                    |  14 -
 31 files changed, 2056 deletions(-)
 delete mode 100644 supporting-docs/_config.yml
 delete mode 100644 supporting-docs/_includes/footer.html
 delete mode 100644 supporting-docs/_includes/head.html
 delete mode 100644 supporting-docs/_includes/nav.html
 delete mode 100644 supporting-docs/_layouts/default.html
 delete mode 100644 supporting-docs/_layouts/page.html
 delete mode 100644 supporting-docs/_layouts/post.html
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/.gitignore
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/README.rst
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/converter.rb
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/directives.py
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/rst2html.py
 delete mode 100644 supporting-docs/_plugins/jekyll-rst-master/transform.py
 delete mode 100644 supporting-docs/_plugins/project_version_tag.rb
 delete mode 100644 supporting-docs/_posts/index.html
 delete mode 100644 supporting-docs/_sass/_base.scss
 delete mode 100644 supporting-docs/_sass/_layout.scss
 delete mode 100644 supporting-docs/_sass/_syntax-highlighting.scss
 delete mode 100644 supporting-docs/about.md
 delete mode 100644 supporting-docs/css/basic.css
 delete mode 100644 supporting-docs/css/faq.css
 delete mode 100755 supporting-docs/css/main.scss
 delete mode 100644 supporting-docs/css/nature.css
 delete mode 100644 supporting-docs/css/site_overrides.css
 delete mode 100644 supporting-docs/feed.xml
 rename supporting-docs/{_posts => guides}/2015-08-10-client-server.rst (100%)
 rename supporting-docs/{_posts => guides}/2015-08-14-getting_involved.md (100%)
 rename supporting-docs/{_posts => guides}/2015-08-19-faq.md (100%)
 rename supporting-docs/{_posts => guides}/2015-08-21-application_services.md (100%)
 delete mode 100644 supporting-docs/index.html

diff --git a/supporting-docs/_config.yml b/supporting-docs/_config.yml
deleted file mode 100644
index 1b3ea192..00000000
--- a/supporting-docs/_config.yml
+++ /dev/null
@@ -1,16 +0,0 @@
-# Site settings
-title: Matrix
-email: webmaster@matrix.org
-description: > # this means to ignore newlines until "baseurl:"
-  Matrix.org documentation
-baseurl: "/docs/guides" # the subpath of your site, e.g. /blog/
-url: "http://matrix.org" # the base hostname & protocol for your site
-twitter_username: matrixdotorg
-github_username:  matrix-org
-
-# Build settings
-markdown: kramdown
-
-#defaults:
-#permalink: /:categories/:title.html # can use this when/if we use jekyll for all docs
-permalink: :title.html
diff --git a/supporting-docs/_includes/footer.html b/supporting-docs/_includes/footer.html
deleted file mode 100644
index 3e020c64..00000000
--- a/supporting-docs/_includes/footer.html
+++ /dev/null
@@ -1,25 +0,0 @@
-  <div id="footer"><div id="footerContent">&copy 2014-2015 Matrix.org</div></div>
-	<script>
-  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
-  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
-  ga('create', 'UA-54779209-1', 'auto');
-  ga('send', 'pageview');
-</script>
-<script type='text/javascript'>
-/* <![CDATA[ */
-var tocplus = {"smooth_scroll":"1"};
-/* ]]> */
-</script>
-<script type='text/javascript' src='https://matrix.org/blog/wp-content/plugins/table-of-contents-plus/front.min.js?ver=1509'></script>
-<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/jquery.fitvids.js?ver=2.2'></script>
-<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/waypoints.min.js?ver=2.2'></script>
-<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/jquery.magnific-popup.js?ver=2.2'></script>
-<script type='text/javascript'>
-/* <![CDATA[ */
-var et_custom = {"ajaxurl":"https:\/\/matrix.org\/blog\/wp-admin\/admin-ajax.php","images_uri":"https:\/\/matrix.org\/blog\/wp-content\/themes\/Divi-child\/images","et_load_nonce":"dc61fa2f94","subscription_failed":"Please, check the fields below to make sure you entered the correct information.","fill":"Fill","field":"field","invalid":"Invalid email","captcha":"Captcha","prev":"Prev","next":"Next"};
-/* ]]> */
-</script>
-<script type='text/javascript' src='https://matrix.org/blog/wp-content/themes/Divi-child/js/custom.js?ver=2.2'></script>
diff --git a/supporting-docs/_includes/head.html b/supporting-docs/_includes/head.html
deleted file mode 100644
index 213e1796..00000000
--- a/supporting-docs/_includes/head.html
+++ /dev/null
@@ -1,31 +0,0 @@
-  <meta charset="utf-8">
-  <meta http-equiv="X-UA-Compatible" content="IE=edge">
-  <meta name="viewport" content="width=device-width, initial-scale=1">
-
-  <title>{% if page.title %}{{ page.title }}{% else %}{{ site.title }}{% endif %}</title>
-  <meta name="description" content="{% if page.excerpt %}{{ page.excerpt | strip_html | strip_newlines | truncate: 160 }}{% else %}{{ site.description }}{% endif %}">
-
-  <link rel="stylesheet" href="http://matrix.org/site.css">
-  <link rel="stylesheet" href="{{ "/css/main.css" | prepend: site.baseurl }}">
-
-  <link rel="stylesheet" href="{{ "/css/site_overrides.css" | prepend: site.baseurl }}">
-  <link rel="stylesheet" href="{{ "/css/basic.css" | prepend: site.baseurl }}">
-  <link rel="stylesheet" href="{{ "/css/nature.css" | prepend: site.baseurl }}">
-
-  <link rel='stylesheet' id='toc-screen-css'  href='https://matrix.org/blog/wp-content/plugins/table-of-contents-plus/screen.min.css?ver=1509' type='text/css' media='all' />
-  <link rel='stylesheet' id='divi-fonts-css'  href='https://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,800italic,400,300,700,800&#038;subset=latin,latin-ext' type='text/css' media='all' />
-  <link rel='stylesheet' id='divi-style-css'  href='https://matrix.org/blog/wp-content/themes/Divi-child/style.css?ver=2.2' type='text/css' media='all' />
-
-  <link rel="canonical" href="{{ page.url | replace:'index.html','' | prepend: site.baseurl | prepend: site.url }}">
-  <link rel="alternate" type="application/rss+xml" title="{{ site.title }}" href="{{ "/feed.xml" | prepend: site.baseurl | prepend: site.url }}" />
-  
-<script>
-  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
-  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
-  ga('create', 'UA-54779209-1', 'auto');
-  ga('send', 'pageview');
-</script>
-
diff --git a/supporting-docs/_includes/nav.html b/supporting-docs/_includes/nav.html
deleted file mode 100644
index b7e2d629..00000000
--- a/supporting-docs/_includes/nav.html
+++ /dev/null
@@ -1,33 +0,0 @@
-<header id="main-header" class="et_nav_text_color_dark et-fixed-header">
-  <div class="container clearfix" >
-    <a href="https://matrix.org/blog/">
-      <img src="https://matrix.org/blog/wp-content/uploads/2015/01/logo1.png" alt="Matrix.org" id="logo" />
-    </a>
-    <div id="et-top-navigation">
-      <nav id="top-menu-nav">
-        <ul id="top-menu" class="nav"><li id="menu-item-17" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-17"><a href="http://matrix.org/">Home</a></li>
-<li id="menu-item-794" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-794"><a href="https://matrix.org/blog/try-matrix-now">Try Matrix Now!</a></li>
-<li id="menu-item-348" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-348"><a href="/docs/howtos">HOWTOs</a></li>
-<li id="menu-item-349" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-349"><a href="/docs/spec">Spec</a></li>
-<li id="menu-item-350" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-350"><a href="/docs/api">APIs</a></li>
-<li id="menu-item-351" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-351"><a href="/code">Code</a></li>
-<li id="menu-item-352" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-352"><a href="/jira">JIRA</a></li>
-<li id="menu-item-347" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-347"><a href="https://matrix.org/blog/faq/">FAQ</a></li>
-<li id="menu-item-353" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-353"><a href="/blog">Blog</a></li>
-        </ul>
-      </nav>
-      <div id="et_top_search">
-        <span id="et_search_icon"></span>
-        <form role="search" method="get" class="et-search-form et-hidden" action="https://matrix.org/blog/">
-          <input type="search" class="et-search-field" placeholder="Search &hellip;" value="" name="s" title="Search for:" />
-        </form>
-      </div>
-      <div id="et_mobile_nav_menu">
-        <a href="#" class="mobile_nav closed">
-          <span class="select_page">Select Page</span>
-          <span class="mobile_menu_bar"></span>
-        </a>
-      </div>
-    </div> <!-- #et-top-navigation -->
-  </div> <!-- .container -->
-</header> <!-- #main-header -->
diff --git a/supporting-docs/_layouts/default.html b/supporting-docs/_layouts/default.html
deleted file mode 100644
index da18483e..00000000
--- a/supporting-docs/_layouts/default.html
+++ /dev/null
@@ -1,23 +0,0 @@
-<!DOCTYPE html>
-<html>
-  <head>
-  {% include head.html %}
-
-  </head>
-  <body class="blog et_fixed_nav et_cover_background linux et_right_sidebar chrome">
-
-    <div class="page-content" id="page-container" style="padding-top: 78px;">
-      {% include nav.html %}
-      <div id="main-content">
-
-      <div class="wrapper" id="wrapper">
-        <div class="document_foo" id="document">
-      {{ content }}
-        </div>
-      </div>
-      <div class="push"></div>
-    </div>
-    {% include footer.html %}
-  </body>
-
-</html>
diff --git a/supporting-docs/_layouts/page.html b/supporting-docs/_layouts/page.html
deleted file mode 100644
index 74c1a118..00000000
--- a/supporting-docs/_layouts/page.html
+++ /dev/null
@@ -1,14 +0,0 @@
----
-layout: default
----
-<div class="post">
-
-  <header class="post-header">
-    <h1 class="post-title">{{ page.title }}</h1>
-  </header>
-
-  <article class="post-content">
-    {{ content }}
-  </article>
-
-</div>
diff --git a/supporting-docs/_layouts/post.html b/supporting-docs/_layouts/post.html
deleted file mode 100644
index 64fafcbc..00000000
--- a/supporting-docs/_layouts/post.html
+++ /dev/null
@@ -1,6 +0,0 @@
----
-layout: default
----
-
-    {{ content }}
-
diff --git a/supporting-docs/_plugins/jekyll-rst-master/.gitignore b/supporting-docs/_plugins/jekyll-rst-master/.gitignore
deleted file mode 100644
index 7e99e367..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-*.pyc
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt b/supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt
deleted file mode 100644
index 988ac9eb..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/LICENSE.txt
+++ /dev/null
@@ -1,20 +0,0 @@
-The MIT License (MIT)
-Copyright (c) 2011 Greg Thornton, http://xdissent.com
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/README.rst b/supporting-docs/_plugins/jekyll-rst-master/README.rst
deleted file mode 100644
index 48c21f42..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/README.rst
+++ /dev/null
@@ -1,97 +0,0 @@
-Overview
-========
-
-This plugin adds `ReStructuredText`_ support to `Jekyll`_ and `Octopress`_. 
-It renders ReST in posts and pages, and provides a custom directive to
-support Octopress-compatible syntax highlighting.
-
-Requirements
-============
-
-* Jekyll *or* Octopress >= 2.0
-* Docutils
-* Pygments
-* `RbST`_
-
-Installation
-============
-
-1. Install Docutils and Pygments. 
-
-   The most convenient way is to use virtualenv_burrito:
-
-   ::
-
-      $ curl -s https://raw.github.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh | bash
-      $ source /Users/xdissent/.venvburrito/startup.sh
-      $ mkvirtualenv jekyll-rst
-      $ pip install docutils pygments
-
-2. Install RbST.
-
-   If you use `bundler`_ with Octopress, add ``gem 'RbST'`` to 
-   your ``Gemfile`` in the ``development`` group, then run 
-   ``bundle install``. Otherwise, ``gem install RbST``.
-
-3. Install the plugin.
-
-   For Jekyll:
-
-   ::
-
-      $ cd <jekyll-project-path>
-      $ git submodule add https://github.com/xdissent/jekyll-rst.git _plugins/jekyll-rst
-
-   For Octopress:
-
-   ::
-
-      $ cd <octopress-project-path>
-      $ git submodule add https://github.com/xdissent/jekyll-rst.git plugins/jekyll-rst
-
-4. Start blogging in ReStructuredText. Any file with the ``.rst`` extension
-   will be parsed as ReST and rendered into HTML.
-
-   .. note:: Be sure to activate the ``jekyll-rst`` virtualenv before generating
-      the site by issuing a ``workon jekyll-rst``. I suggest you follow `Harry
-      Marr's advice`_ and create a ``.venv`` file that will  automatically 
-      activate the ``jekyll-rst`` virtualenv when you ``cd`` into your project.
-
-Source Code Highlighting
-========================
-
-A ``code-block`` ReST directive is registered and aliased as ``sourcecode``. 
-It adds syntax highlighting to code blocks in your documents::
-
-   .. code-block:: ruby
-      
-      # Output "I love ReST"
-      say = "I love ReST"
-      puts say
-
-Optional arguments exist to supply a caption, link, and link title::
-
-   .. code-block:: console
-      :caption: Read Hacker News on a budget
-      :url: http://news.ycombinator.com
-      :title: Hacker News
-
-      $ curl http://news.ycombinator.com | less
-
-Octopress already includes style sheets for syntax highlighting, but you'll
-need to generate one yourself if using Jekyll::
-
-   $ pygmentize -S default -f html > css/pygments.css
-
-Octopress Tips
-==============
-
-* Use ``.. more`` in your ReST documents to indicate where Octopress's 
-  ``excerpt`` tag should split your content for summary views.
-
-.. _ReStructuredText: http://docutils.sourceforge.net/rst.html
-.. _Jekyll: http://jekyllrb.com/
-.. _Octopress: http://octopress.com/
-.. _RbST: http://rubygems.org/gems/RbST
-.. _bundler: http://gembundler.com/
-.. _Harry Marr's advice: http://hmarr.com/2010/jan/19/making-virtualenv-play-nice-with-git/
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/converter.rb b/supporting-docs/_plugins/jekyll-rst-master/converter.rb
deleted file mode 100644
index 64b639cc..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/converter.rb
+++ /dev/null
@@ -1,30 +0,0 @@
-require 'rbst'
-
-module Jekyll
-  class RestConverter < Converter
-    safe true
-
-    priority :low
-
-    def matches(ext)
-      ext =~ /rst/i
-    end 
-
-    def output_ext(ext)
-      ".html"
-    end
-
-    def convert(content)
-      RbST.executables = {:html => "#{File.expand_path(File.dirname(__FILE__))}/rst2html.py"}
-      RbST.new(content).to_html(:part => :fragment, :initial_header_level => 2)
-    end
-  end
-
-  module Filters
-    def restify(input)
-      site = @context.registers[:site]
-      converter = site.getConverterImpl(Jekyll::RestConverter)
-      converter.convert(input)
-    end
-  end
-end  
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/directives.py b/supporting-docs/_plugins/jekyll-rst-master/directives.py
deleted file mode 100644
index bfd2b78b..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/directives.py
+++ /dev/null
@@ -1,97 +0,0 @@
-# Define a new directive `code-block` (aliased as `sourcecode`) that uses the 
-# `pygments` source highlighter to render code in color. 
-#
-# Incorporates code from the `Pygments`_ documentation for `Using Pygments in 
-# ReST documents`_ and `Octopress`_.
-#
-# .. _Pygments: http://pygments.org/
-# .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/
-# .. _Octopress: http://octopress.org/
-
-import re
-import os
-import md5
-import __main__
-
-# Absolute path to pygments cache dir
-PYGMENTS_CACHE_DIR = os.path.abspath(os.path.join(os.path.dirname(__main__.__file__), '../../.pygments-cache'))
-
-# Ensure cache dir exists
-if not os.path.exists(PYGMENTS_CACHE_DIR):
-    os.makedirs(PYGMENTS_CACHE_DIR)
-
-from pygments.formatters import HtmlFormatter
-
-from docutils import nodes
-from docutils.parsers.rst import directives, Directive
-
-from pygments import highlight
-from pygments.lexers import get_lexer_by_name, TextLexer
-
-class Pygments(Directive):
-    """ Source code syntax hightlighting.
-    """
-    required_arguments = 1
-    optional_arguments = 0
-    final_argument_whitespace = True
-    string_opts = ['title', 'url', 'caption']
-    option_spec = dict([(key, directives.unchanged) for key in string_opts])
-    has_content = True
-
-    def run(self):
-        self.assert_has_content()
-        try:
-            lexer_name = self.arguments[0]
-            lexer = get_lexer_by_name(lexer_name)
-        except ValueError:
-            # no lexer found - use the text one instead of an exception
-            lexer_name = 'text'
-            lexer = TextLexer()
-        formatter = HtmlFormatter()
-
-        # Construct cache filename
-        cache_file = None
-        content_text = u'\n'.join(self.content)
-        cache_file_name = '%s-%s.html' % (lexer_name, md5.new(content_text).hexdigest())
-        cached_path = os.path.join(PYGMENTS_CACHE_DIR, cache_file_name)
-
-        # Look for cached version, otherwise parse
-        if os.path.exists(cached_path):
-            cache_file = open(cached_path, 'r')
-            parsed = cache_file.read()
-        else:
-            parsed = highlight(content_text, lexer, formatter)
-
-        # Strip pre tag and everything outside it
-        pres = re.compile("<pre>(.+)<\/pre>", re.S)
-        stripped = pres.search(parsed).group(1)
-
-        # Create tabular code with line numbers
-        table = '<div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers">'
-        lined = ''
-        for idx, line in enumerate(stripped.splitlines(True)):
-            table += '<span class="line-number">%d</span>\n' % (idx + 1)
-            lined  += '<span class="line">%s</span>' % line
-        table += '</pre></td><td class="code"><pre><code class="%s">%s</code></pre></td></tr></table></div>' % (lexer_name, lined)
-
-        # Add wrapper with optional caption and link
-        code = '<figure class="code">'
-        if self.options:
-            caption = ('<span>%s</span>' % self.options['caption']) if 'caption' in self.options else ''
-            title = self.options['title'] if 'title' in self.options else 'link'
-            link = ('<a href="%s">%s</a>' % (self.options['url'], title)) if 'url' in self.options else ''
-
-            if caption or link:
-                code += '<figcaption>%s %s</figcaption>' % (caption, link)
-        code += '%s</figure>' % table
-
-        # Write cache
-        if cache_file is None:
-            cache_file = open(cached_path, 'w')
-            cache_file.write(parsed)
-        cache_file.close()
-
-        return [nodes.raw('', code, format='html')]
-
-directives.register_directive('code-block', Pygments)
-directives.register_directive('sourcecode', Pygments)
\ No newline at end of file
diff --git a/supporting-docs/_plugins/jekyll-rst-master/rst2html.py b/supporting-docs/_plugins/jekyll-rst-master/rst2html.py
deleted file mode 100644
index 0190da05..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/rst2html.py
+++ /dev/null
@@ -1,39 +0,0 @@
-#!/usr/bin/env python
-
-# :Author: David Goodger, the Pygments team, Guenter Milde
-# :Date: $Date: $
-# :Copyright: This module has been placed in the public domain.
-
-# This is a merge of the `Docutils`_ `rst2html` front end with an extension
-# suggestion taken from the `Pygments`_ documentation, reworked specifically
-# for `Octopress`_.
-#
-# .. _Pygments: http://pygments.org/
-# .. _Docutils: http://docutils.sourceforge.net/
-# .. _Octopress: http://octopress.org/
-
-"""
-A front end to docutils, producing HTML with syntax colouring using pygments
-"""
-
-try:
-    import locale
-    locale.setlocale(locale.LC_ALL, '')
-except:
-    pass
-
-from transform import transform
-from docutils.writers.html4css1 import Writer
-from docutils.core import default_description
-from directives import Pygments
-
-description = ('Generates (X)HTML documents from standalone reStructuredText '
-               'sources. Uses `pygments` to colorize the content of'
-               '"code-block" directives. Needs an adapted stylesheet' 
-               + default_description)
-
-def main():
-    return transform(writer=Writer(), part='html_body')
-
-if __name__ == '__main__':
-    print(main())
diff --git a/supporting-docs/_plugins/jekyll-rst-master/transform.py b/supporting-docs/_plugins/jekyll-rst-master/transform.py
deleted file mode 100644
index 51ad4ef6..00000000
--- a/supporting-docs/_plugins/jekyll-rst-master/transform.py
+++ /dev/null
@@ -1,42 +0,0 @@
-import sys
-from docutils.core import publish_parts
-from optparse import OptionParser
-from docutils.frontend import OptionParser as DocutilsOptionParser
-from docutils.parsers.rst import Parser
-
-def transform(writer=None, part=None):
-    p = OptionParser(add_help_option=False)
-    
-    # Collect all the command line options
-    docutils_parser = DocutilsOptionParser(components=(writer, Parser()))
-    for group in docutils_parser.option_groups:
-        p.add_option_group(group.title, None).add_options(group.option_list)
-    
-    p.add_option('--part', default=part)
-    
-    opts, args = p.parse_args()
-    
-    settings = dict({
-        'file_insertion_enabled': False,
-        'raw_enabled': False,
-        'doctitle_xform': False,
-    })
-    
-    if len(args) == 1:
-        try:
-            content = open(args[0], 'r').read()
-        except IOError:
-            content = args[0]
-    else:
-        content = sys.stdin.read()
-    
-    parts = publish_parts(
-        source=content,
-        writer=writer,
-        settings_overrides=settings
-    )
-    if 'html_body' in parts:
-      return parts['html_body']
-    if opts.part in parts:
-        return parts[opts.part]
-    return ''
diff --git a/supporting-docs/_plugins/project_version_tag.rb b/supporting-docs/_plugins/project_version_tag.rb
deleted file mode 100644
index c647eef8..00000000
--- a/supporting-docs/_plugins/project_version_tag.rb
+++ /dev/null
@@ -1,51 +0,0 @@
-module Jekyll
-  class ProjectVersionTag < Liquid::Tag
-    NO_GIT_MESSAGE = 'Oops, are you sure this is a git project?'
-    UNABLE_TO_PARSE_MESSAGE = 'Sorry, could not read project version at the moment'
-
-    def render(context)
-      if git_repo?
-        current_version.chomp
-      else
-        NO_GIT_MESSAGE
-      end
-    end
-
-    private
-
-    def current_version
-      @_current_version ||= begin
-        # attempt to find the latest tag, falling back to last commit
-        version = git_describe || parse_head
-
-        version || UNABLE_TO_PARSE_MESSAGE
-      end
-    end
-
-    def git_describe
-      tagged_version = %x{ git describe --tags --always }
-
-      if command_succeeded?
-        tagged_version
-      end
-    end
-
-    def parse_head
-      head_commitish = %x{ git rev-parse --short HEAD }
-
-      if command_succeeded?
-        head_commitish
-      end
-    end
-
-    def command_succeeded?
-      !$?.nil? && $?.success?
-    end
-
-    def git_repo?
-      system('git rev-parse')
-    end
-  end
-end
-
-Liquid::Template.register_tag('project_version', Jekyll::ProjectVersionTag)
diff --git a/supporting-docs/_posts/index.html b/supporting-docs/_posts/index.html
deleted file mode 100644
index 15c7d5c2..00000000
--- a/supporting-docs/_posts/index.html
+++ /dev/null
@@ -1,18 +0,0 @@
----
-layout: default
----
-  {% include basic.css %}
-  {% include nature.css %}
-
-<div class="post">
-
-  <header class="post-header">
-    <div class="post-title">{{ page.title }}</div>
-    <p class="post-meta">{{ page.date | date: "%b %-d, %Y" }}{% if page.author %} • {{ page.author }}{% endif %}{% if page.meta %} • {{ page.meta }}{% endif %}</p>
-  </header>
-
-  <article class="post-content">
-    {{ content }}
-  </article>
-
-</div>
diff --git a/supporting-docs/_sass/_base.scss b/supporting-docs/_sass/_base.scss
deleted file mode 100644
index e5fd0fd8..00000000
--- a/supporting-docs/_sass/_base.scss
+++ /dev/null
@@ -1,204 +0,0 @@
-/**
- * Reset some basic elements
- */
-body, h1, h2, h3, h4, h5, h6,
-p, blockquote, pre, hr,
-dl, dd, ol, ul, figure {
-    margin: 0;
-    padding: 0;
-}
-
-
-
-/**
- * Basic styling
- */
-body {
-    font-family: $base-font-family;
-    font-size: $base-font-size;
-    line-height: $base-line-height;
-    font-weight: 300;
-    color: $text-color;
-    background-color: $background-color;
-    -webkit-text-size-adjust: 100%;
-}
-
-
-
-/**
- * Set `margin-bottom` to maintain vertical rhythm
- */
-h1, h2, h3, h4, h5, h6,
-p, blockquote, pre,
-ul, ol, dl, figure,
-%vertical-rhythm {
-    margin-bottom: $spacing-unit / 2;
-}
-
-
-
-/**
- * Images
- */
-img {
-    max-width: 100%;
-    vertical-align: middle;
-}
-
-
-
-/**
- * Figures
- */
-figure > img {
-    display: block;
-}
-
-figcaption {
-    font-size: $small-font-size;
-}
-
-
-
-/**
- * Lists
- */
-ul, ol {
-    margin-left: $spacing-unit;
-}
-
-li {
-    > ul,
-    > ol {
-         margin-bottom: 0;
-    }
-}
-
-
-
-/**
- * Headings
- */
-h1, h2, h3, h4, h5, h6 {
-    font-weight: 300;
-}
-
-
-
-/**
- * Links
- */
-a {
-    color: $brand-color;
-    text-decoration: none;
-
-    &:visited {
-        color: darken($brand-color, 15%);
-    }
-
-    &:hover {
-        color: $text-color;
-        text-decoration: underline;
-    }
-}
-
-
-
-/**
- * Blockquotes
- */
-blockquote {
-    color: $grey-color;
-    border-left: 4px solid $grey-color-light;
-    padding-left: $spacing-unit / 2;
-    font-size: 18px;
-    letter-spacing: -1px;
-    font-style: italic;
-
-    > :last-child {
-        margin-bottom: 0;
-    }
-}
-
-
-
-/**
- * Code formatting
- */
-pre,
-code {
-    font-size: 15px;
-    border: 1px solid $grey-color-light;
-    border-radius: 3px;
-    background-color: #eef;
-}
-
-code {
-    padding: 1px 5px;
-}
-
-pre {
-    padding: 8px 12px;
-    overflow-x: scroll;
-
-    > code {
-        border: 0;
-        padding-right: 0;
-        padding-left: 0;
-    }
-}
-
-
-
-/**
- * Wrapper
- */
-.wrapper {
-    max-width: -webkit-calc(#{$content-width} - (#{$spacing-unit} * 2));
-    max-width:         calc(#{$content-width} - (#{$spacing-unit} * 2));
-    margin-right: auto;
-    margin-left: auto;
-    padding-right: $spacing-unit;
-    padding-left: $spacing-unit;
-    @extend %clearfix;
-
-    @include media-query($on-laptop) {
-        max-width: -webkit-calc(#{$content-width} - (#{$spacing-unit}));
-        max-width:         calc(#{$content-width} - (#{$spacing-unit}));
-        padding-right: $spacing-unit / 2;
-        padding-left: $spacing-unit / 2;
-    }
-}
-
-
-
-/**
- * Clearfix
- */
-%clearfix {
-
-    &:after {
-        content: "";
-        display: table;
-        clear: both;
-    }
-}
-
-
-
-/**
- * Icons
- */
-.icon {
-
-    > svg {
-        display: inline-block;
-        width: 16px;
-        height: 16px;
-        vertical-align: middle;
-
-        path {
-            fill: $grey-color;
-        }
-    }
-}
diff --git a/supporting-docs/_sass/_layout.scss b/supporting-docs/_sass/_layout.scss
deleted file mode 100644
index def56f89..00000000
--- a/supporting-docs/_sass/_layout.scss
+++ /dev/null
@@ -1,236 +0,0 @@
-/**
- * Site header
- */
-.site-header {
-    border-top: 5px solid $grey-color-dark;
-    border-bottom: 1px solid $grey-color-light;
-    min-height: 56px;
-
-    // Positioning context for the mobile navigation icon
-    position: relative;
-}
-
-.site-title {
-    font-size: 26px;
-    line-height: 56px;
-    letter-spacing: -1px;
-    margin-bottom: 0;
-    float: left;
-
-    &,
-    &:visited {
-        color: $grey-color-dark;
-    }
-}
-
-.site-nav {
-    float: right;
-    line-height: 56px;
-
-    .menu-icon {
-        display: none;
-    }
-
-    .page-link {
-        color: $text-color;
-        line-height: $base-line-height;
-
-        // Gaps between nav items, but not on the first one
-        &:not(:first-child) {
-            margin-left: 20px;
-        }
-    }
-
-    @include media-query($on-palm) {
-        position: absolute;
-        top: 9px;
-        right: 30px;
-        background-color: $background-color;
-        border: 1px solid $grey-color-light;
-        border-radius: 5px;
-        text-align: right;
-
-        .menu-icon {
-            display: block;
-            float: right;
-            width: 36px;
-            height: 26px;
-            line-height: 0;
-            padding-top: 10px;
-            text-align: center;
-
-            > svg {
-                width: 18px;
-                height: 15px;
-
-                path {
-                    fill: $grey-color-dark;
-                }
-            }
-        }
-
-        .trigger {
-            clear: both;
-            display: none;
-        }
-
-        &:hover .trigger {
-            display: block;
-            padding-bottom: 5px;
-        }
-
-        .page-link {
-            display: block;
-            padding: 5px 10px;
-        }
-    }
-}
-
-
-
-/**
- * Site footer
- */
-.site-footer {
-    border-top: 1px solid $grey-color-light;
-    padding: $spacing-unit 0;
-}
-
-.footer-heading {
-    font-size: 18px;
-    margin-bottom: $spacing-unit / 2;
-}
-
-.contact-list,
-.social-media-list {
-    list-style: none;
-    margin-left: 0;
-}
-
-.footer-col-wrapper {
-    font-size: 15px;
-    color: $grey-color;
-    margin-left: -$spacing-unit / 2;
-    @extend %clearfix;
-}
-
-.footer-col {
-    float: left;
-    margin-bottom: $spacing-unit / 2;
-    padding-left: $spacing-unit / 2;
-}
-
-.footer-col-1 {
-    width: -webkit-calc(35% - (#{$spacing-unit} / 2));
-    width:         calc(35% - (#{$spacing-unit} / 2));
-}
-
-.footer-col-2 {
-    width: -webkit-calc(20% - (#{$spacing-unit} / 2));
-    width:         calc(20% - (#{$spacing-unit} / 2));
-}
-
-.footer-col-3 {
-    width: -webkit-calc(45% - (#{$spacing-unit} / 2));
-    width:         calc(45% - (#{$spacing-unit} / 2));
-}
-
-@include media-query($on-laptop) {
-    .footer-col-1,
-    .footer-col-2 {
-        width: -webkit-calc(50% - (#{$spacing-unit} / 2));
-        width:         calc(50% - (#{$spacing-unit} / 2));
-    }
-
-    .footer-col-3 {
-        width: -webkit-calc(100% - (#{$spacing-unit} / 2));
-        width:         calc(100% - (#{$spacing-unit} / 2));
-    }
-}
-
-@include media-query($on-palm) {
-    .footer-col {
-        float: none;
-        width: -webkit-calc(100% - (#{$spacing-unit} / 2));
-        width:         calc(100% - (#{$spacing-unit} / 2));
-    }
-}
-
-
-
-/**
- * Page content
- */
-.page-content {
-    padding: $spacing-unit 0;
-}
-
-.page-heading {
-    font-size: 20px;
-}
-
-.post-list {
-    margin-left: 0;
-    list-style: none;
-
-    > li {
-        margin-bottom: $spacing-unit;
-    }
-}
-
-.post-meta {
-    font-size: $small-font-size;
-    color: $grey-color;
-}
-
-.post-link {
-    display: block;
-    font-size: 24px;
-}
-
-
-
-/**
- * Posts
- */
-.post-header {
-    margin-bottom: $spacing-unit;
-}
-
-.post-title {
-    font-size: 42px;
-    letter-spacing: -1px;
-    line-height: 1;
-
-    @include media-query($on-laptop) {
-        font-size: 36px;
-    }
-}
-
-.post-content {
-    margin-bottom: $spacing-unit;
-
-    h2 {
-        font-size: 32px;
-
-        @include media-query($on-laptop) {
-            font-size: 28px;
-        }
-    }
-
-    h3 {
-        font-size: 26px;
-
-        @include media-query($on-laptop) {
-            font-size: 22px;
-        }
-    }
-
-    h4 {
-        font-size: 20px;
-
-        @include media-query($on-laptop) {
-            font-size: 18px;
-        }
-    }
-}
diff --git a/supporting-docs/_sass/_syntax-highlighting.scss b/supporting-docs/_sass/_syntax-highlighting.scss
deleted file mode 100644
index e36627da..00000000
--- a/supporting-docs/_sass/_syntax-highlighting.scss
+++ /dev/null
@@ -1,67 +0,0 @@
-/**
- * Syntax highlighting styles
- */
-.highlight {
-    background: #fff;
-    @extend %vertical-rhythm;
-
-    .c     { color: #998; font-style: italic } // Comment
-    .err   { color: #a61717; background-color: #e3d2d2 } // Error
-    .k     { font-weight: bold } // Keyword
-    .o     { font-weight: bold } // Operator
-    .cm    { color: #998; font-style: italic } // Comment.Multiline
-    .cp    { color: #999; font-weight: bold } // Comment.Preproc
-    .c1    { color: #998; font-style: italic } // Comment.Single
-    .cs    { color: #999; font-weight: bold; font-style: italic } // Comment.Special
-    .gd    { color: #000; background-color: #fdd } // Generic.Deleted
-    .gd .x { color: #000; background-color: #faa } // Generic.Deleted.Specific
-    .ge    { font-style: italic } // Generic.Emph
-    .gr    { color: #a00 } // Generic.Error
-    .gh    { color: #999 } // Generic.Heading
-    .gi    { color: #000; background-color: #dfd } // Generic.Inserted
-    .gi .x { color: #000; background-color: #afa } // Generic.Inserted.Specific
-    .go    { color: #888 } // Generic.Output
-    .gp    { color: #555 } // Generic.Prompt
-    .gs    { font-weight: bold } // Generic.Strong
-    .gu    { color: #aaa } // Generic.Subheading
-    .gt    { color: #a00 } // Generic.Traceback
-    .kc    { font-weight: bold } // Keyword.Constant
-    .kd    { font-weight: bold } // Keyword.Declaration
-    .kp    { font-weight: bold } // Keyword.Pseudo
-    .kr    { font-weight: bold } // Keyword.Reserved
-    .kt    { color: #458; font-weight: bold } // Keyword.Type
-    .m     { color: #099 } // Literal.Number
-    .s     { color: #d14 } // Literal.String
-    .na    { color: #008080 } // Name.Attribute
-    .nb    { color: #0086B3 } // Name.Builtin
-    .nc    { color: #458; font-weight: bold } // Name.Class
-    .no    { color: #008080 } // Name.Constant
-    .ni    { color: #800080 } // Name.Entity
-    .ne    { color: #900; font-weight: bold } // Name.Exception
-    .nf    { color: #900; font-weight: bold } // Name.Function
-    .nn    { color: #555 } // Name.Namespace
-    .nt    { color: #000080 } // Name.Tag
-    .nv    { color: #008080 } // Name.Variable
-    .ow    { font-weight: bold } // Operator.Word
-    .w     { color: #bbb } // Text.Whitespace
-    .mf    { color: #099 } // Literal.Number.Float
-    .mh    { color: #099 } // Literal.Number.Hex
-    .mi    { color: #099 } // Literal.Number.Integer
-    .mo    { color: #099 } // Literal.Number.Oct
-    .sb    { color: #d14 } // Literal.String.Backtick
-    .sc    { color: #d14 } // Literal.String.Char
-    .sd    { color: #d14 } // Literal.String.Doc
-    .s2    { color: #d14 } // Literal.String.Double
-    .se    { color: #d14 } // Literal.String.Escape
-    .sh    { color: #d14 } // Literal.String.Heredoc
-    .si    { color: #d14 } // Literal.String.Interpol
-    .sx    { color: #d14 } // Literal.String.Other
-    .sr    { color: #009926 } // Literal.String.Regex
-    .s1    { color: #d14 } // Literal.String.Single
-    .ss    { color: #990073 } // Literal.String.Symbol
-    .bp    { color: #999 } // Name.Builtin.Pseudo
-    .vc    { color: #008080 } // Name.Variable.Class
-    .vg    { color: #008080 } // Name.Variable.Global
-    .vi    { color: #008080 } // Name.Variable.Instance
-    .il    { color: #099 } // Literal.Number.Integer.Long
-}
diff --git a/supporting-docs/about.md b/supporting-docs/about.md
deleted file mode 100644
index 3ed64bb6..00000000
--- a/supporting-docs/about.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-layout: page
-title: About
-permalink: /about/
----
-
-This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at [jekyllrb.com](http://jekyllrb.com/)
-
-You can find the source code for the Jekyll new theme at: [github.com/jglovier/jekyll-new](https://github.com/jglovier/jekyll-new)
-
-You can find the source code for Jekyll at [github.com/jekyll/jekyll](https://github.com/jekyll/jekyll)
diff --git a/supporting-docs/css/basic.css b/supporting-docs/css/basic.css
deleted file mode 100644
index bf8f4712..00000000
--- a/supporting-docs/css/basic.css
+++ /dev/null
@@ -1,512 +0,0 @@
-/*
- * basic.css
- * ~~~~~~~~~
- *
- * Sphinx stylesheet -- basic theme.
- *
- * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
- * :license: BSD, see LICENSE for details.
- *
- */
-
-/* -- main layout ----------------------------------------------------------- */
-
-div.clearer {
-    clear: both;
-}
-
-/* -- relbar ---------------------------------------------------------------- */
-
-div.related {
-    width: 100%;
-    font-size: 90%;
-}
-
-div.related h3 {
-    display: none;
-}
-
-div.related ul {
-    margin: 0;
-    padding: 0 0 0 10px;
-    list-style: none;
-}
-
-div.related li {
-    display: inline;
-}
-
-div.related li.right {
-    float: right;
-    margin-right: 5px;
-}
-
-/* -- sidebar --------------------------------------------------------------- */
-
-div.sphinxsidebarwrapper {
-    padding: 10px 5px 0 10px;
-}
-
-div.sphinxsidebar {
-    float: left;
-    width: 230px;
-    margin-left: -100%;
-    font-size: 90%;
-}
-
-div.sphinxsidebar ul {
-    list-style: none;
-}
-
-div.sphinxsidebar ul ul,
-div.sphinxsidebar ul.want-points {
-    margin-left: 20px;
-    list-style: square;
-}
-
-div.sphinxsidebar ul ul {
-    margin-top: 0;
-    margin-bottom: 0;
-}
-
-div.sphinxsidebar form {
-    margin-top: 10px;
-}
-
-div.sphinxsidebar input {
-    border: 1px solid #98dbcc;
-    font-family: sans-serif;
-    font-size: 1em;
-}
-
-img {
-    border: 0;
-}
-
-/* -- search page ----------------------------------------------------------- */
-
-ul.search {
-    margin: 10px 0 0 20px;
-    padding: 0;
-}
-
-ul.search li {
-    padding: 5px 0 5px 20px;
-    background-image: url(file.png);
-    background-repeat: no-repeat;
-    background-position: 0 7px;
-}
-
-ul.search li a {
-    font-weight: bold;
-}
-
-ul.search li div.context {
-    color: #888;
-    margin: 2px 0 0 30px;
-    text-align: left;
-}
-
-ul.keywordmatches li.goodmatch a {
-    font-weight: bold;
-}
-
-/* -- index page ------------------------------------------------------------ */
-
-table.contentstable {
-    width: 90%;
-}
-
-table.contentstable p.biglink {
-    line-height: 150%;
-}
-
-a.biglink {
-    font-size: 1.3em;
-}
-
-span.linkdescr {
-    font-style: italic;
-    padding-top: 5px;
-    font-size: 90%;
-}
-
-/* -- general index --------------------------------------------------------- */
-
-table.indextable {
-    width: 100%;
-}
-
-table.indextable td {
-    text-align: left;
-    vertical-align: top;
-}
-
-table.indextable dl, table.indextable dd {
-    margin-top: 0;
-    margin-bottom: 0;
-}
-
-table.indextable tr.pcap {
-    height: 10px;
-}
-
-table.indextable tr.cap {
-    margin-top: 10px;
-    background-color: #f2f2f2;
-}
-
-img.toggler {
-    margin-right: 3px;
-    margin-top: 3px;
-    cursor: pointer;
-}
-
-div.modindex-jumpbox {
-    border-top: 1px solid #ddd;
-    border-bottom: 1px solid #ddd;
-    margin: 1em 0 1em 0;
-    padding: 0.4em;
-}
-
-div.genindex-jumpbox {
-    border-top: 1px solid #ddd;
-    border-bottom: 1px solid #ddd;
-    margin: 1em 0 1em 0;
-    padding: 0.4em;
-}
-
-/* -- general body styles --------------------------------------------------- */
-
-a.headerlink {
-    visibility: hidden;
-}
-
-h1:hover > a.headerlink,
-h2:hover > a.headerlink,
-h3:hover > a.headerlink,
-h4:hover > a.headerlink,
-h5:hover > a.headerlink,
-h6:hover > a.headerlink,
-dt:hover > a.headerlink {
-    visibility: visible;
-}
-
-div.document p.caption {
-    text-align: inherit;
-}
-
-div.document td {
-    text-align: left;
-}
-
-.field-list ul {
-    padding-left: 1em;
-}
-
-.first {
-    margin-top: 0 !important;
-}
-
-p.rubric {
-    margin-top: 30px;
-    font-weight: bold;
-}
-
-.align-left {
-    text-align: left;
-}
-
-.align-center {
-    clear: both;
-    text-align: center;
-}
-
-.align-right {
-    text-align: right;
-}
-
-/* -- sidebars -------------------------------------------------------------- */
-
-div.sidebar {
-    margin: 0 0 0.5em 1em;
-    border: 1px solid #ddb;
-    padding: 7px 7px 0 7px;
-    background-color: #ffe;
-    width: 40%;
-    float: right;
-}
-
-p.sidebar-title {
-    font-weight: bold;
-}
-
-/* -- topics ---------------------------------------------------------------- */
-
-div.topic {
-    border: 1px solid #ccc;
-    padding: 7px 7px 0 7px;
-    margin: 10px 0 10px 0;
-}
-
-p.topic-title {
-    font-size: 1.1em;
-    font-weight: bold;
-    margin-top: 10px;
-}
-
-/* -- admonitions ----------------------------------------------------------- */
-
-div.admonition {
-    margin-top: 10px;
-    margin-bottom: 10px;
-    padding: 7px;
-}
-
-div.admonition dt {
-    font-weight: bold;
-}
-
-div.admonition dl {
-    margin-bottom: 0;
-}
-
-p.admonition-title {
-    margin: 0px 10px 5px 0px;
-    font-weight: bold;
-}
-
-div.document p.centered {
-    text-align: center;
-    margin-top: 25px;
-}
-
-/* -- tables ---------------------------------------------------------------- */
-
-table.docutils {
-    border: 0;
-    border-collapse: collapse;
-}
-
-table.docutils td, table.docutils th {
-    padding: 1px 8px 1px 5px;
-    border-top: 0;
-    border-left: 0;
-    border-right: 0;
-    border-bottom: 1px solid #aaa;
-}
-
-table.field-list td, table.field-list th {
-    border: 0 !important;
-}
-
-table.footnote td, table.footnote th {
-    border: 0 !important;
-}
-
-th {
-    text-align: left;
-    padding-right: 5px;
-}
-
-table.citation {
-    border-left: solid 1px gray;
-    margin-left: 1px;
-}
-
-table.citation td {
-    border-bottom: none;
-}
-
-/* -- other body styles ----------------------------------------------------- */
-
-ol.arabic {
-    list-style: decimal;
-}
-
-ol.loweralpha {
-    list-style: lower-alpha;
-}
-
-ol.upperalpha {
-    list-style: upper-alpha;
-}
-
-ol.lowerroman {
-    list-style: lower-roman;
-}
-
-ol.upperroman {
-    list-style: upper-roman;
-}
-
-dl {
-    margin-bottom: 15px;
-}
-
-dd p {
-    margin-top: 0px;
-}
-
-dd ul, dd table {
-    margin-bottom: 10px;
-}
-
-dd {
-    margin-top: 3px;
-    margin-bottom: 10px;
-    margin-left: 30px;
-}
-
-dt:target, .highlighted {
-    background-color: #fbe54e;
-}
-
-dl.glossary dt {
-    font-weight: bold;
-    font-size: 1.1em;
-}
-
-.field-list ul {
-    margin: 0;
-    padding-left: 1em;
-}
-
-.field-list p {
-    margin: 0;
-}
-
-.refcount {
-    color: #060;
-}
-
-.optional {
-    font-size: 1.3em;
-}
-
-.versionmodified {
-    font-style: italic;
-}
-
-.system-message {
-    background-color: #fda;
-    padding: 5px;
-    border: 3px solid red;
-}
-
-.footnote:target  {
-    background-color: #ffa
-}
-
-.line-block {
-    display: block;
-    margin-top: 1em;
-    margin-bottom: 1em;
-}
-
-.line-block .line-block {
-    margin-top: 0;
-    margin-bottom: 0;
-    margin-left: 1.5em;
-}
-
-.guilabel, .menuselection {
-    font-family: sans-serif;
-}
-
-.accelerator {
-    text-decoration: underline;
-}
-
-.classifier {
-    font-style: oblique;
-}
-
-/* -- code displays --------------------------------------------------------- */
-
-pre {
-    overflow: auto;
-    font-size: 1.1em;
-
-}
-
-td.linenos pre {
-    padding: 5px 0px;
-    border: 0;
-    background-color: transparent;
-    color: #aaa;
-}
-
-table.highlighttable {
-    margin-left: 0.5em;
-}
-
-table.highlighttable td {
-    padding: 0 0.5em 0 0.5em;
-}
-
-tt.descname {
-    background-color: transparent;
-    font-weight: bold;
-    font-size: 1.2em;
-}
-
-tt.descclassname {
-    background-color: transparent;
-}
-
-tt.xref, a tt {
-    background-color: transparent;
-    font-weight: bold;
-}
-
-h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt {
-    background-color: transparent;
-}
-
-.viewcode-link {
-    float: right;
-}
-
-.viewcode-back {
-    float: right;
-    font-family: sans-serif;
-}
-
-div.viewcode-block:target {
-    margin: -1px -10px;
-    padding: 0 10px;
-}
-
-/* -- math display ---------------------------------------------------------- */
-
-img.math {
-    vertical-align: middle;
-}
-
-div.document div.math p {
-    text-align: center;
-}
-
-span.eqno {
-    float: right;
-}
-
-/* -- printout stylesheet --------------------------------------------------- */
-
-@media print {
-    div.document,
-    div.documentwrapper,
-    div.bodywrapper {
-        margin: 0 !important;
-        width: 100%;
-    }
-
-    div.sphinxsidebar,
-    div.related,
-    div.footer,
-    #top-link {
-        display: none;
-    }
-}
-
diff --git a/supporting-docs/css/faq.css b/supporting-docs/css/faq.css
deleted file mode 100644
index 9730efa3..00000000
--- a/supporting-docs/css/faq.css
+++ /dev/null
@@ -1,40 +0,0 @@
-.toc {
-    background: #f9f9f9;
-    border: 1px solid #aaaaaa;
-    display: table;
-    padding-top: 16px;
-    padding-right: 16px;
-    padding-bottom: 16px;
-    list-style: none;
-    color: #2EA3F2;
-}
-
-ul {
-    padding-left: 30px;
-}
-
-p {
-    padding-left: 30px;
-    color: #666;
-
-}
-
-h3 {
-    padding-top: 48px;
-    color: #333;
-}
-
-h4, h5 {
-    border-top: 40px solid transparent;
-    margin-top: -40px;
-    -webkit-background-clip: padding-box;
-    -moz-background-clip: padding;
-    background-clip: padding-box;
-    font-style: italic;
-    font-size: 16px;
-    padding-top: 16px;
-    color: #333;
-
-}
-
-
diff --git a/supporting-docs/css/main.scss b/supporting-docs/css/main.scss
deleted file mode 100755
index b8a3e2be..00000000
--- a/supporting-docs/css/main.scss
+++ /dev/null
@@ -1,52 +0,0 @@
----
-# Only the main Sass file needs front matter (the dashes are enough)
----
-@charset "utf-8";
-
-
-
-// Our variables
-$base-font-family: Helvetica, Arial, sans-serif;
-$base-font-size:   16px;
-$small-font-size:  $base-font-size * 0.875;
-$base-line-height: 1.5;
-
-$spacing-unit:     30px;
-
-$text-color:       #111;
-$background-color: #fdfdfd;
-$brand-color:      #2a7ae2;
-
-$grey-color:       #828282;
-$grey-color-light: lighten($grey-color, 40%);
-$grey-color-dark:  darken($grey-color, 25%);
-
-// Width of the content area
-$content-width:    1340px;
-
-$on-palm:          600px;
-$on-laptop:        800px;
-
-
-
-// Using media queries with like this:
-// @include media-query($on-palm) {
-//     .wrapper {
-//         padding-right: $spacing-unit / 2;
-//         padding-left: $spacing-unit / 2;
-//     }
-// }
-@mixin media-query($device) {
-    @media screen and (max-width: $device) {
-        @content;
-    }
-}
-
-
-
-// Import partials from `sass_dir` (defaults to `_sass`)
-@import
-        "base",
-        "layout",
-        "syntax-highlighting"
-;
diff --git a/supporting-docs/css/nature.css b/supporting-docs/css/nature.css
deleted file mode 100644
index 9bf583be..00000000
--- a/supporting-docs/css/nature.css
+++ /dev/null
@@ -1,285 +0,0 @@
-/*
- * nature.css_t
- * ~~~~~~~~~~~~
- *
- * Sphinx stylesheet -- nature theme.
- *
- * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
- * :license: BSD, see LICENSE for details.
- *
- */
- 
-/* -- page layout ----------------------------------------------------------- */
- 
-body {
-    font-family: Arial, sans-serif;
-    font-size: 100%;
-    /*background-color: #111;*/
-    color: #555;
-    margin: 0;
-    padding: 0;
-}
-
-div.documentwrapper {
-    float: left;
-    width: 100%;
-}
-
-div.bodywrapper {
-    margin: 0 0 0 230px;
-}
-
-hr {
-    border: 1px solid #B1B4B6;
-}
- 
-/*
-div.document {
-    background-color: #eee;
-}
-*/
- 
-div.document {
-    background-color: #ffffff;
-    color: #3E4349;
-    padding: 0 30px 30px 30px;
-    font-size: 0.9em;
-}
- 
-div.footer {
-    color: #555;
-    width: 100%;
-    padding: 13px 0;
-    text-align: center;
-    font-size: 75%;
-}
- 
-div.footer a {
-    color: #444;
-    text-decoration: underline;
-}
- 
-div.related {
-    background-color: #6BA81E;
-    line-height: 32px;
-    color: #fff;
-    text-shadow: 0px 1px 0 #444;
-    font-size: 0.9em;
-}
- 
-div.related a {
-    color: #E2F3CC;
-}
- 
-div.sphinxsidebar {
-    font-size: 0.75em;
-    line-height: 1.5em;
-}
-
-div.sphinxsidebarwrapper{
-    padding: 20px 0;
-}
- 
-div.sphinxsidebar h3,
-div.sphinxsidebar h4 {
-    font-family: Arial, sans-serif;
-    color: #222;
-    font-size: 1.2em;
-    font-weight: normal;
-    margin: 0;
-    padding: 5px 10px;
-    background-color: #ddd;
-    text-shadow: 1px 1px 0 white
-}
-
-div.sphinxsidebar h4{
-    font-size: 1.1em;
-}
- 
-div.sphinxsidebar h3 a {
-    color: #444;
-}
- 
- 
-div.sphinxsidebar p {
-    color: #888;
-    padding: 5px 20px;
-}
- 
-div.sphinxsidebar p.topless {
-}
- 
-div.sphinxsidebar ul {
-    margin: 10px 20px;
-    padding: 0;
-    color: #000;
-}
- 
-div.sphinxsidebar a {
-    color: #444;
-}
- 
-div.sphinxsidebar input {
-    border: 1px solid #ccc;
-    font-family: sans-serif;
-    font-size: 1em;
-}
-
-div.sphinxsidebar input[type=text]{
-    margin-left: 20px;
-}
- 
-/* -- body styles ----------------------------------------------------------- */
- 
-a {
-    color: #005B81;
-    text-decoration: none;
-}
- 
-a:hover {
-    color: #E32E00;
-    text-decoration: underline;
-}
- 
-div.document h1,
-div.document h2,
-div.document h3,
-div.document h4,
-div.document h5,
-div.document h6 {
-    font-family: Arial, sans-serif;
-    background-color: #BED4EB;
-    font-weight: normal;
-    color: #212224;
-    margin: 30px 0px 10px 0px;
-    padding: 5px 0 5px 10px;
-    text-shadow: 0px 1px 0 white
-}
- 
-div.document h1 { border-top: 20px solid white; margin-top: 0; font-size: 200%; }
-div.document h2 { font-size: 150%; background-color: #C8D5E3; }
-div.document h3 { font-size: 120%; background-color: #D8DEE3; }
-div.document h4 { font-size: 110%; background-color: #D8DEE3; }
-div.document h5 { font-size: 100%; background-color: #D8DEE3; }
-div.document h6 { font-size: 100%; background-color: #D8DEE3; }
- 
-a.headerlink {
-    color: #c60f0f;
-    font-size: 0.8em;
-    padding: 0 4px 0 4px;
-    text-decoration: none;
-}
- 
-a.headerlink:hover {
-    background-color: #c60f0f;
-    color: white;
-}
- 
-div.document p, div.document dd, div.document li {
-    line-height: 1.5em;
-}
- 
-div.admonition p.admonition-title + p {
-    display: inline;
-}
-
-div.highlight{
-    background-color: white;
-}
-
-div.note {
-    background-color: #eee;
-    border: 1px solid #ccc;
-}
- 
-div.seealso {
-    background-color: #ffc;
-    border: 1px solid #ff6;
-}
- 
-div.topic {
-    background-color: #eee;
-}
- 
-div.warning {
-    background-color: #ffe4e4;
-    border: 1px solid #f66;
-}
- 
-p.admonition-title {
-    display: inline;
-}
- 
-p.admonition-title:after {
-    content: ":";
-}
- 
-pre {
-    padding: 10px;
-    background-color: White;
-    color: #222;
-    line-height: 1.2em;
-    border: 1px solid #C6C9CB;
-    font-size: 1.1em;
-    margin: 1.5em 0 1.5em 0;
-    -webkit-box-shadow: 1px 1px 1px #d8d8d8;
-    -moz-box-shadow: 1px 1px 1px #d8d8d8;
-}
- 
-tt {
-    background-color: #ecf0f3;
-    color: #222;
-    /* padding: 1px 2px; */
-    font-size: 1.1em;
-    font-family: monospace;
-}
-
-.viewcode-back {
-    font-family: Arial, sans-serif;
-}
-
-div.viewcode-block:target {
-    background-color: #f4debf;
-    border-top: 1px solid #ac9;
-    border-bottom: 1px solid #ac9;
-}
-
-p {
-	margin: 0;
-}
-
-ul li dd {
-	margin-top: 0;
-}
-
-ul li dl {
-	margin-bottom: 0;
-}
-
-li dl dd {
-	margin-bottom: 0;
-}
-
-dd ul {
-	padding-left: 0;
-}
-
-li dd ul {
-	margin-bottom: 0;
-}
-
-table {
-    margin-top: 10px;
-    margin-bottom: 10px;
-    border: 0;
-    border-collapse: collapse;
-}
-
-td[colspan]:not([colspan="1"]) {
-    background: #eeeeee;
-}
-
-thead {
-    background: #eeeeee;
-}
-
diff --git a/supporting-docs/css/site_overrides.css b/supporting-docs/css/site_overrides.css
deleted file mode 100644
index f7640e1c..00000000
--- a/supporting-docs/css/site_overrides.css
+++ /dev/null
@@ -1,62 +0,0 @@
-p {
-    line-height: 1.5;
-    color: #666;
-}
-
-a.anchor {  
-    display: block; 
-    position: relative; 
-    top: -50px; 
-    visibility: hidden;
-}
-
-a:link {
-    color: #2EA3F2;
-    text-decoration: none;
-}
-
-a:visited {
-    color: #2EA3F2;
-    text-decoration: none;
-}
-
-a:active {
-    color: #2EA3F2;
-    text-decoration: none;
-}
-
-a:hover {
-    color: #2EA3F2;
-    text-decoration: underline;
-}
-
-h1,h2,h3,h4,h5 {
-    color: #333;
-}
-
-.page-content {
-    min-height: 100%;
-    height: auto !important; /* This line and the next line are not necessary unless you need IE6 support */
-    height: 100%;
-    margin: 0 auto -33px; /* the bottom margin is the negative value of the footer's height */
-    padding-bottom: 0;
-    padding-top: 0;
-}
-
-.push {
-    height: 33px; 
-}
-
-#page {
-    margin-bottom: -33px;
-}
-
-#document {
-    margin-top: 10px; /* We want a little whitespace before the page content starts */
-    width: 1080px;
-}
-
-#wrapper {
-    max-width: 1080px;
-}
-
diff --git a/supporting-docs/feed.xml b/supporting-docs/feed.xml
deleted file mode 100644
index a6628bd8..00000000
--- a/supporting-docs/feed.xml
+++ /dev/null
@@ -1,30 +0,0 @@
----
-layout: null
----
-<?xml version="1.0" encoding="UTF-8"?>
-<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
-  <channel>
-    <title>{{ site.title | xml_escape }}</title>
-    <description>{{ site.description | xml_escape }}</description>
-    <link>{{ site.url }}{{ site.baseurl }}/</link>
-    <atom:link href="{{ "/feed.xml" | prepend: site.baseurl | prepend: site.url }}" rel="self" type="application/rss+xml"/>
-    <pubDate>{{ site.time | date_to_rfc822 }}</pubDate>
-    <lastBuildDate>{{ site.time | date_to_rfc822 }}</lastBuildDate>
-    <generator>Jekyll v{{ jekyll.version }}</generator>
-    {% for post in site.posts limit:10 %}
-      <item>
-        <title>{{ post.title | xml_escape }}</title>
-        <description>{{ post.content | xml_escape }}</description>
-        <pubDate>{{ post.date | date_to_rfc822 }}</pubDate>
-        <link>{{ post.url | prepend: site.baseurl | prepend: site.url }}</link>
-        <guid isPermaLink="true">{{ post.url | prepend: site.baseurl | prepend: site.url }}</guid>
-        {% for tag in post.tags %}
-        <category>{{ tag | xml_escape }}</category>
-        {% endfor %}
-        {% for cat in post.categories %}
-        <category>{{ cat | xml_escape }}</category>
-        {% endfor %}
-      </item>
-    {% endfor %}
-  </channel>
-</rss>
diff --git a/supporting-docs/_posts/2015-08-10-client-server.rst b/supporting-docs/guides/2015-08-10-client-server.rst
similarity index 100%
rename from supporting-docs/_posts/2015-08-10-client-server.rst
rename to supporting-docs/guides/2015-08-10-client-server.rst
diff --git a/supporting-docs/_posts/2015-08-14-getting_involved.md b/supporting-docs/guides/2015-08-14-getting_involved.md
similarity index 100%
rename from supporting-docs/_posts/2015-08-14-getting_involved.md
rename to supporting-docs/guides/2015-08-14-getting_involved.md
diff --git a/supporting-docs/_posts/2015-08-19-faq.md b/supporting-docs/guides/2015-08-19-faq.md
similarity index 100%
rename from supporting-docs/_posts/2015-08-19-faq.md
rename to supporting-docs/guides/2015-08-19-faq.md
diff --git a/supporting-docs/_posts/2015-08-21-application_services.md b/supporting-docs/guides/2015-08-21-application_services.md
similarity index 100%
rename from supporting-docs/_posts/2015-08-21-application_services.md
rename to supporting-docs/guides/2015-08-21-application_services.md
diff --git a/supporting-docs/index.html b/supporting-docs/index.html
deleted file mode 100644
index 5a874f06..00000000
--- a/supporting-docs/index.html
+++ /dev/null
@@ -1,14 +0,0 @@
----
-layout: default
----
-<div class="home">
-
-  <h1>Guides</h1>
-
-  <p>Here is a collection of guides that might help you get involved with Matrix.</p>
-  <p>First, there is the <a href="./getting_involved.html" title="Getting Involved">Getting Involved</a> guide, which explains various ways of getting started with Matrix.</p>
-  <p>The <a href="./client-server.html" title="Client-Server API">Client-Server API</a> guide explains in detail how to use the CS API, which is useful if you want to write a client (or modify an existing one) - or if you're just interested in how it works "under the hood".</p>
-  <p>The <a href="./application_services.html" title="Application services">Application services</a> guide introduces and explains Application services, and what they can be used for.
-  <p>Finally there is the <a href="./faq.html" title="FAQ">FAQ</a>, which tries to answer all your questions relating to Matrix!</p>
-
-</div>

From a0cf4859190efc4b8ce02c34ba68a42508a0dbd6 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 9 Oct 2015 10:43:27 +0100
Subject: [PATCH 288/467] speculator: Specify merge target

---
 scripts/speculator/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 842d20b7..d807bf78 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -86,7 +86,7 @@ func gitFetchAndMerge(path string) error {
 	if err := runGitCommand(path, []string{"fetch"}); err != nil {
 		return err
 	}
-	return runGitCommand(path, []string{"merge"})
+	return runGitCommand(path, []string{"merge", "origin/master"})
 }
 
 func runGitCommand(path string, args []string) error {

From 0dbc6d82770b59a329f356e118762fe2e117be7b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 9 Oct 2015 10:59:26 +0100
Subject: [PATCH 289/467] Checkout SHA of origin/master rather than merging

---
 scripts/speculator/main.go | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index d807bf78..ad0a897b 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -82,11 +82,8 @@ func gitCheckout(path, sha string) error {
 	return runGitCommand(path, []string{"checkout", sha})
 }
 
-func gitFetchAndMerge(path string) error {
-	if err := runGitCommand(path, []string{"fetch"}); err != nil {
-		return err
-	}
-	return runGitCommand(path, []string{"merge", "origin/master"})
+func gitFetch(path string) error {
+	return runGitCommand(path, []string{"fetch"})
 }
 
 func runGitCommand(path string, args []string) error {
@@ -145,7 +142,7 @@ type server struct {
 // generateAt generates spec from repo at sha.
 // Returns the path where the generation was done.
 func (s *server) generateAt(sha string) (dst string, err error) {
-	err = gitFetchAndMerge(s.matrixDocCloneURL)
+	err = gitFetch(s.matrixDocCloneURL)
 	if err != nil {
 		return
 	}
@@ -162,11 +159,27 @@ func (s *server) generateAt(sha string) (dst string, err error) {
 	return
 }
 
+func (s *server) getSHAOf(ref string) (string, error) {
+	cmd := exec.Command("git", "rev-list", ref, "-n1")
+	cmd.Dir = path.Join(s.matrixDocCloneURL)
+	var b bytes.Buffer
+	cmd.Stdout = &b
+	err := cmd.Run()
+	if err != nil {
+		return "", fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())
+	}
+	return strings.TrimSpace(b.String()), nil
+}
+
 func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 	var sha string
 
 	if strings.ToLower(req.URL.Path) == "/spec/head" {
-		sha = "HEAD"
+		originHead, err := s.getSHAOf("origin/master")
+		if err != nil {
+			writeError(w, 500, err)
+		}
+		sha = originHead
 	} else {
 		pr, err := lookupPullRequest(*req.URL, "/spec")
 		if err != nil {

From 481298db83d257466aa24d7433798da8240e58c1 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 9 Oct 2015 11:01:35 +0100
Subject: [PATCH 290/467] Revert "Checkout SHA of origin/master rather than
 merging"

This reverts commit 0dbc6d82770b59a329f356e118762fe2e117be7b.
---
 scripts/speculator/main.go | 27 +++++++--------------------
 1 file changed, 7 insertions(+), 20 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index ad0a897b..d807bf78 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -82,8 +82,11 @@ func gitCheckout(path, sha string) error {
 	return runGitCommand(path, []string{"checkout", sha})
 }
 
-func gitFetch(path string) error {
-	return runGitCommand(path, []string{"fetch"})
+func gitFetchAndMerge(path string) error {
+	if err := runGitCommand(path, []string{"fetch"}); err != nil {
+		return err
+	}
+	return runGitCommand(path, []string{"merge", "origin/master"})
 }
 
 func runGitCommand(path string, args []string) error {
@@ -142,7 +145,7 @@ type server struct {
 // generateAt generates spec from repo at sha.
 // Returns the path where the generation was done.
 func (s *server) generateAt(sha string) (dst string, err error) {
-	err = gitFetch(s.matrixDocCloneURL)
+	err = gitFetchAndMerge(s.matrixDocCloneURL)
 	if err != nil {
 		return
 	}
@@ -159,27 +162,11 @@ func (s *server) generateAt(sha string) (dst string, err error) {
 	return
 }
 
-func (s *server) getSHAOf(ref string) (string, error) {
-	cmd := exec.Command("git", "rev-list", ref, "-n1")
-	cmd.Dir = path.Join(s.matrixDocCloneURL)
-	var b bytes.Buffer
-	cmd.Stdout = &b
-	err := cmd.Run()
-	if err != nil {
-		return "", fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())
-	}
-	return strings.TrimSpace(b.String()), nil
-}
-
 func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 	var sha string
 
 	if strings.ToLower(req.URL.Path) == "/spec/head" {
-		originHead, err := s.getSHAOf("origin/master")
-		if err != nil {
-			writeError(w, 500, err)
-		}
-		sha = originHead
+		sha = "HEAD"
 	} else {
 		pr, err := lookupPullRequest(*req.URL, "/spec")
 		if err != nil {

From b46f0e30b8f22c159b55053a8502fd2b7e78101a Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 9 Oct 2015 10:59:26 +0100
Subject: [PATCH 291/467] Checkout SHA of origin/master rather than merging

---
 scripts/speculator/main.go | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index d807bf78..ad0a897b 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -82,11 +82,8 @@ func gitCheckout(path, sha string) error {
 	return runGitCommand(path, []string{"checkout", sha})
 }
 
-func gitFetchAndMerge(path string) error {
-	if err := runGitCommand(path, []string{"fetch"}); err != nil {
-		return err
-	}
-	return runGitCommand(path, []string{"merge", "origin/master"})
+func gitFetch(path string) error {
+	return runGitCommand(path, []string{"fetch"})
 }
 
 func runGitCommand(path string, args []string) error {
@@ -145,7 +142,7 @@ type server struct {
 // generateAt generates spec from repo at sha.
 // Returns the path where the generation was done.
 func (s *server) generateAt(sha string) (dst string, err error) {
-	err = gitFetchAndMerge(s.matrixDocCloneURL)
+	err = gitFetch(s.matrixDocCloneURL)
 	if err != nil {
 		return
 	}
@@ -162,11 +159,27 @@ func (s *server) generateAt(sha string) (dst string, err error) {
 	return
 }
 
+func (s *server) getSHAOf(ref string) (string, error) {
+	cmd := exec.Command("git", "rev-list", ref, "-n1")
+	cmd.Dir = path.Join(s.matrixDocCloneURL)
+	var b bytes.Buffer
+	cmd.Stdout = &b
+	err := cmd.Run()
+	if err != nil {
+		return "", fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())
+	}
+	return strings.TrimSpace(b.String()), nil
+}
+
 func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 	var sha string
 
 	if strings.ToLower(req.URL.Path) == "/spec/head" {
-		sha = "HEAD"
+		originHead, err := s.getSHAOf("origin/master")
+		if err != nil {
+			writeError(w, 500, err)
+		}
+		sha = originHead
 	} else {
 		pr, err := lookupPullRequest(*req.URL, "/spec")
 		if err != nil {

From 69228506432f8e6781cfb89b2f378f0ad3ce5782 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 9 Oct 2015 11:02:27 +0100
Subject: [PATCH 292/467] Add missing return

---
 scripts/speculator/main.go | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index ad0a897b..d641d7da 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -178,6 +178,7 @@ func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 		originHead, err := s.getSHAOf("origin/master")
 		if err != nil {
 			writeError(w, 500, err)
+			return
 		}
 		sha = originHead
 	} else {

From e378de8379abcb85db8048b7dbd80b3851b4ac51 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 9 Oct 2015 11:04:11 +0100
Subject: [PATCH 293/467] Shuffle history visibility module to be in the module
 format

---
 specification/0-events.rst                   |  2 -
 specification/modules/history_visibility.rst | 44 +++++++++++++++-----
 2 files changed, 33 insertions(+), 13 deletions(-)

diff --git a/specification/0-events.rst b/specification/0-events.rst
index 16948462..e2373f96 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -26,8 +26,6 @@ prefixed with ``m.``
 
 {{m_room_create_event}}
 
-{{m_room_history_visibility_event}}
-
 {{m_room_join_rules_event}}
 
 {{m_room_member_event}}
diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst
index 371282bd..4f0c8976 100644
--- a/specification/modules/history_visibility.rst
+++ b/specification/modules/history_visibility.rst
@@ -1,21 +1,36 @@
 Room History Visibility
------------------------
+=======================
 
 .. _module:history-visibility:
 
-Whether a member of a room can see the events that happened in a room from
-before they joined the room is controlled by the ``history_visibility`` key
-of the ``m.room.history_visibility`` state event. The valid values for
-``history_visibility`` are:
+This module adds support for controlling the visibility of previous events in a
+room. Whether a member of a room can see the events that happened in a room from
+before they joined the room is controlled by the ``m.room.history_visibility``
+event outlined below. Visibility can take the form of one of three options:
+
+- ``shared`` - Previous events are always shown to newly joined members.
+- ``invited`` - Previous events are shown to newly joined members from the point
+  they were invited onwards.
+- ``joined`` - Previous events are shown to newly joined members from the point
+  they joined the room onwards.
+
+Events
+------
+
+{{m_room_history_visibility_event}}
 
-- ``shared``
-- ``invited``
-- ``joined``
+Client behaviour
+----------------
 
-By default if no ``history_visibility`` is set it is assumed to be ``shared``.
+Clients that implement this module MUST present to the user the possible options
+for setting history visibility when creating a room. 
 
-The rules governing whether a user is allowed to see an event depend solely on
-the state of the room at that event:
+Server behaviour
+----------------
+
+By default if no ``history_visibility`` is set the visibility is assumed to be
+``shared``. The rules governing whether a user is allowed to see an event depend
+solely on the state of the room *at that event*:
 
 1. If the user was joined, allow.
 2. If the user was invited and the ``history_visibility`` was set to
@@ -24,3 +39,10 @@ the state of the room at that event:
    was set to ``shared``, allow.
 4. Otherwise, deny.
 
+Security considerations
+-----------------------
+
+The default value for ``history_visibility`` is ``shared`` for
+backwards-compatibility reasons. Clients need to be aware that by not setting
+this event they are exposing all of their room history to anyone in the room.
+

From 89426c93224d005a30538f14d13ae8bd7b3370f2 Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Fri, 9 Oct 2015 17:43:46 +0100
Subject: [PATCH 294/467] typos and fixes

---
 supporting-docs/guides/2015-08-19-faq.md                  | 6 +++---
 supporting-docs/guides/2015-08-21-application_services.md | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/supporting-docs/guides/2015-08-19-faq.md b/supporting-docs/guides/2015-08-19-faq.md
index ce58c158..ff30827c 100644
--- a/supporting-docs/guides/2015-08-19-faq.md
+++ b/supporting-docs/guides/2015-08-19-faq.md
@@ -204,7 +204,7 @@ on:
 -   Disruptive net-splits
 -   Non-extensible
 
-[IRCv3](http://ircv3.net) exists and is addressing some of issues;
+[IRCv3](http://ircv3.net) exists and is addressing some of these issues;
 this is great news and we wish them well.  It's almost a contradiction
 in terms to get competitive between openly interoperable communication
 projects - we look forward to increasing the richness of Matrix\<-\>IRC
@@ -519,8 +519,8 @@ Follow the instructions for the homeserver you want to run. If you want to run S
 ##### Can I run my own identity server?
 
 Yes - the reference implementation is
-[sydent](https://github.com/matrix-org/sydent) and you can run your own
-ID server cluster that tracks 3rd party to Matrix ID mappings.  This won't be very useful right now, though, and we don't recommend it.
+[sydent](https://github.com/matrix-org/sydent) and you can run your own ID server cluster that tracks 3rd party to Matrix ID mappings.  This won't be very useful right now, though, and we don't recommend it.
+
 If you want your server to participate in the global replicated Matrix ID
 service then please get in touch with us. Meanwhile, we are looking at
 ways of decentralising the 'official' Matrix identity service so that
diff --git a/supporting-docs/guides/2015-08-21-application_services.md b/supporting-docs/guides/2015-08-21-application_services.md
index bd0453e8..dd532d42 100644
--- a/supporting-docs/guides/2015-08-21-application_services.md
+++ b/supporting-docs/guides/2015-08-21-application_services.md
@@ -17,7 +17,7 @@ One of the main use cases for application services is protocol bridges. Our Matr
 ### Some of the features of the IRC application service we have since implemented include:
 
 - Specific channel-to-matrix room bridging : This is what the original IRC bot did. You can specify specific channels and specific room IDs, and messages will be bridged.
-- Dynamic channel-to-matrix room bridging> : This allows Matrix users to join any channel on an IRC network, rather than being forced to use one of the specific channels configured.
+- Dynamic channel-to-matrix room bridging : This allows Matrix users to join any channel on an IRC network, rather than being forced to use one of the specific channels configured.
 - Two-way PM support : IRC users can PM the virtual "M-" users and private Matrix rooms will be created. Likewise, Matrix users can invite the virtual "@irc_Nick:domain" user IDs to a room and a PM to the IRC nick will be made.
 - IRC nick changing support: Matrix users are no longer forced to use "M-" nicks and can change them by sending "!nick" messages directly to the bridge.
 - Ident support: This allows usernames to be authenticated for virtual IRC clients, which means IRC bans can be targeted at the Matrix user rather than the entire application service.

From 8f5c796884bd0312318f6046faa4c9599be9dc37 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 12 Oct 2015 17:50:46 +0100
Subject: [PATCH 295/467] Linkify 'pushers'

---
 api/client-server/v1/pusher.yaml | 2 +-
 specification/modules/push.rst   | 4 ++++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/pusher.yaml b/api/client-server/v1/pusher.yaml
index 99a70ee7..8c243f2b 100644
--- a/api/client-server/v1/pusher.yaml
+++ b/api/client-server/v1/pusher.yaml
@@ -22,7 +22,7 @@ paths:
     post:
       summary: Modify a pusher for this user on the homeserver.
       description: |-
-        This endpoint allows the creation, modification and deletion of pushers
+        This endpoint allows the creation, modification and deletion of `pushers`_
         for this user ID. The behaviour of this endpoint varies depending on the
         values in the JSON body.
       security:
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index 7948475c..6e9c8536 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -64,6 +64,8 @@ Push Gateway
   devices or GCM for Android devices. Clients inform the homeserver which
   Push Gateway to send notifications to when it sets up a Pusher.
 
+.. _def:pushers:
+
 Pusher
   A pusher is a worker on the homeserver that manages the sending
   of HTTP notifications for a user. A user can have multiple pushers: one per
@@ -89,6 +91,8 @@ There is a single API endpoint for this, as described below.
 
 {{pusher_http_api}}
 
+.. _pushers: `def:pushers`_
+
 Push Rules
 ~~~~~~~~~~
 A push rule is a single rule that states under what *conditions* an event should

From 4a1fb74897d949cee53431dae03fe4d17c3fcaaa Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 12 Oct 2015 18:08:51 +0100
Subject: [PATCH 296/467] Review comments

---
 specification/modules/third_party_invites.rst | 26 +++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index 9b3b52e1..a9883db5 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -68,7 +68,7 @@ with the querystring
 ?public_key=``public_key``. A JSON object will be returned.
 The invitation is valid if the object contains a key named ``valid`` which is
 ``true``. Otherwise, the invitation MUST be rejected. This request is
-idempotent, and may be retried by the homeserver.
+idempotent and may be retried by the homeserver.
 
 If a homeserver is joining a room for the first time because of an
 ``m.room.third_party_invite``, the server which is already participating in the
@@ -83,24 +83,24 @@ membership is questionable.
 
 For example:
 
-If room R has two participating homeservers, H1, H2
+    If room R has two participating homeservers, H1, H2
 
-And user A on H1 invites a third party identifier to room R
+    And user A on H1 invites a third party identifier to room R
 
-H1 asks the identity server for a binding to a Matrix user ID, and has none,
-so issues an ``m.room.third_party_invite`` event to the room.
+    H1 asks the identity server for a binding to a Matrix user ID, and has none,
+    so issues an ``m.room.third_party_invite`` event to the room.
 
-When the third party user validates their identity, they are told about the
-invite, and ask their homeserver, H3, to join the room.
+    When the third party user validates their identity, they are told about the
+    invite, and ask their homeserver, H3, to join the room.
 
-H3 validates that sign(``token``, ``public_key``) = ``signature``, and may check
-``key_validity_url``.
+    H3 validates that sign(``token``, ``public_key``) = ``signature``, and may check
+    ``key_validity_url``.
 
-H3 then asks H1 to join it to the room. H1 *must* validate that
-sign(``token``, ``public_key``) = ``signature`` *and* check ``key_validity_url``.
+    H3 then asks H1 to join it to the room. H1 *must* validate that
+    sign(``token``, ``public_key``) = ``signature`` *and* check ``key_validity_url``.
 
-Having validated these things, H1 writes the join event to the room, and H3
-begins participating in the room. H2 *must* accept this event.
+    Having validated these things, H1 writes the join event to the room, and H3
+    begins participating in the room. H2 *must* accept this event.
 
 The reason that no other homeserver may reject the event based on checking
 ``key_validity_url`` is that we must ensure event acceptance is deterministic.

From ceebf34fca176ffce998f2de854dc3b7b573953b Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Mon, 12 Oct 2015 19:11:35 +0100
Subject: [PATCH 297/467] generate docs with new nav bar

---
 scripts/matrix-org-gendoc.sh | 50 ++++++++++++++++++++++++++++++++----
 1 file changed, 45 insertions(+), 5 deletions(-)

diff --git a/scripts/matrix-org-gendoc.sh b/scripts/matrix-org-gendoc.sh
index 18961046..187e27da 100755
--- a/scripts/matrix-org-gendoc.sh
+++ b/scripts/matrix-org-gendoc.sh
@@ -1,21 +1,61 @@
 #! /bin/bash
 
 if [ -z "$1" ]; then
-    echo "Expected /includes/nav.html file as arg."
+    echo "Expected /includes/head.html file as 1st arg."
     exit 1
 fi
 
-NAV_BAR=$1
+if [ -z "$2" ]; then
+    echo "Expected /includes/nav.html file as 2nd arg."
+    exit 1
+fi
+
+if [ -z "$3" ]; then
+    echo "Expected /includes/footer.html file as 3rd arg."
+    exit 1
+fi
+
+
+HEADER=$1
+NAV_BAR=$2
+FOOTER=$3
+
+if [ ! -f $HEADER ]; then
+    echo $HEADER " does not exist"
+    exit 1
+fi
 
 if [ ! -f $NAV_BAR ]; then
     echo $NAV_BAR " does not exist"
     exit 1
 fi
 
+if [ ! -f $FOOTER ]; then
+    echo $FOOTER " does not exist"
+    exit 1
+fi
+
 python gendoc.py
 
-perl -pi -e 's#<head>#<head><link rel="stylesheet" href="/site.css">#' gen/specification.html gen/howtos.html
+perl -MFile::Slurp -pi -e 'BEGIN { $header = read_file("'$HEADER'") } s#<head>#<head>$header#' gen/specification.html gen/howtos.html
 
-perl -MFile::Slurp -pi -e 'BEGIN { $nav = read_file("'$NAV_BAR'") } s#<body>#<body><div id="header"><div id="headerContent">$nav</div></div><div id="page"><div id="wrapper"><div style="text-align: center; padding: 40px;"><a href="/"><img src="/matrix.png" width="305" height="130" alt="[matrix]"/></a></div>#' gen/specification.html gen/howtos.html
+perl -MFile::Slurp -pi -e 'BEGIN { $nav = read_file("'$NAV_BAR'") } s#<body>#  <body class="blog et_fixed_nav et_cover_background et_right_sidebar">
+   <div id="page-wrapper">
+    <div class="page-content" id="page-container">
+      $nav
+       <div id="main-content">
+         <div class="wrapper" id="wrapper">
+           <div class="document_foo" id="document">
+#' gen/specification.html gen/howtos.html
 
-perl -pi -e 's#</body>#</div></div><div id="footer"><div id="footerContent">&copy 2014-2015 Matrix.org</div></div></body>#' gen/specification.html gen/howtos.html
+perl -MFile::Slurp -pi -e 'BEGIN { $footer = read_file("'$FOOTER'") } s#</body>#
+            </div>
+          </div>
+          <div class="push">
+          </div>
+        </div>
+      </div>
+        $footer
+      </div>
+    </div>
+  </body>#' gen/specification.html gen/howtos.html

From 1cfe4f784f450a9d6ac6f9cb3f64fc50e664ad1f Mon Sep 17 00:00:00 2001
From: Oddvar Lovaas <oddvar@matrix.org>
Date: Mon, 12 Oct 2015 19:57:41 +0100
Subject: [PATCH 298/467] special css for spec/howto

---
 scripts/matrix-org-gendoc.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/matrix-org-gendoc.sh b/scripts/matrix-org-gendoc.sh
index 187e27da..5716786d 100755
--- a/scripts/matrix-org-gendoc.sh
+++ b/scripts/matrix-org-gendoc.sh
@@ -37,7 +37,9 @@ fi
 
 python gendoc.py
 
-perl -MFile::Slurp -pi -e 'BEGIN { $header = read_file("'$HEADER'") } s#<head>#<head>$header#' gen/specification.html gen/howtos.html
+perl -MFile::Slurp -pi -e 'BEGIN { $header = read_file("'$HEADER'") } s#<head>#<head>$header
+  <link rel="stylesheet" href="//matrix.org/docs/guides/css/docs_overrides.css">
+#' gen/specification.html gen/howtos.html
 
 perl -MFile::Slurp -pi -e 'BEGIN { $nav = read_file("'$NAV_BAR'") } s#<body>#  <body class="blog et_fixed_nav et_cover_background et_right_sidebar">
    <div id="page-wrapper">

From 173d00cea04e3c607992396b7395741971578e35 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 13 Oct 2015 11:08:19 +0100
Subject: [PATCH 299/467] Review comments

---
 specification/modules/history_visibility.rst | 24 ++++++++++++++++----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst
index 4f0c8976..12ea7b22 100644
--- a/specification/modules/history_visibility.rst
+++ b/specification/modules/history_visibility.rst
@@ -6,13 +6,27 @@ Room History Visibility
 This module adds support for controlling the visibility of previous events in a
 room. Whether a member of a room can see the events that happened in a room from
 before they joined the room is controlled by the ``m.room.history_visibility``
-event outlined below. Visibility can take the form of one of three options:
-
-- ``shared`` - Previous events are always shown to newly joined members.
+event outlined below. In all cases, the member still needs to be joined to the
+room to receive events for that room. The visibility option simply determines
+which subset of events in the room are presented to the client. Visibility can
+take the form of one of three options:
+
+- ``shared`` - Previous events are always shown to newly joined members. All
+  events in the room are shown, even those sent when the member was not a part
+  of the room.
 - ``invited`` - Previous events are shown to newly joined members from the point
-  they were invited onwards.
+  they were invited onwards. Events stop being shown when the member's state
+  changes to something other than ``invite`` or ``join``.
 - ``joined`` - Previous events are shown to newly joined members from the point
-  they joined the room onwards.
+  they joined the room onwards. Events stop being shown when the members state
+  changes to something other than ``join``.
+
+.. WARNING::
+  These options are applied at the point an event is *sent*. Checks are
+  performed with the state of the ``m.room.history_visibility`` event when the
+  event in question is added to the DAG. This means clients cannot
+  retrospectively choose to show or hide history to new users if the setting at
+  that time was more restrictive.
 
 Events
 ------

From 4170dbd5cf065f7e1c7fde32f4c5a6e622c98055 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 13 Oct 2015 11:29:54 +0100
Subject: [PATCH 300/467] Review comments

---
 event-schemas/schema/v1/m.room.message      |  2 +-
 specification/modules/instant_messaging.rst | 44 +++++++++------------
 2 files changed, 19 insertions(+), 27 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message
index 27b7e925..91c04b7f 100644
--- a/event-schemas/schema/v1/m.room.message
+++ b/event-schemas/schema/v1/m.room.message
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "Message",
-    "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.",
+    "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message. This allows clients to display *something* even if it is just plain text.",
     "allOf": [{
         "$ref": "core-event-schema/room_event.json"
     }],
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 426189b5..487240de 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -12,15 +12,6 @@ Events
 
 {{m_room_message_event}}
 
-
-.. admonition:: Rationale
-
-  Not all clients can display all message types. The most commonly supported
-  message type is raw text. As a result, we chose to have a textual fallback
-  display method represented by the ``body`` key. This means that even if the
-  client cannot display a particular ``msgtype``, they can still display
-  *something*, even if it is just plain text.
-
 {{m_room_message_feedback_event}}
 
 Usage of this event is discouraged for several reasons:
@@ -44,7 +35,7 @@ m.room.message msgtypes
 
 Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type
 of message being sent. Each type has their own required and optional keys, as
-outlined below. If a client cannot display the given ``msgtype`` then it MUST
+outlined below. If a client cannot display the given ``msgtype`` then it SHOULD
 display the fallback plain text ``body`` key instead.
 
 {{msgtype_events}}
@@ -62,9 +53,7 @@ be removed entirely from the messages view.
 
 Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be
 uploaded using the `content repository module`_ where available. The
-resulting ``mxc://`` URI can then be used in the ``url`` key. The
-attachment SHOULD be uploaded *prior* to sending the event in order to stop a
-race condition where the recipient receives a link to a non-existent attachment.
+resulting ``mxc://`` URI can then be used in the ``url`` key.
 
 .. _`content repository module`: `module:content`_
 
@@ -95,12 +84,15 @@ Local echo
 Messages SHOULD appear immediately in the message view when a user presses the
 "send" button. This should occur even if the message is still sending. This is
 referred to as "local echo". Clients SHOULD implement "local echo" of messages.
- 
-Clients need to be able to pair up the "remote echo" from the server with the
-"local echo" to prevent duplicate messages being displayed. Ideally this pairing
-would occur transparently to the user: the UI would not flicker as it transitions
-from local to remote. Flickering cannot be fully avoided in version 1 of the
-client-server API. Two scenarios need to be considered:
+
+Clients need to be able to match the message they are sending with the same
+message which they receive from the event stream. The echo of the same message
+from the event stream is referred to as "remote echo". Both echoes need to be
+identified as the same message in order to prevent duplicate messages being
+displayed. Ideally this pairing would occur transparently to the user: the UI
+would not flicker as it transitions from local to remote. Flickering cannot be
+fully avoided in version 1 of the client-server API. Two scenarios need to be
+considered:
 
 - The client sends a message and the remote echo arrives on the event stream
   *after* the request to send the message completes.
@@ -123,8 +115,8 @@ Displaying membership information with messages
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Clients may wish to show the display name and avatar URL of the room member who
-sent a message. This can be achieved by inspecting the ``m.room.member`` event
-for that user ID.
+sent a message. This can be achieved by inspecting the ``m.room.member`` state
+event for that user ID.
 
 When a user paginates the message history, clients may wish to show the
 **historical** display name and avatar URL for a room member. This is possible
@@ -139,15 +131,15 @@ This can then be used to set the historical display name and avatar URL.
 Server behaviour
 ----------------
 
-Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body``
-and ``msgtype`` keys by 400ing the request to send a message.
+Homeservers SHOULD reject ``m.room.message`` events which don't have a
+``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status
+code of 400.
 
 Security considerations
 -----------------------
 
-Messages sent using this module are not encrypted. Encryption can be layered
-over the top of this module: where the plaintext format is an ``m.room.message``
-conforming to this module. This can be achieved using the `E2E module`_.
+Messages sent using this module are not encrypted. Messages can be encrypted
+using the `E2E module`_.
 
 Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site
 Scripting (XSS) attacks. This includes room names and topics.

From e561a663d30810c0e58ca30b89bc3bc759e92aec Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Tue, 13 Oct 2015 16:53:27 +0100
Subject: [PATCH 301/467] Tweaks to intro, start using swagger APIs in the CS
 API section.

---
 specification/0-intro.rst             | 46 ++++++++++++---------------
 specification/1-client_server_api.rst | 44 ++-----------------------
 2 files changed, 22 insertions(+), 68 deletions(-)

diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index 445b32ef..1c18a4f0 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -99,23 +99,23 @@ Architecture
 ------------
 
 Matrix defines APIs for synchronising extensible JSON objects known as
-``events`` between compatible clients, servers and services. Clients are
+"events" between compatible clients, servers and services. Clients are
 typically messaging/VoIP applications or IoT devices/hubs and communicate by
-synchronising communication history with their ``homeserver`` using the
-``Client-Server API``. Each homeserver stores the communication history and
+synchronising communication history with their "homeserver" using the
+"Client-Server API". Each homeserver stores the communication history and
 account information for all of its clients, and shares data with the wider
 Matrix ecosystem by synchronising communication history with other homeservers
 and their clients.
 
 Clients typically communicate with each other by emitting events in the
-context of a virtual ``room``. Room data is replicated across *all of the
+context of a virtual "room". Room data is replicated across *all of the
 homeservers* whose users are participating in a given room. As such, *no
 single homeserver has control or ownership over a given room*. Homeservers
 model communication history as a partially ordered graph of events known as
-the room's ``event graph``, which is synchronised with eventual consistency
-between the participating servers using the ``Server-Server API``. This process
+the room's "event graph", which is synchronised with eventual consistency
+between the participating servers using the "Server-Server API". This process
 of synchronising shared conversation history between homeservers run by
-different parties is called ``Federation``. Matrix optimises for the the
+different parties is called "Federation". Matrix optimises for the the
 Availability and Partitioned properties of CAP theorem at
 the expense of Consistency.
 
@@ -151,13 +151,13 @@ Users
 ~~~~~
 
 Each client is associated with a user account, which is identified in Matrix
-using a unique "User ID". This ID is namespaced to the home server which
+using a unique "User ID". This ID is namespaced to the homeserver which
 allocated the account and has the form::
 
   @localpart:domain
 
 The ``localpart`` of a user ID may be a user name, or an opaque ID identifying
-this user. They are case-insensitive.
+this user. The ``domain`` of a user ID is the domain of the homeserver.
 
 .. TODO-spec
     - Need to specify precise grammar for Matrix IDs
@@ -183,9 +183,9 @@ Event Graphs
 .. _sect:event-graph:
 
 Events exchanged in the context of a room are stored in a directed acyclic graph
-(DAG) called an ``event graph``. The partial ordering of this graph gives the
+(DAG) called an "event graph". The partial ordering of this graph gives the
 chronological ordering of events within the room. Each event in the graph has a
-list of zero or more ``parent`` events, which refer to any preceding events
+list of zero or more "parent" events, which refer to any preceding events
 which have no chronological successor from the perspective of the homeserver
 which created the event.
 
@@ -292,11 +292,10 @@ Each room can also have multiple "Room Aliases", which look like::
 
 A room alias "points" to a room ID and is the human-readable label by which
 rooms are publicised and discovered.  The room ID the alias is pointing to can
-be obtained by visiting the domain specified. They are case-insensitive. Note
-that the mapping from a room alias to a room ID is not fixed, and may change
-over time to point to a different room ID. For this reason, Clients SHOULD
-resolve the room alias to a room ID once and then use that ID on subsequent
-requests.
+be obtained by visiting the domain specified. Note that the mapping from a room
+alias to a room ID is not fixed, and may change over time to point to a
+different room ID. For this reason, Clients SHOULD resolve the room alias to a
+room ID once and then use that ID on subsequent requests.
 
 When resolving a room alias the server will also respond with a list of servers
 that are in the room that can be used to join via.
@@ -339,12 +338,9 @@ Profiles
 ~~~~~~~~
 
 Users may publish arbitrary key/value data associated with their account - such
-as a human readable ``display name``, a profile photo URL, contact information
+as a human readable display name, a profile photo URL, contact information
 (email address, phone numbers, website URLs etc).
 
-In Client-Server API v2, profile data is typed using namespaced keys for
-interoperability, much like events - e.g. ``m.profile.display_name``.
-
 .. TODO
   Actually specify the different types of data - e.g. what format are display
   names allowed to be?
@@ -431,13 +427,11 @@ Some requests have unique error codes:
 :``M_BAD_PAGINATION``:
   Encountered when specifying bad pagination query parameters.
 
-:``M_LOGIN_EMAIL_URL_NOT_YET``:
-  Encountered when polling for an email link which has not been clicked yet.
 
-The C-S API typically uses ``HTTP POST`` to submit requests. This means these
-requests are not idempotent. The C-S API also allows ``HTTP PUT`` to make
-requests idempotent. In order to use a ``PUT``, paths should be suffixed with
-``/{txnId}``. ``{txnId}`` is a unique client-generated transaction ID which
+The Client-Server API typically uses ``HTTP POST`` to submit requests. This
+means these requests are not idempotent. The C-S API also allows ``HTTP PUT`` to
+make requests idempotent. In order to use a ``PUT``, paths should be suffixed
+with ``/{txnId}``. ``{txnId}`` is a unique client-generated transaction ID which
 identifies the request, and is scoped to a given Client (identified by that
 client's ``access_token``). Crucially, it **only** serves to identify new
 requests from retransmits. After the request has finished, the ``{txnId}``
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 59e6b68e..94d1a184 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -528,7 +528,7 @@ Room events are split into two categories:
 :Message events:
   These are events which describe transient "once-off" activity in a room:
   typically communication such as sending an instant message or setting up a
-  VoIP call. These used to be called 'non-state' events.
+  VoIP call.
 
 This specification outlines several events, all with the event type prefix
 ``m.``. However, applications may wish to add their own type of event, and this
@@ -600,9 +600,6 @@ See `Room Events`_ for the ``m.`` event specification.
 Syncing rooms
 ~~~~~~~~~~~~~
 
-.. NOTE::
-  This section is a work in progress.
-
 When a client logs in, they may have a list of rooms which they have already
 joined. These rooms may also have a list of events associated with them. The
 purpose of 'syncing' is to present the current room and event information in a
@@ -620,45 +617,8 @@ presence events will also be returned. A single syncing API is provided:
    onwards. The event stream cannot do this for a single room currently.
    As a result, commenting room-scoped initial sync at this time.
 
-The |initialSync|_ API contains the following keys:
-
-``presence``
-  Description:
-    Contains a list of presence information for users the client is interested
-    in.
-  Format:
-    A JSON array of ``m.presence`` events.
-
-``end``
-  Description:
-    Contains an event stream token which can be used with the `Event Stream`_.
-  Format:
-    A string containing the event stream token.
 
-``rooms``
-  Description:
-    Contains a list of room information for all rooms the client has joined,
-    and limited room information on rooms the client has been invited to.
-  Format:
-    A JSON array containing Room Information JSON objects.
-
-Room Information:
-  Description:
-    Contains all state events for the room, along with a limited amount of
-    the most recent events, configured via the ``limit`` query
-    parameter. Also contains additional keys with room metadata, such as the
-    ``room_id`` and the client's ``membership`` to the room.
-  Format:
-    A JSON object with the following keys:
-      ``room_id``
-        A string containing the ID of the room being described.
-      ``membership``
-        A string representing the client's membership status in this room.
-      ``messages``
-        An event stream JSON object containing a ``chunk`` of recent
-        events (both state events and non-state events), along with an ``end`` token.
-      ``state``
-        A JSON array containing all the current state events for this room.
+{{sync_http_api}}
 
 Getting events for a room
 ~~~~~~~~~~~~~~~~~~~~~~~~~

From e716e819635babb0216c971940f9e7a8cdbeacc4 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 09:58:03 +0100
Subject: [PATCH 302/467] Add `dir` parameter to pagination. Remove path
 references

This section needs a lot of work to not lie.
---
 specification/1-client_server_api.rst | 30 +++++++++++++++------------
 1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 94d1a184..a4f62b4f 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -348,11 +348,13 @@ Pagination
 
 Querying large datasets in Matrix always uses the same pagination API pattern to
 to give clients a consistent way of selecting subsets of a potentially changing
-dataset. Requests pass in ``from``, ``to`` and ``limit`` parameters which describe
-where to read from the stream. ``from`` and ``to`` are opaque textual 'stream
-tokens' which describe positions in the dataset. The response returns new
-``start`` and ``end`` stream token values which can then be passed to subsequent
-requests to continue pagination.
+dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
+which describe where to read from the stream. ``from`` and ``to`` are opaque
+textual 'stream tokens' which describe the current position in the dataset.
+The ``dir`` parameter is an enum representing the direction of events to return:
+either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and
+``end`` stream token values which can then be passed to subsequent requests to
+continue pagination.
 
 Pagination Request Query Parameters
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -367,24 +369,26 @@ Query parameters:
   limit:
     integer - An integer representing the maximum number of items to 
     return.
+  dir:
+    f|b - The direction to return events in. Typically this is ``b`` to paginate
+    backwards in time.
 
 'START' and 'END' are placeholder values used in these examples to describe the
 start and end of the dataset respectively.
 
-Unless specified, the default pagination parameters are from=START, to=END, 
-without a limit set. This allows you to hit an API like
-/events without any query parameters to get everything.
+Unless specified, the default pagination parameters are ``from=START``,
+``to=END``, without a limit set.
 
-For example, the event stream has events E1 -> E15. The client wants the last 5 
+For example, if an endpoint had events E1 -> E15. The client wants the last 5 
 events and doesn't know any previous events::
 
     S                                                    E
     |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
     |                               |                    |
     |                          _____|                    |
-    |__________________       |       ___________________|
-                       |      |      |
-     GET /events?to=START&limit=5&from=END
+    |__________________       |           _______________|
+                       |      |           |
+     GET /somepath?to=START&limit=5&from=END
      Returns:
        E15,E14,E13,E12,E11
 
@@ -401,7 +405,7 @@ now show page 3 (rooms R11 -> 15)::
                         Currently            |
                         viewing              |
                                              |
-                             GET /rooms/list?from=9&to=END&limit=5
+                             GET /roomslist?from=9&to=END&limit=5
                              Returns: R11,R12,R13,R14,R15
                          
 Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 

From 7bdb71b1c9697eb9ded20e028723a500b1d71a02 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 10:25:39 +0100
Subject: [PATCH 303/467] Tweak the syncing section

Bring together disparate sections to make it more cohesive.
---
 api/client-server/v1/sync.yaml        |  6 +-
 specification/1-client_server_api.rst | 99 +++++++++++----------------
 2 files changed, 43 insertions(+), 62 deletions(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 7c1d43f3..398b0773 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -30,7 +30,9 @@ paths:
         - in: query
           type: string
           name: from
-          description: The token to stream from.
+          description: |-
+            The token to stream from. This token is either from the a previous
+            request to this API or from the initial sync API.
           required: false
           x-example: "s3456_9_0"
         - in: query
@@ -40,7 +42,7 @@ paths:
           required: false
           x-example: "35000"
         - in: query
-          type: string
+          type: boolean
           name: archived
           description: |-
             Whether to include rooms that the user has left. If absent then
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index a4f62b4f..fefdf511 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1,9 +1,6 @@
 Client-Server API
 =================
 
-Overview
---------
-
 The client-server API provides a simple lightweight API to let clients send
 messages, control rooms and synchronise conversation history. It is designed to
 support both lightweight clients which store no state and lazy-load data from
@@ -31,7 +28,7 @@ return with a status of 401 and the error code, ``M_MISSING_TOKEN`` or
 ``M_UNKNOWN_TOKEN`` respectively.
 
 User-Interactive Authentication API
------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 This section refers to API Version 2.
 
 Some API endpoints such as ``login`` or ``register`` require authentication that
@@ -159,7 +156,7 @@ absence of that login stage type in the 'completed' array indicating whether
 that stage is complete.
 
 Example
-~~~~~~~
++++++++
 At a high level, the requests made for an API call completing an auth flow with
 three stages will resemble the following diagram::
   
@@ -201,7 +198,7 @@ This specification defines the following login types:
  - ``m.login.dummy``
 
 Password-based
-~~~~~~~~~~~~~~
+++++++++++++++
 :Type:
   ``m.login.password``
 :Description:
@@ -216,7 +213,7 @@ To respond to this type, reply with an auth dict as follows::
   }
 
 Google ReCaptcha
-~~~~~~~~~~~~~~~~
+++++++++++++++++
 :Type:
   ``m.login.recaptcha``
 :Description:
@@ -230,7 +227,7 @@ To respond to this type, reply with an auth dict as follows::
   }
 
 Token-based
-~~~~~~~~~~~
++++++++++++
 :Type:
   ``m.login.token``
 :Description:
@@ -261,7 +258,7 @@ newly provisioned access_token).
 The ``token`` must be a macaroon.
 
 OAuth2-based
-~~~~~~~~~~~~
+++++++++++++
 :Type:
   ``m.login.oauth2``
 :Description:
@@ -285,7 +282,7 @@ the OAuth flow has completed, the client retries the request with the session
 only, as above.
 
 Email-based (identity server)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
++++++++++++++++++++++++++++++
 :Type:
   ``m.login.email.identity``
 :Description:
@@ -310,7 +307,7 @@ To respond to this type, reply with an auth dict as follows::
   }
 
 Dummy Auth
-~~~~~~~~~~
+++++++++++
 :Type:
   ``m.login.dummy``
 :Description:
@@ -327,7 +324,7 @@ if provided::
 
 
 Fallback
-~~~~~~~~
+++++++++
 Clients cannot be expected to be able to know how to process every single login
 type. If a client does not know how to handle a given login type, it can direct
 the user to a web browser with the URL of a fallback page which will allow the
@@ -346,7 +343,15 @@ the authentication has been completed.
 Pagination
 ----------
 
-Querying large datasets in Matrix always uses the same pagination API pattern to
+.. NOTE::
+  The paths referred to in this section are not actual endpoints. They only
+  serve as examples to explain how pagination functions.
+
+Pagination is the process of dividing a dataset into multiple discrete pages.
+Matrix makes use of pagination to allow clients to view extremely large datasets.
+These datasets are not limited to events in a room (for example clients may want
+to paginate rooms in addition to events within those rooms). Regardless of *what*
+is being paginated, there is a common underlying API which is used to
 to give clients a consistent way of selecting subsets of a potentially changing
 dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
 which describe where to read from the stream. ``from`` and ``to`` are opaque
@@ -354,7 +359,8 @@ textual 'stream tokens' which describe the current position in the dataset.
 The ``dir`` parameter is an enum representing the direction of events to return:
 either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and
 ``end`` stream token values which can then be passed to subsequent requests to
-continue pagination.
+continue pagination. Not all endpoints will make use of all the parameters
+outlined here: see the specific endpoint in question for more information.
 
 Pagination Request Query Parameters
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -385,10 +391,10 @@ events and doesn't know any previous events::
     S                                                    E
     |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
     |                               |                    |
-    |                          _____|                    |
-    |__________________       |           _______________|
-                       |      |           |
-     GET /somepath?to=START&limit=5&from=END
+    |                          _____|  <--backwards--    |
+    |__________________       |         |        ________|
+                       |      |         |        |
+     GET /somepath?to=START&limit=5&dir=b&from=END
      Returns:
        E15,E14,E13,E12,E11
 
@@ -433,9 +439,6 @@ Events
 
 .. _sect:events:
 
-Overview
-~~~~~~~~
-
 The model of conversation history exposed by the client-server API can be
 considered as a list of events. The server 'linearises' the
 eventually-consistent event graph of events into an 'event stream' at any given
@@ -463,7 +466,7 @@ You can visualise the range of events being returned as::
                               |                             |
                         start: '1-2-3'                end: 'a-b-c'
                              
-Now, to receive future events in real-time on the eventstream, you simply GET
+Now, to receive future events in real-time on the event stream, you simply GET
 $PREFIX/events with a ``from`` parameter of 'a-b-c': in other words passing in the
 ``end`` token returned by initial sync. The request blocks until new events are
 available or until your specified timeout elapses, and then returns a
@@ -493,31 +496,30 @@ To continue paginating backwards, one calls the /messages API again, supplying
 the new ``start`` value as the ``from`` parameter.
 
 
-Receiving live updates on a client
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Syncing
+~~~~~~~
 
-Clients receive new events by long-polling the home server via the
-$PREFIX/events API, specifying a timeout in milliseconds in the timeout
-parameter. This will hold open the HTTP connection for a short period of time
-waiting for new events, returning early if an event occurs. This is called the
-`Event Stream`_. All events which are visible to the client will appear in the
-event stream. When the request returns, an ``end`` token is included in the
+Clients receive new events by "long-polling" the home server via the events API.
+This involves specifying a timeout in the request which will hold
+open the HTTP connection for a short period of time waiting for new events,
+returning early if an event occurs. Only the events API supports long-polling.
+All events which are visible to the client will appear in the
+events API. When the request returns, an ``end`` token is included in the
 response. This token can be used in the next request to continue where the
-last request left off.
-
-All events must be de-duplicated based on their event ID.
+last request left off. Multiple events can be returned per long-poll. All events
+must be de-duplicated based on their event ID.
 
 .. TODO
   is deduplication actually a hard requirement in CS v2?
 
 .. TODO-spec
-  Do we ever return multiple events in a single request?
-  Don't we get lots of request setup RTT latency if we only do one event per request?
   Do we ever support streaming requests? Why not websockets?
 
 When the client first logs in, they will need to initially synchronise with
-their home server. This is achieved via the |initialSync|_ API. This API also
-returns an ``end`` token which can be used with the event stream.  See the 'Room Sync' section below.
+their home server. This is achieved via the initial sync API. This API also
+returns an ``end`` token which can be used with the event stream.
+
+{{sync_http_api}}
 
 Events in a room
 ~~~~~~~~~~~~~~~~
@@ -601,29 +603,6 @@ example::
 
 See `Room Events`_ for the ``m.`` event specification.
 
-Syncing rooms
-~~~~~~~~~~~~~
-
-When a client logs in, they may have a list of rooms which they have already
-joined. These rooms may also have a list of events associated with them. The
-purpose of 'syncing' is to present the current room and event information in a
-convenient, compact manner. The events returned are not limited to room events;
-presence events will also be returned. A single syncing API is provided:
-
- - |initialSync|_ : A global sync which will present room and event information
-   for all rooms the user has joined.
-
-.. TODO-spec room-scoped initial sync
- - |/rooms/<room_id>/initialSync|_ : A sync scoped to a single room. Presents
-   room and event information for this room only.
- - Room-scoped initial sync is Very Tricky because typically people would
-   want to sync the room then listen for any new content from that point
-   onwards. The event stream cannot do this for a single room currently.
-   As a result, commenting room-scoped initial sync at this time.
-
-
-{{sync_http_api}}
-
 Getting events for a room
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 

From 07d7a3fa3a4d0c9d7e4db70d62a24bc266824dbb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 11:03:53 +0100
Subject: [PATCH 304/467] Shuffle pagination section around

---
 specification/1-client_server_api.rst | 201 +++++++++++++-------------
 1 file changed, 103 insertions(+), 98 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index fefdf511..d8b9299a 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -340,100 +340,6 @@ This MUST return an HTML page which can perform this authentication stage. This
 page must attempt to call the JavaScript function ``window.onAuthDone`` when
 the authentication has been completed.
 
-Pagination
-----------
-
-.. NOTE::
-  The paths referred to in this section are not actual endpoints. They only
-  serve as examples to explain how pagination functions.
-
-Pagination is the process of dividing a dataset into multiple discrete pages.
-Matrix makes use of pagination to allow clients to view extremely large datasets.
-These datasets are not limited to events in a room (for example clients may want
-to paginate rooms in addition to events within those rooms). Regardless of *what*
-is being paginated, there is a common underlying API which is used to
-to give clients a consistent way of selecting subsets of a potentially changing
-dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
-which describe where to read from the stream. ``from`` and ``to`` are opaque
-textual 'stream tokens' which describe the current position in the dataset.
-The ``dir`` parameter is an enum representing the direction of events to return:
-either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and
-``end`` stream token values which can then be passed to subsequent requests to
-continue pagination. Not all endpoints will make use of all the parameters
-outlined here: see the specific endpoint in question for more information.
-
-Pagination Request Query Parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-       
-Query parameters:
-  from:
-    $streamtoken - The opaque token to start streaming from.
-  to:
-    $streamtoken - The opaque token to end streaming at. Typically,
-    clients will not know the item of data to end at, so this will usually be 
-    omitted.
-  limit:
-    integer - An integer representing the maximum number of items to 
-    return.
-  dir:
-    f|b - The direction to return events in. Typically this is ``b`` to paginate
-    backwards in time.
-
-'START' and 'END' are placeholder values used in these examples to describe the
-start and end of the dataset respectively.
-
-Unless specified, the default pagination parameters are ``from=START``,
-``to=END``, without a limit set.
-
-For example, if an endpoint had events E1 -> E15. The client wants the last 5 
-events and doesn't know any previous events::
-
-    S                                                    E
-    |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
-    |                               |                    |
-    |                          _____|  <--backwards--    |
-    |__________________       |         |        ________|
-                       |      |         |        |
-     GET /somepath?to=START&limit=5&dir=b&from=END
-     Returns:
-       E15,E14,E13,E12,E11
-
-
-Another example: a public room list has rooms R1 -> R17. The client is showing 5 
-rooms at a time on screen, and is on page 2. They want to
-now show page 3 (rooms R11 -> 15)::
-
-    S                                                           E
-    |  0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 | stream token
-    |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room
-                      |____________| |________________|
-                            |                |
-                        Currently            |
-                        viewing              |
-                                             |
-                             GET /roomslist?from=9&to=END&limit=5
-                             Returns: R11,R12,R13,R14,R15
-                         
-Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 
-token from the initial request was '9' which corresponded to R10. When the 2nd
-request was made, R10 did not appear again, even though from=9 was specified. If
-you know the token, you already have the data.
-
-Pagination Response
-~~~~~~~~~~~~~~~~~~~
-
-Responses to pagination requests MUST follow the format::
-
-  {
-    "chunk": [ ... , Responses , ... ],
-    "start" : $streamtoken,
-    "end" : $streamtoken
-  }
-
-Where $streamtoken is an opaque token which can be used in another query to
-get the next set of results. The "start" and "end" keys can only be omitted if
-the complete dataset is provided in "chunk".
-
 Events
 ------
 
@@ -521,8 +427,8 @@ returns an ``end`` token which can be used with the event stream.
 
 {{sync_http_api}}
 
-Events in a room
-~~~~~~~~~~~~~~~~
+Types of room events
+~~~~~~~~~~~~~~~~~~~~
 
 Room events are split into two categories:
 
@@ -544,7 +450,7 @@ convention, e.g. ``com.example.myapp.event``.  This ensures event types are
 suitably namespaced for each application and reduces the risk of clashes.
 
 State events
-~~~~~~~~~~~~
+++++++++++++
 
 State events can be sent by ``PUT`` ing to
 |/rooms/<room_id>/state/<event_type>/<state_key>|_.  These events will be
@@ -587,7 +493,7 @@ In some cases, there may be no need for a ``state_key``, so it can be omitted::
 See `Room Events`_ for the ``m.`` event specification.
 
 Message events
-~~~~~~~~~~~~~~
+++++++++++++++
 
 Message events can be sent by sending a request to
 |/rooms/<room_id>/send/<event_type>|_.  These requests *can* use transaction
@@ -655,6 +561,105 @@ The redaction event should be added under the key ``redacted_because``. When a
 client receives a redaction event it should change the redacted event
 in the same way a server does.
 
+Pagination
+----------
+
+.. NOTE::
+  The paths referred to in this section are not actual endpoints. They only
+  serve as examples to explain how pagination functions.
+
+Pagination is the process of dividing a dataset into multiple discrete pages.
+Matrix makes use of pagination to allow clients to view extremely large datasets.
+These datasets are not limited to events in a room (for example clients may want
+to paginate rooms in addition to events within those rooms). Regardless of *what*
+is being paginated, there is a common underlying API which is used to
+to give clients a consistent way of selecting subsets of a potentially changing
+dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
+which describe where to read from the stream. ``from`` and ``to`` are opaque
+textual 'stream tokens' which describe the current position in the dataset.
+The ``dir`` parameter is an enum representing the direction of events to return:
+either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and
+``end`` stream token values which can then be passed to subsequent requests to
+continue pagination. Not all endpoints will make use of all the parameters
+outlined here: see the specific endpoint in question for more information.
+
+Pagination Request Query Parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+       
+Query parameters:
+  from:
+    $streamtoken - The opaque token to start streaming from.
+  to:
+    $streamtoken - The opaque token to end streaming at. Typically,
+    clients will not know the item of data to end at, so this will usually be 
+    omitted.
+  limit:
+    integer - An integer representing the maximum number of items to 
+    return.
+  dir:
+    f|b - The direction to return events in. Typically this is ``b`` to paginate
+    backwards in time.
+
+'START' and 'END' are placeholder values used in these examples to describe the
+start and end of the dataset respectively.
+
+Unless specified, the default pagination parameters are ``from=START``,
+``to=END``, without a limit set.
+
+For example, if an endpoint had events E1 -> E15. The client wants the last 5 
+events and doesn't know any previous events::
+
+    S                                                    E
+    |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
+    |                               |                    |
+    |                          _____|  <--backwards--    |
+    |__________________       |         |        ________|
+                       |      |         |        |
+     GET /somepath?to=START&limit=5&dir=b&from=END
+     Returns:
+       E15,E14,E13,E12,E11
+
+
+Another example: a public room list has rooms R1 -> R17. The client is showing 5 
+rooms at a time on screen, and is on page 2. They want to
+now show page 3 (rooms R11 -> 15)::
+
+    S                                                           E
+    |  0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 | stream token
+    |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room
+                      |____________| |________________|
+                            |                |
+                        Currently            |
+                        viewing              |
+                                             |
+                             GET /roomslist?from=9&to=END&limit=5
+                             Returns: R11,R12,R13,R14,R15
+                         
+Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 
+token from the initial request was '9' which corresponded to R10. When the 2nd
+request was made, R10 did not appear again, even though from=9 was specified. If
+you know the token, you already have the data.
+
+Pagination Response
+~~~~~~~~~~~~~~~~~~~
+
+Responses to pagination requests MUST follow the format::
+
+  {
+    "chunk": [ ... , Responses , ... ],
+    "start" : $streamtoken,
+    "end" : $streamtoken
+  }
+
+Where $streamtoken is an opaque token which can be used in another query to
+get the next set of results. The "start" and "end" keys can only be omitted if
+the complete dataset is provided in "chunk".
+
+Pagination APIs
+~~~~~~~~~~~~~~~
+
+{{message_pagination_http_api}}
+
 Rooms
 -----
 

From 41fb0645a975d5bfb4e6394db1f7c4fef59463eb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 11:16:58 +0100
Subject: [PATCH 305/467] Add message pagination API

---
 api/client-server/v1/message_pagination.yaml | 131 +++++++++++++++++++
 1 file changed, 131 insertions(+)
 create mode 100644 api/client-server/v1/message_pagination.yaml

diff --git a/api/client-server/v1/message_pagination.yaml b/api/client-server/v1/message_pagination.yaml
new file mode 100644
index 00000000..b3317c13
--- /dev/null
+++ b/api/client-server/v1/message_pagination.yaml
@@ -0,0 +1,131 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Rooms API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/messages":
+    get:
+      summary: Get a list of events for this room
+      description: |-
+        This API returns a list of message and state events for a room. It uses
+        pagination query parameters to paginate history in the room.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to get events from.
+          required: true
+          x-example: "!636q39766251:example.com"
+        - in: query
+          type: string
+          name: from
+          description: |-
+            The token to start returning events from. This token can be obtained
+            from the initial sync API.
+          required: true
+          x-example: "s345_678_333"
+        - in: query
+          type: string
+          enum: ["b", "f"]
+          name: dir
+          description: |-
+            The direction to return events from.
+          required: true
+          x-example: "b"
+        - in: query
+          type: integer
+          name: limit
+          description: |-
+            The maximum number of events to return. Default: 10.
+          x-example: "3"
+      responses:
+        200:
+          description: A list of messages with a new token to request more.
+          schema:
+            type: object
+            description: A list of messages with a new token to request more.
+            properties:
+              start:
+                type: string
+                description: |-
+                  The token to start paginating from. If ``dir=b`` this will be
+                  the token supplied in ``from``.
+              end:
+                type: string
+                description: |-
+                  The token the pagination ends at. If ``dir=b`` this token should
+                  be used again to request even earlier events.
+              chunk:
+                type: array
+                description: |-
+                  A list of room events.
+                items:
+                  type: object
+                  title: RoomEvent
+          examples:
+            application/json: |-
+              {
+                "start": "t47429-4392820_219380_26003_2265",
+                "end": "t47409-4357353_219380_26003_2265",
+                "chunk": [
+                  {
+                    "origin_server_ts": 1444812213737,
+                    "user_id": "@alice:example.com",
+                    "event_id": "$1444812213350496Caaaa:example.com",
+                    "content": {
+                      "body": "hello world",
+                      "msgtype":"m.text"
+                    },
+                    "room_id":"!Xq3620DUiqCaoxq:example.com",
+                    "type":"m.room.message",
+                    "age": 1042
+                  },
+                  {
+                    "origin_server_ts": 1444812194656 ,
+                    "user_id": "@bob:example.com",
+                    "event_id": "$1444812213350496Cbbbb:example.com",
+                    "content": {
+                      "body": "the world is big",
+                      "msgtype":"m.text"
+                    },
+                    "room_id":"!Xq3620DUiqCaoxq:example.com",
+                    "type":"m.room.message",
+                    "age": 20123
+                  },
+                  {
+                    "origin_server_ts": 1444812163990,
+                    "user_id": "@bob:example.com",
+                    "event_id": "$1444812213350496Ccccc:example.com",
+                    "content": {
+                      "name": "New room name"
+                    },
+                    "prev_content": {
+                      "name": "Old room name"
+                    },
+                    "state_key": "",
+                    "room_id":"!Xq3620DUiqCaoxq:example.com",
+                    "type":"m.room.name",
+                    "age": 50789
+                  }
+                ]
+              }
+        403:
+          description: >
+            You aren't a member of the room.
\ No newline at end of file

From 6f6861a11d45b86c5cbcad7b864fd527a77f5724 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 13:17:51 +0100
Subject: [PATCH 306/467] Swaggerify the createRoom API

---
 api/client-server/v1/create_room.yaml | 148 ++++++++++++++++++++++++++
 specification/1-client_server_api.rst | 135 ++---------------------
 2 files changed, 155 insertions(+), 128 deletions(-)
 create mode 100644 api/client-server/v1/create_room.yaml

diff --git a/api/client-server/v1/create_room.yaml b/api/client-server/v1/create_room.yaml
new file mode 100644
index 00000000..5bb4e17c
--- /dev/null
+++ b/api/client-server/v1/create_room.yaml
@@ -0,0 +1,148 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Room Creation API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/createRoom":
+    post:
+      summary: Create a new room
+      description: |-
+        Create a new room with various configuration options.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: body
+          description: The desired room configuration.
+          schema:
+            type: object
+            example: |-
+              {
+                "preset": "public_chat",
+                "room_alias_name": "thepub",
+                "name": "The Grand Duke Pub",
+                "topic": "All about happy hour",
+                "creation_content": {
+                    "m.federate": false
+                }
+              }
+            properties:
+              visibility:
+                type: string
+                enum: ["public", "private"]
+                description: |-
+                  A ``public`` visibility indicates that the room will be shown
+                  in the published room list. A ``private`` visibility will hide
+                  the room from the published room list. Rooms default to
+                  ``private`` visibility if this key is not included. NB: This
+                  should not be confused with ``join_rules`` which also uses the
+                  word ``public``.
+              room_alias_name:
+                type: string
+                description: |-
+                  The desired room alias **local part**. If this is included, a
+                  room alias will be created and mapped to the newly created
+                  room. The alias will belong on the *same* home server which
+                  created the room. For example, if this was set to "foo" and
+                  sent to the homeserver "example.com" the complete room alias
+                  would be ``#foo:example.com``.
+              name:
+                type: string
+                description: |-
+                  If this is included, an ``m.room.name`` event will be sent
+                  into the room to indicate the name of the room. See Room
+                  Events for more information on ``m.room.name``.
+              topic:
+                type: string
+                description: |-
+                  If this is included, an ``m.room.topic`` event will be sent
+                  into the room to indicate the topic for the room. See Room
+                  Events for more information on ``m.room.topic``.
+              invite:
+                type: array
+                description: |-
+                  A list of user IDs to invite to the room. This will tell the
+                  server to invite everyone in the list to the newly created room.
+                items:
+                  type: string
+              creation_content:
+                title: CreationContent
+                type: object
+                description: |-
+                  Extra keys to be added to the content of the ``m.room.create``.
+                  The server will clober the following keys: ``creator``. Future
+                  versions of the specification may allow the server to clobber
+                  other keys.
+              initial_state:
+                type: array
+                description: |-
+                  A list of state events to set in the new room. This allows
+                  the user to override the default state events set in the new
+                  room. The expected format of the state events are an object
+                  with type, state_key and content keys set.
+                  Takes precedence over events set by ``presets``, but gets
+                  overriden by ``name`` and ``topic`` keys.
+                items:
+                  type: object
+                  title: StateEvent
+                  properties:
+                    type:
+                      type: string
+                    state_key:
+                      type: string
+                    content:
+                      type: string
+              preset:
+                type: string
+                enum: ["private_chat", "public_chat", "trusted_private_chat"]
+                description: |-
+                  Convenience parameter for setting various default state events
+                  based on a preset. Must be either:
+
+                  ``private_chat`` =>
+                    ``join_rules`` is set to ``invite``.
+                    ``history_visibility`` is set to ``shared``.
+
+                  ``trusted_private_chat`` =>
+                      ``join_rules`` is set to ``invite``.
+                      ``history_visibility`` is set to ``shared``.
+                      All invitees are given the same power level as the room creator.
+
+                  ``public_chat``: =>
+                      ``join_rules`` is set to ``public``.
+                      ``history_visibility`` is set to ``shared``.
+
+      responses:
+        200:
+          description: Information about the newly created room.
+          schema:
+            type: object
+            description: Information about the newly created room.
+            properties:
+              room_id:
+                type: string
+                description: |-
+                  The created room's ID.
+          examples:
+            application/json: |-
+              {
+                "room_id": "!sefiuhWgwghwWgh:example.com"
+              }
+        400:
+          description: >
+            The request body is malformed or the room alias specified is already taken.
\ No newline at end of file
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index d8b9299a..f27e4d42 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -665,140 +665,19 @@ Rooms
 
 Creation
 ~~~~~~~~
-To create a room, a client has to use the |createRoom|_ API. There are various
-options which can be set when creating a room:
-
-``visibility``
-  Type:
-    String
-  Optional:
-    Yes
-  Value:
-    Either ``public`` or ``private``.
-  Description:
-    A ``public`` visibility indicates that the room will be shown in the public
-    room list. A ``private`` visibility will hide the room from the public room
-    list. Rooms default to ``private`` visibility if this key is not included.
-
-``room_alias_name``
-  Type:
-    String
-  Optional:
-    Yes
-  Value:
-    The room alias localpart.
-  Description:
-    If this is included, a room alias will be created and mapped to the newly
-    created room.  The alias will belong on the same home server which created
-    the room, e.g.  ``!qadnasoi:domain.com >>> #room_alias_name:domain.com``
-
-``name``
-  Type:
-    String
-  Optional:
-    Yes
-  Value:
-    The ``name`` value for the ``m.room.name`` state event.
-  Description:
-    If this is included, an ``m.room.name`` event will be sent into the room to
-    indicate the name of the room. See `Room Events`_ for more information on
-    ``m.room.name``.
-
-``topic``
-  Type:
-    String
-  Optional:
-    Yes
-  Value:
-    The ``topic`` value for the ``m.room.topic`` state event.
-  Description:
-    If this is included, an ``m.room.topic`` event will be sent into the room
-    to indicate the topic for the room. See `Room Events`_ for more information
-    on ``m.room.topic``.
-
-``invite``
-  Type:
-    List
-  Optional:
-    Yes
-  Value:
-    A list of user ids to invite.
-  Description:
-    This will tell the server to invite everyone in the list to the newly
-    created room.
-
-``creation_content``
-  Type:
-    Object
-  Optional:
-    Yes
-  Value:
-    Extra keys to be added to the content of the ``m.room.create``. The server
-    will clober the following keys: ``creator``. Future versions of this
-    spec may allow the server to clobber other keys if required.
-  Description:
-    Allows clients to add keys to the content of ``m.room.create``.
-
-``preset``
-  Type:
-    String
-  Optional:
-    Yes
-  Value:
-    ``private_chat``, ``trusted_private_chat`` or ``public_chat``
-  Description:
-    Convenience parameter for setting various default state events based on a
-    preset.
-
-    Three presets are defined:
-
-    - ``private_chat``: Sets the ``join_rules`` to ``invite`` and
-      ``history_visibility`` to ``shared``
-    - ``trusted_private_chat``: Set the ``join_rules`` to ``invite``,
-      ``history_visibility`` to ``shared`` and gives all invitees the same
-      power level as the creator.
-    - ``public_chat``: Sets the ``join_rules`` to ``public`` and
-      ``history_visibility`` to ``shared``
-
-``initial_state``
-  Type:
-    List
-  Optional:
-    Yes
-  Value:
-    A list of state events to set in the new room.
-  Description:
-    Allows the user to override the default state events set in the new room.
-
-    The expected format of the state events are an object with ``type``,
-    ``state_key`` and ``content`` keys set.
-
-    Takes precedence over events set by ``presets``, but gets overriden by
-    ``name`` and ``topic`` keys.
-
-Example::
-
-  {
-    "preset": "public_chat",
-    "room_alias_name": "thepub",
-    "name": "The Grand Duke Pub",
-    "topic": "All about happy hour",
-    "creation_content": {
-        "m.federate": false
-    }
-  }
-
-The home server will create a ``m.room.create`` event when the room is created,
-which serves as the root of the PDU graph for this room. This event also has a
+The home server will create a ``m.room.create`` event when a room is created,
+which serves as the root of the event graph for this room. This event also has a
 ``creator`` key which contains the user ID of the room creator. It will also
 generate several other events in order to manage permissions in this room. This
 includes:
 
- - ``m.room.power_levels`` : Sets the power levels of users and required power
-    levels.
+ - ``m.room.power_levels`` : Sets the power levels of users and required power levels.
  - ``m.room.join_rules`` : Whether the room is "invite-only" or not.
 
-See `Room Events`_ for more information on these events.
+See `Room Events`_ for more information on these events. To create a room, a
+client has to use the the following API.
+
+{{create_room_http_api}}
 
 Room aliases
 ~~~~~~~~~~~~

From 34bd8edec5395a8accde330be4b04dbca41f770b Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 13:26:58 +0100
Subject: [PATCH 307/467] Move related auth sections together

---
 specification/0-events.rst            |   4 +-
 specification/1-client_server_api.rst | 274 +++++++++++++-------------
 2 files changed, 138 insertions(+), 140 deletions(-)

diff --git a/specification/0-events.rst b/specification/0-events.rst
index 16948462..64007568 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -1,5 +1,5 @@
-Events
-======
+Event Structure
+===============
 
 All communication in Matrix is expressed in the form of data objects called
 Events. These are the fundamental building blocks common to the client-server,
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index f27e4d42..319fed91 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -340,6 +340,141 @@ This MUST return an HTML page which can perform this authentication stage. This
 page must attempt to call the JavaScript function ``window.onAuthDone`` when
 the authentication has been completed.
 
+Registration
+~~~~~~~~~~~~
+This section refers to API Version 2. These API calls currently use the prefix
+``/_matrix/client/v2_alpha``.
+
+Registering for a user account is done using the request::
+
+  POST $V2PREFIX/register
+
+This API endpoint uses the User-Interactive Authentication API.
+This API endpoint does not require an access token.
+
+The body of the POST request is a JSON object containing:
+
+username
+  Optional. This is the local part of the desired Matrix ID. If omitted, the
+  Home Server must generate a Matrix ID local part.
+password
+  Required. The desired password for the account.
+bind_email
+  Optional. If ``true``, the server binds the email used for authentication to
+  the Matrix ID with the ID Server.
+
+On success, this returns a JSON object with keys:
+
+user_id
+  The fully-qualified Matrix ID that has been registered.
+access_token
+  An access token for the new account.
+home_server
+  The hostname of the Home Server on which the account has been registered.
+
+This endpoint may also return the following error codes:
+
+M_USER_IN_USE
+  If the Matrix ID is already in use
+M_EXCLUSIVE
+  If the requested Matrix ID is in the exclusive namespace of an application
+  service.
+
+Home Servers MUST perform the relevant checks and return these codes before
+performing User-Interactive Authentication, although they may also return
+them after authentication is completed if, for example, the requested user ID
+was registered whilst the client was performing authentication.
+
+Old V1 API docs: |register|_
+
+{{login_http_api}}
+
+Changing Password
++++++++++++++++++
+This section refers to API Version 2. These API calls currently use the prefix
+``/_matrix/client/v2_alpha``.
+
+Request::
+
+  POST $V2PREFIX/account/password
+
+This API endpoint uses the User-Interactive Authentication API. An access token
+should be submitted to this endpoint if the client has an active session. The
+Home Server may change the flows available depending on whether a valid access
+token is provided.
+
+The body of the POST request is a JSON object containing:
+
+new_password
+  The new password for the account.
+
+On success, an empty JSON object is returned.
+
+The error code M_NOT_FOUND is returned if the user authenticated with a third
+party identifier but the Home Server could not find a matching account in its
+database.
+
+Adding a Third Party Identifier
++++++++++++++++++++++++++++++++
+This section refers to API Version 2. These API calls currently use the prefix
+``/_matrix/client/v2_alpha``.
+
+Request::
+
+  POST $V2PREFIX/account/3pid
+
+Used to add a third party identifier to the user's account.
+
+The body of the POST request is a JSON object containing:
+
+threePidCreds
+  An object containing third party identifier credentials.
+bind
+  Optional. A boolean indicating whether the Home Server should also bind this
+  third party identifier to the account's matrix ID with the Identity Server. If
+  supplied and true, the Home Server must bind the 3pid accordingly.
+
+The third party identifier credentials object comprises:
+
+id_server
+  The colon-separated hostname and port of the Identity Server used to
+  authenticate the third party identifier. If the port is the default, it and the
+  colon should be omitted.
+sid
+  The session ID given by the Identity Server
+client_secret
+  The client secret used in the session with the Identity Server.
+
+On success, the empty JSON object is returned.
+
+May also return error codes:
+
+M_THREEPID_AUTH_FAILED
+  If the credentials provided could not be verified with the ID Server.
+
+Fetching Currently Associated Third Party Identifiers
++++++++++++++++++++++++++++++++++++++++++++++++++++++
+This section refers to API Version 2. These API calls currently use the prefix
+``/_matrix/client/v2_alpha``.
+
+Request::
+
+  GET $V2PREFIX/account/3pid
+
+This returns a list of third party identifiers that the Home Server has
+associated with the user's account. This is *not* the same as the list of third
+party identifiers bound to the user's Matrix ID in Identity Servers. Identifiers
+in this list may be used by the Home Server as, for example, identifiers that it
+will accept to reset the user's account password.
+
+Returns a JSON object with the key ``threepids`` whose contents is an array of
+objects with the following keys:
+
+medium
+  The medium of the 3pid (eg, ``email``)
+address
+  The textual address of the 3pid, eg. the email address
+
 Events
 ------
 
@@ -742,7 +877,7 @@ certain operations such as kicking, banning and sending state events. See
 `m.room.power_levels`_ for more information.
 
 Joining rooms
--------------
+~~~~~~~~~~~~~
 Users need to be a member of a room in order to send and receive events in that
 room. There are several states in which a user may be, in relation to a room:
 
@@ -821,143 +956,6 @@ member's state, by making a request to
     "membership": "ban"
   }
 
-
-Registration
-------------
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Registering for a user account is done using the request::
-
-  POST $V2PREFIX/register
-
-This API endpoint uses the User-Interactive Authentication API.
-This API endpoint does not require an access token.
-
-The body of the POST request is a JSON object containing:
-
-username
-  Optional. This is the local part of the desired Matrix ID. If omitted, the
-  Home Server must generate a Matrix ID local part.
-password
-  Required. The desired password for the account.
-bind_email
-  Optional. If ``true``, the server binds the email used for authentication to
-  the Matrix ID with the ID Server.
-
-On success, this returns a JSON object with keys:
-
-user_id
-  The fully-qualified Matrix ID that has been registered.
-access_token
-  An access token for the new account.
-home_server
-  The hostname of the Home Server on which the account has been registered.
-
-This endpoint may also return the following error codes:
-
-M_USER_IN_USE
-  If the Matrix ID is already in use
-M_EXCLUSIVE
-  If the requested Matrix ID is in the exclusive namespace of an application
-  service.
-
-Home Servers MUST perform the relevant checks and return these codes before
-performing User-Interactive Authentication, although they may also return
-them after authentication is completed if, for example, the requested user ID
-was registered whilst the client was performing authentication.
-
-Old V1 API docs: |register|_
-
-{{login_http_api}}
-
-Changing Password
-~~~~~~~~~~~~~~~~~
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Request::
-
-  POST $V2PREFIX/account/password
-
-This API endpoint uses the User-Interactive Authentication API. An access token
-should be submitted to this endpoint if the client has an active session. The
-Home Server may change the flows available depending on whether a valid access
-token is provided.
-
-The body of the POST request is a JSON object containing:
-
-new_password
-  The new password for the account.
-
-On success, an empty JSON object is returned.
-
-The error code M_NOT_FOUND is returned if the user authenticated with a third
-party identifier but the Home Server could not find a matching account in its
-database.
-
-Adding a Third Party Identifier
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Request::
-
-  POST $V2PREFIX/account/3pid
-
-Used to add a third party identifier to the user's account.
-
-The body of the POST request is a JSON object containing:
-
-threePidCreds
-  An object containing third party identifier credentials.
-bind
-  Optional. A boolean indicating whether the Home Server should also bind this
-  third party identifier to the account's matrix ID with the Identity Server. If
-  supplied and true, the Home Server must bind the 3pid accordingly.
-
-The third party identifier credentials object comprises:
-
-id_server
-  The colon-separated hostname and port of the Identity Server used to
-  authenticate the third party identifier. If the port is the default, it and the
-  colon should be omitted.
-sid
-  The session ID given by the Identity Server
-client_secret
-  The client secret used in the session with the Identity Server.
-
-On success, the empty JSON object is returned.
-
-May also return error codes:
-
-M_THREEPID_AUTH_FAILED
-  If the credentials provided could not be verified with the ID Server.
-
-Fetching Currently Associated Third Party Identifiers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Request::
-
-  GET $V2PREFIX/account/3pid
-
-This returns a list of third party identifiers that the Home Server has
-associated with the user's account. This is *not* the same as the list of third
-party identifiers bound to the user's Matrix ID in Identity Servers. Identifiers
-in this list may be used by the Home Server as, for example, identifiers that it
-will accept to reset the user's account password.
-
-Returns a JSON object with the key ``threepids`` whose contents is an array of
-objects with the following keys:
-
-medium
-  The medium of the 3pid (eg, ``email``)
-address
-  The textual address of the 3pid, eg. the email address
-
-
 Profiles
 --------
 

From 4a9db39277a6f61bb4efc27bb6a14815a89acd05 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Wed, 14 Oct 2015 13:38:28 +0100
Subject: [PATCH 308/467] Clean up some untruths in the login api doc

add "type", and "username" -> "user"
---
 api/client-server/v1/login.yaml | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml
index 3d415c29..bbab46df 100644
--- a/api/client-server/v1/login.yaml
+++ b/api/client-server/v1/login.yaml
@@ -33,17 +33,21 @@ paths:
             type: object
             example: |-
               {
-                "username": "cheeky_monkey",
+                "type": "m.login.pasword",
+                "user": "cheeky_monkey",
                 "password": "ilovebananas"
               }
             properties:
-              username:
+              type:
+                type: string
+                description: The login type being used. Currently only "m.login.password" is supported.
+              user:
                 type: string
                 description: The fully qualified user ID or just local part of the user ID, to log in.
               password:
                 type: string
                 description: The user's password.
-            required: ["username", "password"]
+            required: ["type", "user", "password"]
       responses:
         200:
           description: The user has been authenticated.
@@ -78,6 +82,15 @@ paths:
               home_server:
                 type: string
                 description: The hostname of the Home Server on which the account has been registered.
+        400:
+          description: |-
+            Part of the request was invalid. For example, the login type may not be recognised.
+          examples:
+            application/json: |-
+              {
+                  "errcode": "M_UNKNOWN",
+                  "error": "Bad login type."
+              }
         403:
           description: |-
             The login attempt failed. For example, the password may have been incorrect.

From e42c8b5b63a637eae7863285f342938fad6adc18 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 14:19:11 +0100
Subject: [PATCH 309/467] Add search API

---
 api/client-server/v1/search.yaml | 126 +++++++++++++++++++++++++++++++
 1 file changed, 126 insertions(+)
 create mode 100644 api/client-server/v1/search.yaml

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
new file mode 100644
index 00000000..c9e9b14b
--- /dev/null
+++ b/api/client-server/v1/search.yaml
@@ -0,0 +1,126 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Search API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/search":
+    post:
+      summary: Search server side for things.
+      description: |-
+        Performs a full text search across different categories.
+      security:
+        - accessToken: []
+      parameters:
+        - in: body
+          name: body
+          schema:
+            type: object
+            example: |-
+              {
+                "search_categories": {
+                  "room_events": {
+                    "keys": [
+                      "content.body"
+                    ],
+                    "search_term": "martians and men"
+                  }
+                }
+              }
+            properties:
+              search_categories:
+                type: object
+                description: Describes which categories to search in and
+                  their creteria.
+                properties:
+                  room_events:
+                    type: object
+                    description: Mapping of category name to search criteria.
+                    properties:
+                      search_term:
+                        type: string
+                        description: The string to search events for
+                      keys:
+                        type: array
+                        items:
+                          type: string
+                          enum: ["content.body", "content.name", "content.topic"]
+                        description: The keys to search. Defaults to all.
+                    required: ["search_term"]
+            required: ["search_categories"]
+      responses:
+        200:
+          description: Results of the search.
+          schema:
+            type: object
+            required: ["search_categories"]
+            properties:
+              search_categories:
+                type: object
+                description: Describes which categories to search in and
+                  their creteria.
+                properties:
+                  room_events:
+                    type: object
+                    description: Mapping of category name to search criteria.
+                    properties:
+                      count:
+                        type: number
+                        description: Total number of results found
+                      results:
+                        type: object
+                        description: Mapping of event_id to result.
+                        additionalProperties:
+                          type: object
+                          desciption: The result object.
+                          properties:
+                            rank:
+                              type: number
+                              description: A number that describes how closely
+                                this result matches the search. Higher is
+                                closer.
+                            result:
+                                type: object
+                                description: The event that matched.
+                                allOf:
+                                  - "$ref": "core-event-schema/room_event.json"
+          examples:
+            application/json: |-
+               {
+                  "search_categories": {
+                    "room_events": {
+                      "count": 24,
+                      "results": {
+                        "$144429830826TWwbB:localhost": {
+                          "rank": 0.00424866,
+                          "result": {
+                            "age": 526228296,
+                            "content": {
+                              "body": "Test content",
+                              "msgtype": "m.text"
+                            },
+                            "event_id": "$144429830826TWwbB:localhost",
+                            "origin_server_ts": 1444298308034,
+                            "room_id": "!qPewotXpIctQySfjSy:localhost",
+                            "type": "m.room.message",
+                            "user_id": "@test:localhost"
+                          }
+                        }
+                      }
+                    }
+                  }
+                }

From 270e1d7aa81b8692dbffe4604df81c96de99d6cb Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Wed, 14 Oct 2015 14:29:36 +0100
Subject: [PATCH 310/467] Add richvdh to list of trusted pushers for
 spectulator

richvdh is awesome, he should have all the permissions.
---
 scripts/speculator/main.go | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index d641d7da..dab624df 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -364,6 +364,7 @@ func main() {
 		"illicitonion":  true,
 		"Kegsay":        true,
 		"NegativeMjark": true,
+		"richvdh":       true,
 	}
 	rand.Seed(time.Now().Unix())
 	masterCloneDir, err := gitClone(matrixDocCloneURL, false)

From a7a5cb088b8b84d93d46f83b10fdf5d222c8987c Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 14:35:08 +0100
Subject: [PATCH 311/467] Add titles

---
 api/client-server/v1/search.yaml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index c9e9b14b..fa6ab2a6 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -67,15 +67,18 @@ paths:
           description: Results of the search.
           schema:
             type: object
+            title: Results
             required: ["search_categories"]
             properties:
               search_categories:
                 type: object
+                title: Categories
                 description: Describes which categories to search in and
                   their creteria.
                 properties:
                   room_events:
                     type: object
+                    title: Room Event Results
                     description: Mapping of category name to search criteria.
                     properties:
                       count:
@@ -83,9 +86,11 @@ paths:
                         description: Total number of results found
                       results:
                         type: object
+                        title: Results
                         description: Mapping of event_id to result.
                         additionalProperties:
                           type: object
+                          title: Result
                           desciption: The result object.
                           properties:
                             rank:

From 013b6db8958b7b02000044ee6829ccbdbd24a08f Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Wed, 14 Oct 2015 14:42:52 +0100
Subject: [PATCH 312/467] Add a note on building the speculator

... because the wiki says you need to read the README on how to build it.
---
 scripts/speculator/README | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/speculator/README b/scripts/speculator/README
index 0a9f53fd..c54e0f2b 100644
--- a/scripts/speculator/README
+++ b/scripts/speculator/README
@@ -8,3 +8,7 @@ It serves the following HTTP endpoints:
 
 To run it, you must install the `go` tool, and run:
  `go run main.go`
+
+To build the binary (which is necessary for deployment to the matrix.org
+servers), you must again install `go`, and then run:
+ `go build`

From 3608f3a282d9d9eae775dafb30eb22b813d319a6 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 14:44:09 +0100
Subject: [PATCH 313/467] SPEC-144 : Use https examples not http.

---
 specification/modules/push.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index 6e9c8536..525e730d 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -353,14 +353,14 @@ Examples
 To create a rule that suppresses notifications for the room with ID
 ``!dj234r78wl45Gh4D:matrix.org``::
 
-  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \
+  curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \
   '{
      "actions" : ["dont_notify"]
    }'
 
 To suppress notifications for the user ``@spambot:matrix.org``::
 
-  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \
+  curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \
   '{
      "actions" : ["dont_notify"]
    }'
@@ -368,7 +368,7 @@ To suppress notifications for the user ``@spambot:matrix.org``::
 To always notify for messages that contain the work 'cake' and set a specific
 sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``)::
 
-  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \
+  curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \
   '{
      "pattern": "cake",
      "actions" : ["notify", {"set_sound":"cakealarm.wav"}]
@@ -377,7 +377,7 @@ sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``)::
 To add a rule suppressing notifications for messages starting with 'cake' but
 ending with 'lie', superseding the previous rule::
 
-  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \
+  curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \
   '{
      "pattern": "cake*lie",
      "actions" : ["notify"]
@@ -387,7 +387,7 @@ To add a custom sound for notifications messages containing the word 'beer' in
 any rooms with 10 members or fewer (with greater importance than the room,
 sender and content rules)::
 
-  curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \
+  curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \
   '{
      "conditions": [
        {"kind": "event_match", "key": "content.body", "pattern": "beer" },

From ca9e44baaca52882fc9b661edf1ee71428f3713c Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 15:01:21 +0100
Subject: [PATCH 314/467] Moar titles

---
 api/client-server/v1/search.yaml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index fa6ab2a6..e62f2e8e 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -44,11 +44,13 @@ paths:
             properties:
               search_categories:
                 type: object
+                title: "Categories"
                 description: Describes which categories to search in and
                   their creteria.
                 properties:
                   room_events:
                     type: object
+                    title: "Room Events"
                     description: Mapping of category name to search criteria.
                     properties:
                       search_term:

From 392a1c5ad821941ccedac27052b7022a62fd2b2c Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 15:06:16 +0100
Subject: [PATCH 315/467] Another title

---
 api/client-server/v1/search.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index e62f2e8e..3efef9f5 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -102,6 +102,7 @@ paths:
                                 closer.
                             result:
                                 type: object
+                                title: Event
                                 description: The event that matched.
                                 allOf:
                                   - "$ref": "core-event-schema/room_event.json"

From cd34d4e19b334c3a8165bb48deaa5adc7085fecd Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 15:12:15 +0100
Subject: [PATCH 316/467] SPEC-182: Make it clearer how ASes are supposed to
 create users by linkifying the right text.

---
 specification/0-feature_profiles.rst        | 2 +-
 specification/3-application_service_api.rst | 8 +++++++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index b9f12b74..6c11dd2f 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -1,7 +1,7 @@
 Feature Profiles
 ================
 
-.. sect:feature-profiles:
+.. _sect:feature-profiles:
 
 Matrix supports many different kinds of clients: from embedded IoT devices to
 desktop clients. Not all clients can provide the same feature sets as other
diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index a6e82137..d1799158 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -94,7 +94,7 @@ API called when:
  - HS receives an event for an unknown user ID in the AS's namespace, e.g. an
    invite event to a room.
 Notes:
- - When the AS receives this request, if the user exists, it must create the user via
+ - When the AS receives this request, if the user exists, it must `create the user`_ via
    the CS API.
  - It can also set arbitrary information about the user (e.g. display name, join rooms, etc)
    using the CS API.
@@ -226,6 +226,9 @@ Ordering notes:
     ]
   }
 
+
+.. _create the user: `sect:asapi-permissions`_
+
 Client-Server v2 API Extensions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -291,6 +294,9 @@ Notes:
 
 Server admin style permissions
 ++++++++++++++++++++++++++++++
+
+.. _sect:asapi-permissions:
+
 The home server needs to give the application service *full control* over its
 namespace, both for users and for room aliases. This means that the AS should
 be able to create/edit/delete any room alias in its namespace, as well as

From 45b1d08c7e338586d1a716e447e3f025a960aaa6 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 15:16:12 +0100
Subject: [PATCH 317/467] SPEC-207: Add a warning about needing AS tokens to be
 unique and why

---
 specification/3-application_service_api.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index a6e82137..557e16aa 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -54,6 +54,11 @@ An example HS configuration required to pass traffic to the AS is:
       aliases: []  # Namespaces of room aliases which should be delegated to the AS
       rooms: [] # Namespaces of room ids which should be delegated to the AS
 
+.. WARNING::
+  If the homeserver in question has multiple application services, each
+  ``as_token`` MUST be unique per application service as this token is used to
+  identify the application service.
+
 - An application service can state whether they should be the only ones who 
   can manage a specified namespace. This is referred to as an "exclusive" 
   namespace. An exclusive namespace prevents humans and other application 

From 44b19a8b0e16a25950899c4f5d90b2ea99b4de69 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 15:27:56 +0100
Subject: [PATCH 318/467] SPEC-165: Outline directory traversal attacks on MXC
 URIs

---
 specification/modules/content_repo.rst | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index 9ac5e199..52937ead 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -52,7 +52,17 @@ The HTTP GET endpoint does not require any authentication. Knowing the URL of
 the content is sufficient to retrieve the content, even if the entity isn't in
 the room.
 
-Homeservers have additional concerns:
+MXC URIs are vulnerable to directory traversal attacks such as
+``mxc://127.0.0.1/../../../some_service/etc/passwd``. This would cause the target
+homeserver to try to access and return this file. As such, homeservers MUST
+sanitise MXC URIs by allowing only alphanumeric (``A-Za-z0-9``), ``_``
+and  ``-`` characters in the ``server-name`` and ``media-id`` values. This set
+of whitelisted characters allows URL-safe base64 encodings specified in RFC 4648.
+Applying this character whitelist is preferable to blacklisting ``.`` and ``/``
+as there are techniques around blacklisted characters (percent-encoded characters,
+UTF-8 encoded traversals, etc).
+
+Homeservers have additional content-specific concerns:
 
  - Clients may try to upload very large files. Homeservers should not store files
    that are too large and should not serve them to clients.

From 2b9484cf48249a4c34e1487ddb11863ad1f00bdf Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 15:35:39 +0100
Subject: [PATCH 319/467] Spell things correctly

---
 api/client-server/v1/search.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index 3efef9f5..a40d49c0 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -93,7 +93,7 @@ paths:
                         additionalProperties:
                           type: object
                           title: Result
-                          desciption: The result object.
+                          description: The result object.
                           properties:
                             rank:
                               type: number

From f0d80529517eddac9432daab9984c38784528a76 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 15:42:44 +0100
Subject: [PATCH 320/467] SPEC-205: Warn about password strengths in
 m.login.password section

---
 specification/1-client_server_api.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 59e6b68e..a63c114a 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -215,6 +215,12 @@ To respond to this type, reply with an auth dict as follows::
     "password": "<password>"
   }
 
+.. WARNING::
+  Clients SHOULD enforce that the password provided is suitably complex. The
+  password SHOULD include a lower-case letter, an upper-case letter, a number
+  and a symbol and be at a minimum 8 characters in length. Servers MAY reject
+  weak passwords with an error code ``M_WEAK_PASSWORD``.
+
 Google ReCaptcha
 ~~~~~~~~~~~~~~~~
 :Type:

From d8bc0c9315f1dcddb4a819d2249bdae7ee0c494a Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 16:13:43 +0100
Subject: [PATCH 321/467] Add search module

---
 specification/modules/search.rst | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 specification/modules/search.rst

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
new file mode 100644
index 00000000..c571d68d
--- /dev/null
+++ b/specification/modules/search.rst
@@ -0,0 +1,14 @@
+Server Side Search
+==================
+
+.. _module:search:
+
+TODO: Add summary.
+
+Client behaviour
+----------------
+{{search_http_api}}
+
+Security considerations
+-----------------------
+The server must only return results that the user has permission to see.

From 6a0595bc46c3751090d5d24e2ab94647c9573915 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 16:15:55 +0100
Subject: [PATCH 322/467] Add in size limits as per SPEC-222

---
 .../v1/core-event-schema/state_event.json       |  2 +-
 event-schemas/schema/v1/m.room.name             |  2 +-
 specification/0-events.rst                      | 17 +++++++++++++++++
 specification/0-intro.rst                       |  2 +-
 4 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/event-schemas/schema/v1/core-event-schema/state_event.json b/event-schemas/schema/v1/core-event-schema/state_event.json
index f70118f4..88b4900a 100644
--- a/event-schemas/schema/v1/core-event-schema/state_event.json
+++ b/event-schemas/schema/v1/core-event-schema/state_event.json
@@ -8,7 +8,7 @@
   "properties": {
     "state_key": {
       "type": "string",
-      "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event."
+      "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event. The key MUST NOT start with '_'."
     },
     "prev_content": {
       "type": "object",
diff --git a/event-schemas/schema/v1/m.room.name b/event-schemas/schema/v1/m.room.name
index b43f02cc..0e0b25be 100644
--- a/event-schemas/schema/v1/m.room.name
+++ b/event-schemas/schema/v1/m.room.name
@@ -11,7 +11,7 @@
             "properties": {
                 "name": {
                     "type": "string",
-                    "description": "The name of the room."
+                    "description": "The name of the room. This MUST NOT exceed 255 bytes."
                 }
             },
             "required": ["name"]
diff --git a/specification/0-events.rst b/specification/0-events.rst
index 16948462..ae5a698a 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -12,6 +12,23 @@ server-server and application-service APIs, and are described below.
 {{common_state_event_fields}}
 
 
+Size limits
+-----------
+
+The total size of any event MUST NOT exceed 65 KB. There are additional
+restrictions on sizes per key:
+
+ - ``user_id`` MUST NOT exceed 255 bytes (including domain).
+ - ``room_id`` MUST NOT exceed 255 bytes.
+ - ``state_key`` MUST NOT exceed 255 bytes.
+ - ``type`` MUST NOT exceed 255 bytes.
+ - ``event_id`` MUST NOT exceed 255 bytes.
+ - ``user_id`` MUST NOT exceed 255 bytes.
+
+Some event types have additional sizes restrictions which are specified in
+the description of the event. Additional keys have no limit other than that
+implied by the total 65 KB limit on events.
+
 Room Events
 -----------
 .. NOTE::
diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index 445b32ef..e5cf1ac8 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -296,7 +296,7 @@ be obtained by visiting the domain specified. They are case-insensitive. Note
 that the mapping from a room alias to a room ID is not fixed, and may change
 over time to point to a different room ID. For this reason, Clients SHOULD
 resolve the room alias to a room ID once and then use that ID on subsequent
-requests.
+requests. Room aliases MUST NOT exceed 255 bytes.
 
 When resolving a room alias the server will also respond with a list of servers
 that are in the room that can be used to join via.

From 0efa3fd98128d9e7657a884cb59d5dc987de2245 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 16:17:04 +0100
Subject: [PATCH 323/467] Add module

---
 specification/0-feature_profiles.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index b9f12b74..ec234a0e 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -26,6 +26,7 @@ Summary
  `Content Repository`_                 Required   Required   Required   Optional   Optional
  `Managing History Visibility`_        Required   Required   Required   Required   Optional
  `End-to-End Encryption`_              Optional   Optional   Optional   Optional   Optional
+ `Search`_                             Optional   Optional   Optional   Optional   Optional
 ===================================== ========== ========== ========== ========== ==========
 
 *Please see each module for more details on what clients need to implement.*
@@ -39,6 +40,7 @@ Summary
 .. _VoIP: `module:voip`_
 .. _Content Repository: `module:content`_
 .. _Managing History Visibility: `module:history-visibility`_
+.. _Search: `module:search`_
 
 Clients
 -------

From 38fcc563dac266b782c296363d3e62f75fe3767b Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 16:20:08 +0100
Subject: [PATCH 324/467] Spell out full title

---
 specification/0-feature_profiles.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/0-feature_profiles.rst b/specification/0-feature_profiles.rst
index ec234a0e..128dc9b4 100644
--- a/specification/0-feature_profiles.rst
+++ b/specification/0-feature_profiles.rst
@@ -26,7 +26,7 @@ Summary
  `Content Repository`_                 Required   Required   Required   Optional   Optional
  `Managing History Visibility`_        Required   Required   Required   Required   Optional
  `End-to-End Encryption`_              Optional   Optional   Optional   Optional   Optional
- `Search`_                             Optional   Optional   Optional   Optional   Optional
+ `Server Side Search`_                 Optional   Optional   Optional   Optional   Optional
 ===================================== ========== ========== ========== ========== ==========
 
 *Please see each module for more details on what clients need to implement.*
@@ -40,7 +40,7 @@ Summary
 .. _VoIP: `module:voip`_
 .. _Content Repository: `module:content`_
 .. _Managing History Visibility: `module:history-visibility`_
-.. _Search: `module:search`_
+.. _Server Side Search: `module:search`_
 
 Clients
 -------

From 5e48b0b79a5f7263a54fd27962da4f08616e8fa4 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 16:21:12 +0100
Subject: [PATCH 325/467] Add search to targets

---
 specification/targets.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/specification/targets.yaml b/specification/targets.yaml
index c2b6eeda..1cccf648 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -24,6 +24,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/history_visibility.rst
     - modules/push.rst
     - modules/third_party_invites.rst
+    - modules/search.rst
 
 title_styles: ["=", "-", "~", "+", "^", "`"]
 

From 007bb1a69a6d528acfb0f743ac8599592290c403 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 16:26:14 +0100
Subject: [PATCH 326/467] Add extra new line

---
 specification/modules/search.rst | 1 +
 specification/targets.yaml       | 1 +
 2 files changed, 2 insertions(+)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index c571d68d..05471959 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -12,3 +12,4 @@ Client behaviour
 Security considerations
 -----------------------
 The server must only return results that the user has permission to see.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 1cccf648..b549ca53 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -26,6 +26,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/third_party_invites.rst
     - modules/search.rst
 
+
 title_styles: ["=", "-", "~", "+", "^", "`"]
 
 # The templating system doesn't know the right title style to use when generating

From 70704240dcc16ab8e4c4a982ab5b73a0cc047d44 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 16:29:18 +0100
Subject: [PATCH 327/467] gendoc should just add newlines rather than
 complaining about the lack thereof

---
 scripts/gendoc.py | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 8db604c4..74a6c1da 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -128,11 +128,8 @@ def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles):
                 )
             else:
                 rst = f.read()
-            if rst[-2:] != "\n\n":
-                raise Exception(
-                    ("File %s should end with TWO new-line characters to ensure " +
-                    "file concatenation works correctly.") % (file_info,)
-                )
+
+            rst += "\n\n"
             return rst
     # dicts look like {0: filepath, 1: filepath} where the key is the title level
     elif isinstance(file_info, dict):

From 84af5776d7487473aae3d8679120e8d3992b4d4e Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 17:02:39 +0100
Subject: [PATCH 328/467] Newlines

---
 api/client-server/v1/create_room.yaml        | 2 +-
 api/client-server/v1/message_pagination.yaml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/create_room.yaml b/api/client-server/v1/create_room.yaml
index 5bb4e17c..051c4b84 100644
--- a/api/client-server/v1/create_room.yaml
+++ b/api/client-server/v1/create_room.yaml
@@ -145,4 +145,4 @@ paths:
               }
         400:
           description: >
-            The request body is malformed or the room alias specified is already taken.
\ No newline at end of file
+            The request body is malformed or the room alias specified is already taken.
diff --git a/api/client-server/v1/message_pagination.yaml b/api/client-server/v1/message_pagination.yaml
index b3317c13..d2bc0554 100644
--- a/api/client-server/v1/message_pagination.yaml
+++ b/api/client-server/v1/message_pagination.yaml
@@ -128,4 +128,4 @@ paths:
               }
         403:
           description: >
-            You aren't a member of the room.
\ No newline at end of file
+            You aren't a member of the room.

From 1f2f14dc08a8b4dbed83e37dfc667a0777e4accb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 17:07:25 +0100
Subject: [PATCH 329/467] YAML tweaks

---
 api/client-server/v1/sync.yaml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 398b0773..50af8546 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -31,7 +31,7 @@ paths:
           type: string
           name: from
           description: |-
-            The token to stream from. This token is either from the a previous
+            The token to stream from. This token is either from a previous
             request to this API or from the initial sync API.
           required: false
           x-example: "s3456_9_0"
@@ -45,10 +45,10 @@ paths:
           type: boolean
           name: archived
           description: |-
-            Whether to include rooms that the user has left. If absent then
+            Whether to include rooms that the user has left. If ``false`` then
             only rooms that the user has been invited to or has joined are
-            included. If set to "true" then rooms that the user has left are
-            included as well.
+            included. If set to ``true`` then rooms that the user has left are
+            included as well. By default this is ``false``.
           required: false
           x-example: "true"
       responses:

From 728b5a79512c1f3c35d3c2f54f50f60eeb009d50 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Wed, 14 Oct 2015 17:20:00 +0100
Subject: [PATCH 330/467] Add very short summary

---
 specification/modules/search.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 05471959..d6bfd391 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -3,7 +3,8 @@ Server Side Search
 
 .. _module:search:
 
-TODO: Add summary.
+The search API allows clients to perform full text search across events in
+their rooms without having to download the entire histories.
 
 Client behaviour
 ----------------

From e6eb19c46115f1e324fa3c474341c627b4219cf9 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 14 Oct 2015 17:34:25 +0100
Subject: [PATCH 331/467] Review comments

---
 specification/1-client_server_api.rst | 37 +++++++++++++++++----------
 1 file changed, 24 insertions(+), 13 deletions(-)

diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 319fed91..ecd5a3e2 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -29,6 +29,9 @@ return with a status of 401 and the error code, ``M_MISSING_TOKEN`` or
 
 User-Interactive Authentication API
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. _sect:auth-api:
+
 This section refers to API Version 2.
 
 Some API endpoints such as ``login`` or ``register`` require authentication that
@@ -349,9 +352,11 @@ Registering for a user account is done using the request::
 
   POST $V2PREFIX/register
 
-This API endpoint uses the User-Interactive Authentication API.
+This API endpoint uses the `User-Interactive Authentication API`_.
 This API endpoint does not require an access token.
 
+.. _User-Interactive Authentication API: `sect:auth-api`_
+
 The body of the POST request is a JSON object containing:
 
 username
@@ -371,6 +376,9 @@ access_token
   An access token for the new account.
 home_server
   The hostname of the Home Server on which the account has been registered.
+refresh_token
+  A token that may be exchanged for a new ``access_token`` using the
+  ``/tokenrefresh`` API endpoint.
 
 This endpoint may also return the following error codes:
 
@@ -381,10 +389,12 @@ M_EXCLUSIVE
   service.
 
 Home Servers MUST perform the relevant checks and return these codes before
-performing User-Interactive Authentication, although they may also return
+performing `User-Interactive Authentication`_, although they may also return
 them after authentication is completed if, for example, the requested user ID
 was registered whilst the client was performing authentication.
 
+.. _User-Interactive Authentication: `sect:auth-api`_
+
 Old V1 API docs: |register|_
 
 {{login_http_api}}
@@ -414,8 +424,8 @@ The error code M_NOT_FOUND is returned if the user authenticated with a third
 party identifier but the Home Server could not find a matching account in its
 database.
 
-Adding a Third Party Identifier
-+++++++++++++++++++++++++++++++
+Adding Account Administrative Contact Information
++++++++++++++++++++++++++++++++++++++++++++++++++
 This section refers to API Version 2. These API calls currently use the prefix
 ``/_matrix/client/v2_alpha``.
 
@@ -423,18 +433,18 @@ Request::
 
   POST $V2PREFIX/account/3pid
 
-Used to add a third party identifier to the user's account.
+Used to add contact information to the user's account.
 
 The body of the POST request is a JSON object containing:
 
 threePidCreds
-  An object containing third party identifier credentials.
+  An object containing contact information.
 bind
   Optional. A boolean indicating whether the Home Server should also bind this
   third party identifier to the account's matrix ID with the Identity Server. If
   supplied and true, the Home Server must bind the 3pid accordingly.
 
-The third party identifier credentials object comprises:
+The contact information object comprises:
 
 id_server
   The colon-separated hostname and port of the Identity Server used to
@@ -452,8 +462,8 @@ May also return error codes:
 M_THREEPID_AUTH_FAILED
   If the credentials provided could not be verified with the ID Server.
 
-Fetching Currently Associated Third Party Identifiers
-+++++++++++++++++++++++++++++++++++++++++++++++++++++
+Fetching Currently Associated Contact Information
++++++++++++++++++++++++++++++++++++++++++++++++++
 This section refers to API Version 2. These API calls currently use the prefix
 ``/_matrix/client/v2_alpha``.
 
@@ -706,8 +716,8 @@ Pagination
 Pagination is the process of dividing a dataset into multiple discrete pages.
 Matrix makes use of pagination to allow clients to view extremely large datasets.
 These datasets are not limited to events in a room (for example clients may want
-to paginate rooms in addition to events within those rooms). Regardless of *what*
-is being paginated, there is a common underlying API which is used to
+to paginate a list of rooms in addition to events within those rooms). Regardless
+of *what* is being paginated, there is a common underlying API which is used to
 to give clients a consistent way of selecting subsets of a potentially changing
 dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
 which describe where to read from the stream. ``from`` and ``to`` are opaque
@@ -800,13 +810,14 @@ Rooms
 
 Creation
 ~~~~~~~~
-The home server will create a ``m.room.create`` event when a room is created,
+The home server will create an ``m.room.create`` event when a room is created,
 which serves as the root of the event graph for this room. This event also has a
 ``creator`` key which contains the user ID of the room creator. It will also
 generate several other events in order to manage permissions in this room. This
 includes:
 
- - ``m.room.power_levels`` : Sets the power levels of users and required power levels.
+ - ``m.room.power_levels`` : Sets the power levels of users and required power
+   levels for various actions within the room such as sending events.
  - ``m.room.join_rules`` : Whether the room is "invite-only" or not.
 
 See `Room Events`_ for more information on these events. To create a room, a

From c4d1b568435aca0216762e1331a6d42f49bf5f68 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 09:57:21 +0100
Subject: [PATCH 332/467] Add a comment about enforcement.

---
 specification/3-application_service_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index 557e16aa..8d0efdef 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -57,7 +57,7 @@ An example HS configuration required to pass traffic to the AS is:
 .. WARNING::
   If the homeserver in question has multiple application services, each
   ``as_token`` MUST be unique per application service as this token is used to
-  identify the application service.
+  identify the application service. The homeserver MUST enforce this.
 
 - An application service can state whether they should be the only ones who 
   can manage a specified namespace. This is referred to as an "exclusive" 

From c82c07ddd26055cb832d00e8df920a571ead9074 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 09:58:39 +0100
Subject: [PATCH 333/467] Review comments

---
 specification/0-events.rst | 3 +--
 specification/0-intro.rst  | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/specification/0-events.rst b/specification/0-events.rst
index ae5a698a..f24ad984 100644
--- a/specification/0-events.rst
+++ b/specification/0-events.rst
@@ -23,9 +23,8 @@ restrictions on sizes per key:
  - ``state_key`` MUST NOT exceed 255 bytes.
  - ``type`` MUST NOT exceed 255 bytes.
  - ``event_id`` MUST NOT exceed 255 bytes.
- - ``user_id`` MUST NOT exceed 255 bytes.
 
-Some event types have additional sizes restrictions which are specified in
+Some event types have additional size restrictions which are specified in
 the description of the event. Additional keys have no limit other than that
 implied by the total 65 KB limit on events.
 
diff --git a/specification/0-intro.rst b/specification/0-intro.rst
index e5cf1ac8..90ec94ef 100644
--- a/specification/0-intro.rst
+++ b/specification/0-intro.rst
@@ -296,7 +296,7 @@ be obtained by visiting the domain specified. They are case-insensitive. Note
 that the mapping from a room alias to a room ID is not fixed, and may change
 over time to point to a different room ID. For this reason, Clients SHOULD
 resolve the room alias to a room ID once and then use that ID on subsequent
-requests. Room aliases MUST NOT exceed 255 bytes.
+requests. Room aliases MUST NOT exceed 255 bytes (including the domain).
 
 When resolving a room alias the server will also respond with a list of servers
 that are in the room that can be used to join via.

From f51ee706330be5904077b85b595d209794f290d3 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 10:19:51 +0100
Subject: [PATCH 334/467] Review comments round 2

---
 specification/modules/instant_messaging.rst | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 487240de..0b4e3e64 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -84,6 +84,9 @@ Local echo
 Messages SHOULD appear immediately in the message view when a user presses the
 "send" button. This should occur even if the message is still sending. This is
 referred to as "local echo". Clients SHOULD implement "local echo" of messages.
+Clients MAY display messages in a different format to indicate that the server
+has not processed the message. This format should be removed when the server
+responds.
 
 Clients need to be able to match the message they are sending with the same
 message which they receive from the event stream. The echo of the same message
@@ -91,7 +94,7 @@ from the event stream is referred to as "remote echo". Both echoes need to be
 identified as the same message in order to prevent duplicate messages being
 displayed. Ideally this pairing would occur transparently to the user: the UI
 would not flicker as it transitions from local to remote. Flickering cannot be
-fully avoided in version 1 of the client-server API. Two scenarios need to be
+fully avoided in the current client-server API. Two scenarios need to be
 considered:
 
 - The client sends a message and the remote echo arrives on the event stream
@@ -106,8 +109,8 @@ arrives before the client has obtained an event ID. This makes it impossible to
 identify it as a duplicate event. This results in the client displaying the
 message twice for a fraction of a second before the the original request to send
 the message completes. Once it completes, the client can take remedial actions
-to remove the duplicate event by looking for duplicate event IDs. Version 2 of
-the client-server API resolves this by attaching the transaction ID of the
+to remove the duplicate event by looking for duplicate event IDs. A future version
+of the client-server API will resolve this by attaching the transaction ID of the
 sending request to the event itself.
 
 
@@ -132,7 +135,7 @@ Server behaviour
 ----------------
 
 Homeservers SHOULD reject ``m.room.message`` events which don't have a
-``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status
+``msgtype`` key, or which don't have a textual ``body`` key, with an HTTP status
 code of 400.
 
 Security considerations
@@ -141,7 +144,7 @@ Security considerations
 Messages sent using this module are not encrypted. Messages can be encrypted
 using the `E2E module`_.
 
-Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site
+Clients should sanitise **all displayed keys** for unsafe HTML to prevent Cross-Site
 Scripting (XSS) attacks. This includes room names and topics.
 
 .. _`E2E module`: `module:e2e`_

From 507cb29e39c0dec1f0ad51095f350d4e5f37e4e7 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 10:37:15 +0100
Subject: [PATCH 335/467] Review comments round deux

---
 specification/modules/history_visibility.rst | 30 +++++++++++---------
 1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst
index 12ea7b22..0405b3f1 100644
--- a/specification/modules/history_visibility.rst
+++ b/specification/modules/history_visibility.rst
@@ -4,21 +4,25 @@ Room History Visibility
 .. _module:history-visibility:
 
 This module adds support for controlling the visibility of previous events in a
-room. Whether a member of a room can see the events that happened in a room from
-before they joined the room is controlled by the ``m.room.history_visibility``
-event outlined below. In all cases, the member still needs to be joined to the
-room to receive events for that room. The visibility option simply determines
-which subset of events in the room are presented to the client. Visibility can
-take the form of one of three options:
-
-- ``shared`` - Previous events are always shown to newly joined members. All
-  events in the room are shown, even those sent when the member was not a part
+room.
+
+In all cases, a user needs to join a room to view events in that room. Once they
+have joined a room, they will gain access to a subset of events in the room. How
+this subset is chosen is controlled by the ``m.room.history_visibility`` event
+outlined below. After a user has left a room, they may seen any events which they
+were allowed to see before they left the room, but no events received after they
+left.
+
+The three options for this event are:
+
+- ``shared`` - Previous events are always accessible to newly joined members. All
+  events in the room are accessible, even those sent when the member was not a part
   of the room.
-- ``invited`` - Previous events are shown to newly joined members from the point
-  they were invited onwards. Events stop being shown when the member's state
+- ``invited`` - Previous events are accessible to newly joined members from the point
+  they were invited onwards. Events stop being accessible when the member's state
   changes to something other than ``invite`` or ``join``.
-- ``joined`` - Previous events are shown to newly joined members from the point
-  they joined the room onwards. Events stop being shown when the members state
+- ``joined`` - Previous events are accessible to newly joined members from the point
+  they joined the room onwards. Events stop being accessible when the members state
   changes to something other than ``join``.
 
 .. WARNING::

From 5aad67f74d7fe611981aa75c3b42d7c17726b4de Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 11:01:11 +0100
Subject: [PATCH 336/467] Review comments round III

---
 specification/modules/history_visibility.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst
index 0405b3f1..f26d1b59 100644
--- a/specification/modules/history_visibility.rst
+++ b/specification/modules/history_visibility.rst
@@ -13,7 +13,7 @@ outlined below. After a user has left a room, they may seen any events which the
 were allowed to see before they left the room, but no events received after they
 left.
 
-The three options for this event are:
+The three options for the ``m.room.history_visibility`` event are:
 
 - ``shared`` - Previous events are always accessible to newly joined members. All
   events in the room are accessible, even those sent when the member was not a part
@@ -22,7 +22,7 @@ The three options for this event are:
   they were invited onwards. Events stop being accessible when the member's state
   changes to something other than ``invite`` or ``join``.
 - ``joined`` - Previous events are accessible to newly joined members from the point
-  they joined the room onwards. Events stop being accessible when the members state
+  they joined the room onwards. Events stop being accessible when the member's state
   changes to something other than ``join``.
 
 .. WARNING::

From 96a4996c767fe4dc8337c248296d1e7789511079 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 11:02:37 +0100
Subject: [PATCH 337/467] Mention search categories

---
 specification/modules/search.rst | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index d6bfd391..9fee097c 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -10,6 +10,13 @@ Client behaviour
 ----------------
 {{search_http_api}}
 
+Search Categories
+~~~~~~~~~~~~~~~~~
+
+The search API allows clients to search in different categories of items.
+Currently, the only specified category is ``room_events``, which searches for
+events in rooms the user had joined.
+
 Security considerations
 -----------------------
 The server must only return results that the user has permission to see.

From af347baa686b069fa7d77668506d96a66a0fb388 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 11:06:15 +0100
Subject: [PATCH 338/467] Better phrasing

---
 specification/modules/search.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 9fee097c..87116c2c 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -14,7 +14,7 @@ Search Categories
 ~~~~~~~~~~~~~~~~~
 
 The search API allows clients to search in different categories of items.
-Currently, the only specified category is ``room_events``, which searches for
+Currently the only specified category is ``room_events``, which include
 events in rooms the user had joined.
 
 Security considerations

From c47a94658ae0701e52ea912861697460784f7638 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 11:12:00 +0100
Subject: [PATCH 339/467] Mention supported keys

---
 specification/modules/search.rst | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 87116c2c..665709c8 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -11,11 +11,22 @@ Client behaviour
 {{search_http_api}}
 
 Search Categories
-~~~~~~~~~~~~~~~~~
+-----------------
 
 The search API allows clients to search in different categories of items.
-Currently the only specified category is ``room_events``, which include
-events in rooms the user had joined.
+Currently the only specified category is ``room_events``.
+
+``room_events``
+~~~~~~~~~~~~~~~
+
+This category covers all events in rooms that the user had joined. The search
+is performed on certain keys of certain event types.
+
+The supported keys to search over are:
+
+- ``content.body`` in ``m.room.message``
+- ``content.name`` in ``m.room.name``
+- ``content.topic`` in ``m.room.topic``
 
 Security considerations
 -----------------------

From 6b72ddfb8cbf8628f2ea2efe24d05a609588a211 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 13:36:43 +0100
Subject: [PATCH 340/467] Spelling and typos

---
 specification/3-application_service_api.rst | 19 ++++++++++---------
 specification/4-server_server_api.rst       |  2 +-
 specification/6-appendices.rst              |  6 +++---
 3 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index 8d0efdef..cf4942af 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -322,7 +322,7 @@ including the AS token on a ``/register`` request, along with a login type of
 
 Application services which attempt to create users or aliases *outside* of
 their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly,
-normal users who attempt to create users or alises *inside* an application
+normal users who attempt to create users or aliases *inside* an application
 service-defined namespace will receive the same ``M_EXCLUSIVE`` error code,
 but only if the application service has defined the namespace as ``exclusive``.
 
@@ -375,9 +375,10 @@ an API is exposed.
 Room Aliases
 ++++++++++++
 We may want to expose some 3P network rooms so Matrix users can join them directly,
-e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto,
-tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by
-the application service. Private rooms (e.g. sending an email to someone) should not
+e.g. IRC rooms. We don't want to expose every 3P network room though, e.g.
+``mailto``, ``tel``. Rooms which are publicly accessible (e.g. IRC rooms) can be
+exposed as an alias by the application service. Private rooms
+(e.g. sending an email to someone) should not
 be exposed in this way, but should instead operate using normal invite/join semantics.
 Therefore, the ID conventions discussed below are only valid for public rooms which 
 expose room aliases.
@@ -397,9 +398,9 @@ SHOULD be mapped in the same way as "user" URIs.
   
 Event fields
 ++++++++++++
-We recommend that any gatewayed events should include an ``external_url`` field
-in their content to provide a way for Matrix clients to link into the 'native'
-client from which the event originated.  For instance, this could contain the
-message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
-blog comment traffic in & out of matrix
+We recommend that any events that originated from a remote network should
+include an ``external_url`` field in their content to provide a way for Matrix
+clients to link into the 'native' client from which the event originated.
+For instance, this could contain the message-ID for emails/nntp posts, or a link
+to a blog comment when bridging blog comment traffic in & out of Matrix.
 
diff --git a/specification/4-server_server_api.rst b/specification/4-server_server_api.rst
index c5ff2b87..66367cb0 100644
--- a/specification/4-server_server_api.rst
+++ b/specification/4-server_server_api.rst
@@ -630,7 +630,7 @@ because HTTP services like Matrix are often deployed behind load balancers that
 handle the TLS and these load balancers make it difficult to check TLS client
 certificates.
 
-A home server may provide a TLS client certficate and the receiving home server
+A home server may provide a TLS client certificate and the receiving home server
 may check that the client certificate matches the certificate of the origin
 home server.
 
diff --git a/specification/6-appendices.rst b/specification/6-appendices.rst
index de1ac290..c45aa0a5 100644
--- a/specification/6-appendices.rst
+++ b/specification/6-appendices.rst
@@ -24,7 +24,7 @@ Threat: Unrecoverable Consistency Violations
 ++++++++++++++++++++++++++++++++++++++++++++
 
 An attacker could send messages which created an unrecoverable "split-brain"
-state in the cluster such that the victim's servers could no longer dervive a
+state in the cluster such that the victim's servers could no longer derive a
 consistent view of the chatroom state.
 
 Threat: Bad History
@@ -63,7 +63,7 @@ Spoofing
 An attacker could try to send a message claiming to be from the victim without
 the victim having sent the message in order to:
 
-* Impersonate the victim while performing illict activity.
+* Impersonate the victim while performing illicit activity.
 * Obtain privileges of the victim.
 
 Threat: Altering Message Contents
@@ -81,7 +81,7 @@ with a phony "origin" field.
 Spamming
 ~~~~~~~~
 
-The attacker could try to send a high volume of solicicted or unsolicted
+The attacker could try to send a high volume of solicited or unsolicited
 messages to the victim in order to:
 
 * Find victims for scams.

From 3d087df5387f59c1d18f3d0020d0aa26914d2208 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 13:40:05 +0100
Subject: [PATCH 341/467] Kill number prefixes for good

---
 .../{6-appendices.rst => appendices.rst}      |  0
 ...ce_api.rst => application_service_api.rst} |  0
 ...t_server_api.rst => client_server_api.rst} |  0
 ...{0-event_signing.rst => event_signing.rst} |  0
 specification/{0-events.rst => events.rst}    |  0
 ...ture_profiles.rst => feature_profiles.rst} |  0
 ...ntity_servers.rst => identity_servers.rst} |  0
 specification/{0-intro.rst => intro.rst}      |  0
 specification/{2-modules.rst => modules.rst}  |  0
 ...r_server_api.rst => server_server_api.rst} |  0
 specification/targets.yaml                    | 20 +++++++++----------
 11 files changed, 10 insertions(+), 10 deletions(-)
 rename specification/{6-appendices.rst => appendices.rst} (100%)
 rename specification/{3-application_service_api.rst => application_service_api.rst} (100%)
 rename specification/{1-client_server_api.rst => client_server_api.rst} (100%)
 rename specification/{0-event_signing.rst => event_signing.rst} (100%)
 rename specification/{0-events.rst => events.rst} (100%)
 rename specification/{0-feature_profiles.rst => feature_profiles.rst} (100%)
 rename specification/{5-identity_servers.rst => identity_servers.rst} (100%)
 rename specification/{0-intro.rst => intro.rst} (100%)
 rename specification/{2-modules.rst => modules.rst} (100%)
 rename specification/{4-server_server_api.rst => server_server_api.rst} (100%)

diff --git a/specification/6-appendices.rst b/specification/appendices.rst
similarity index 100%
rename from specification/6-appendices.rst
rename to specification/appendices.rst
diff --git a/specification/3-application_service_api.rst b/specification/application_service_api.rst
similarity index 100%
rename from specification/3-application_service_api.rst
rename to specification/application_service_api.rst
diff --git a/specification/1-client_server_api.rst b/specification/client_server_api.rst
similarity index 100%
rename from specification/1-client_server_api.rst
rename to specification/client_server_api.rst
diff --git a/specification/0-event_signing.rst b/specification/event_signing.rst
similarity index 100%
rename from specification/0-event_signing.rst
rename to specification/event_signing.rst
diff --git a/specification/0-events.rst b/specification/events.rst
similarity index 100%
rename from specification/0-events.rst
rename to specification/events.rst
diff --git a/specification/0-feature_profiles.rst b/specification/feature_profiles.rst
similarity index 100%
rename from specification/0-feature_profiles.rst
rename to specification/feature_profiles.rst
diff --git a/specification/5-identity_servers.rst b/specification/identity_servers.rst
similarity index 100%
rename from specification/5-identity_servers.rst
rename to specification/identity_servers.rst
diff --git a/specification/0-intro.rst b/specification/intro.rst
similarity index 100%
rename from specification/0-intro.rst
rename to specification/intro.rst
diff --git a/specification/2-modules.rst b/specification/modules.rst
similarity index 100%
rename from specification/2-modules.rst
rename to specification/modules.rst
diff --git a/specification/4-server_server_api.rst b/specification/server_server_api.rst
similarity index 100%
rename from specification/4-server_server_api.rst
rename to specification/server_server_api.rst
diff --git a/specification/targets.yaml b/specification/targets.yaml
index c2b6eeda..4ac34ae3 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -1,17 +1,17 @@
 targets:
   main:  # arbitrary name to identify this build target
     files:  # the sort order of files to cat
-      - 0-intro.rst
-      - 1-client_server_api.rst
-      - { 1: 0-events.rst }
-      - { 1: 0-event_signing.rst }
-      - 2-modules.rst
-      - { 1: 0-feature_profiles.rst }
+      - intro.rst
+      - client_server_api.rst
+      - { 1: events.rst }
+      - { 1: event_signing.rst }
+      - modules.rst
+      - { 1: feature_profiles.rst }
       - { 1: "group:modules" }  # reference a group of files
-      - 3-application_service_api.rst
-      - 4-server_server_api.rst
-      - 5-identity_servers.rst
-      - 6-appendices.rst
+      - application_service_api.rst
+      - server_server_api.rst
+      - identity_servers.rst
+      - appendices.rst
 groups:  # reusable blobs of files when prefixed with 'group:'
   modules:
     - modules/instant_messaging.rst

From 55cc5c5bb5c63900ca1e496d09a8b78f5373a3d8 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 14:53:03 +0100
Subject: [PATCH 342/467] Swaggerify application services

---
 api/client-server/v1/application_service.yaml | 201 ++++++++++++++++
 specification/3-application_service_api.rst   | 219 +++++++-----------
 2 files changed, 282 insertions(+), 138 deletions(-)
 create mode 100644 api/client-server/v1/application_service.yaml

diff --git a/api/client-server/v1/application_service.yaml b/api/client-server/v1/application_service.yaml
new file mode 100644
index 00000000..5e99562f
--- /dev/null
+++ b/api/client-server/v1/application_service.yaml
@@ -0,0 +1,201 @@
+swagger: '2.0'
+info:
+  title: "Matrix Application Service API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: ""
+consumes:
+  - application/json
+produces:
+  - application/json
+paths:
+  "/transactions/{txnId}":
+    put:
+      summary: Send some events to the application service.
+      description: |-
+        This API is called by the HS when the HS wants to push an event (or
+        batch of events) to the AS.
+      parameters:
+        - in: path
+          name: txnId
+          type: string
+          description: |-
+            The transaction ID for this set of events. Homeservers generate
+            these IDs and they are used to ensure idempotency of requests.
+          required: true
+          x-example: "35"
+        - in: body
+          name: body
+          description: A list of events
+          schema:
+            type: object
+            example: |-
+              {
+                "events": [
+                  {
+                    "age": 32,
+                    "content": {
+                        "body": "incoming message",
+                        "msgtype": "m.text"
+                    },
+                    "event_id": "$14328055551tzaee:localhost",
+                    "origin_server_ts": 1432804485886,
+                    "room_id": "!TmaZBKYIFrIPVGoUYp:localhost",
+                    "type": "m.room.message",
+                    "user_id": "@bob:localhost"
+                  },
+                  {
+                    "age": 1984,
+                    "content": {
+                        "body": "another incoming message",
+                        "msgtype": "m.text"
+                    },
+                    "event_id": "$1228055551ffsef:localhost",
+                    "origin_server_ts": 1432804485886,
+                    "room_id": "!TmaZBKYIFrIPVGoUYp:localhost",
+                    "type": "m.room.message",
+                    "user_id": "@bob:localhost"
+                  }
+                ]
+              }
+            description: "Transaction informations"
+            properties:
+              events:
+                type: array
+                description: A list of events
+                items:
+                  type: object
+                  title: Event
+            required: ["events"]
+      responses:
+        200:
+          description: The transaction was processed successfully.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
+
+  "/rooms/{roomAlias}":
+    get:
+      summary: Query if a room alias should exist on the application service.
+      description: |-
+        This endpoint is invoked by the homeserver on an application service to query
+        the existence of a given room alias. The homeserver will only query room
+        aliases inside the application service's ``aliases`` namespace. The
+        homeserver will send this request when it receives a request to join a
+        room alias within the application services's namespace.
+      parameters:
+        - in: path
+          name: roomAlias
+          type: string
+          description: The room alias being queried.
+          required: true
+          x-example: "#magicforest:example.com"
+      responses:
+        200:
+          description: |-
+            The application service indicates that this room alias exists. The
+            application service MUST have created a room and associated it with
+            the queried room alias using the client-server API. Additional
+            information about the room such as its' name and topic can be set
+            before responding.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
+        401:
+          description: |-
+            The homeserver has not supplied credentials to the application service.
+            Optional error information can be included in the body of this response.
+          examples:
+            application/json: |-
+              {
+                "errcode": "ORG.MATRIX.MYAPPSERVICE_UNAUTHORIZED"
+              }
+          schema:
+            type: object
+        403:
+          description: |-
+            The credentials supplied by the homeserver were rejected.
+          examples:
+            application/json: |-
+              {
+                "errcode": "M_FORBIDDEN"
+              }
+          schema:
+            type: object
+        404:
+          description: |-
+            The application service indicates that this room alias does not exist.
+            Optional error information can be included in the body of this response.
+          examples:
+            application/json: |-
+              {
+                "errcode": "ORG.MATRIX.MYAPPSERVICE_NOT_FOUND"
+              }
+          schema:
+            type: object
+  "/users/{userId}":
+    get:
+      summary: Query if a user should exist on the application service.
+      description: |-
+        This endpoint is invoked by the homeserver on an application service to query
+        the existence of a given user ID. The homeserver will only query user IDs
+        inside the application service's ``users`` namespace. The homeserver will
+        send this request when it receives an *event* for an unknown user ID in
+        the application service's namespace.
+      parameters:
+        - in: path
+          name: userId
+          type: string
+          description: The user ID being queried.
+          required: true
+          x-example: "@alice:example.com"
+      responses:
+        200:
+          description: |-
+            The application service indicates that this user exists. The application
+            service MUST create the user using the client-server API.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
+        401:
+          description: |-
+            The homeserver has not supplied credentials to the application service.
+            Optional error information can be included in the body of this response.
+          examples:
+            application/json: |-
+              {
+                "errcode": "ORG.MATRIX.MYAPPSERVICE_UNAUTHORIZED"
+              }
+          schema:
+            type: object
+        403:
+          description: |-
+            The credentials supplied by the homeserver were rejected.
+          examples:
+            application/json: |-
+              {
+                "errcode": "M_FORBIDDEN"
+              }
+          schema:
+            type: object
+        404:
+          description: |-
+            The application service indicates that this user does not exist.
+            Optional error information can be included in the body of this response.
+          examples:
+            application/json: |-
+              {
+                "errcode": "ORG.MATRIX.MYAPPSERVICE_NOT_FOUND"
+              }
+          schema:
+            type: object
+
diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index 8d0efdef..c4bedf3f 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -4,7 +4,7 @@ Application Service API
 The Matrix client-server API and server-server APIs provide the means to
 implement a consistent self-contained federated messaging fabric. However, they
 provide limited means of implementing custom server-side behaviour in Matrix
-(e.g. gateways, filters, extensible hooks etc). The Application Service API
+(e.g. gateways, filters, extensible hooks etc). The Application Service API (AS API)
 defines a standard API to allow such extensible functionality to be implemented
 irrespective of the underlying homeserver implementation.
 
@@ -20,7 +20,10 @@ They cannot prevent events from being sent, nor can they modify the content of
 the event being sent. In order to observe events from a homeserver, the
 homeserver needs to be configured to pass certain types of traffic to the
 application service.  This is achieved by manually configuring the homeserver
-with information about the AS.
+with information about the application service (AS).
+
+Registration
+~~~~~~~~~~~~
 
 .. NOTE::
   Previously, application services could register with a homeserver via HTTP
@@ -38,7 +41,30 @@ with information about the AS.
   A better solution would be to somehow mandate that the API done to avoid
   abuse.
 
-An example HS configuration required to pass traffic to the AS is:
+Application services register "namespaces" of user IDs, room aliases and room IDs.
+These namespaces are represented as regular expressions. An application service
+is said to be "interested" in a given event if one of the IDs in the event match
+the regular expression provided by the application service. An application
+service can also state whether they should be the only ones who 
+can manage a specified namespace. This is referred to as an "exclusive" 
+namespace. An exclusive namespace prevents humans and other application 
+services from creating/deleting entities in that namespace. Typically,
+exclusive namespaces are used when the rooms represent real rooms on
+another service (e.g. IRC). Non-exclusive namespaces are used when the
+application service is merely augmenting the room itself (e.g. providing
+logging or searching facilities). Namespaces are represented by POSIX extended
+regular expressions and look like:
+
+.. code-block:: yaml
+
+   users:
+     - exclusive: true
+       regex: @irc.freenode.net_.*
+
+
+The registration is represented by a series of key-value pairs, which this
+specification will present as YAML. An example HS configuration required to pass
+traffic to the AS is:
 
 .. code-block:: yaml
 
@@ -59,121 +85,59 @@ An example HS configuration required to pass traffic to the AS is:
   ``as_token`` MUST be unique per application service as this token is used to
   identify the application service. The homeserver MUST enforce this.
 
-- An application service can state whether they should be the only ones who 
-  can manage a specified namespace. This is referred to as an "exclusive" 
-  namespace. An exclusive namespace prevents humans and other application 
-  services from creating/deleting entities in that namespace. Typically,
-  exclusive namespaces are used when the rooms represent real rooms on
-  another service (e.g. IRC). Non-exclusive namespaces are used when the
-  application service is merely augmenting the room itself (e.g. providing
-  logging or searching facilities).
-- Namespaces are represented by POSIX extended regular expressions, 
-  e.g:
-
-.. code-block:: yaml
-
-   users:
-     - exclusive: true
-       regex: @irc.freenode.net_.*
-
 
 Home Server -> Application Service API
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This contains application service APIs which are used by the home server. All
-application services MUST implement these APIs.
 
-User Query
-++++++++++
+Pushing events
+++++++++++++++
 
-This API is called by the HS to query the existence of a user on the Application
-Service's namespace.
+The application service API provides a transaction API for sending a list of
+events. Each list of events includes a transaction ID, which works as follows:
 
-Inputs:
- - User ID
- - HS Credentials
-Output:
- - Whether the user exists.
-Side effects:
- - User is created on the HS by the AS via CS APIs during the processing of this request.
-API called when:
- - HS receives an event for an unknown user ID in the AS's namespace, e.g. an
-   invite event to a room.
-Notes:
- - When the AS receives this request, if the user exists, it must create the user via
-   the CS API.
- - It can also set arbitrary information about the user (e.g. display name, join rooms, etc)
-   using the CS API.
- - When this setup is complete, the AS should respond to the HS request. This means the AS 
-   blocks the HS until the user is created.
- - This is deemed more flexible than alternative methods (e.g. returning a JSON blob with the
-   user's display name and get the HS to provision the user).
-Retry notes:
- - The home server cannot respond to the client's request until the response to
-   this API is obtained from the AS.
- - Recommended that home servers try a few times then time out, returning a
-   408 Request Timeout to the client.
-   
 ::
 
- GET /users/$user_id?access_token=$hs_token
- 
- Returns:
-   200 : User is recognised.
-   404 : User not found.
-   401 : Credentials need to be supplied.
-   403 : HS credentials rejected.
- 
- 
-   200 OK response format
- 
-   {}
-   
-Room Alias Query
-++++++++++++++++
-This API is called by the HS to query the existence of a room alias on the 
-Application Service's namespace.
+ Typical
+ HS ---> AS : Home server sends events with transaction ID T.
+    <---    : AS sends back 200 OK.
+    
+ AS ACK Lost
+ HS ---> AS : Home server sends events with transaction ID T.
+    <-/-    : AS 200 OK is lost.
+ HS ---> AS : Home server retries with the same transaction ID of T.
+    <---    : AS sends back 200 OK. If the AS had processed these events 
+              already, it can NO-OP this request (and it knows if it is the same
+              events based on the transaction ID).
+
+The events sent to the application service should be linearised, as if they were
+from the event stream. The homeserver MUST maintain a queue of transactions to
+send to the AS. If the application service cannot be reached, the homeserver
+SHOULD backoff exponentially until the application service is reachable again.
+As application services cannot *modify* the events in any way, these requests can
+be made without blocking other aspects of the homeserver. Homeservers MUST NOT
+alter (e.g. add more) events they were going to send within that transaction ID
+on retries, as the AS may have already processed the events.
+
+Querying
+++++++++
 
-Inputs:
- - Room alias
- - HS Credentials
-Output:
- - Whether the room exists.
-Side effects:
- - Room is created on the HS by the AS via CS APIs during the processing of 
-   this request.
-API called when:
- - HS receives an event to join a room alias in the AS's namespace.
-Notes:
- - When the AS receives this request, if the room exists, it must create the room via
-   the CS API.
- - It can also set arbitrary information about the room (e.g. name, topic, etc)
-   using the CS API.
- - It can send messages as other users in order to populate scrollback.
- - When this setup is complete, the AS should respond to the HS request. This means the AS 
-   blocks the HS until the room is created and configured.
- - This is deemed more flexible than alternative methods (e.g. returning an initial sync
-   style JSON blob and get the HS to provision the room). It also means that the AS knows
-   the room ID -> alias mapping.
-Retry notes:
- - The home server cannot respond to the client's request until the response to
-   this API is obtained from the AS.
- - Recommended that home servers try a few times then time out, returning a
-   408 Request Timeout to the client.
- 
-::
+The application service API includes two querying APIs: for room aliases and for
+user IDs. The application service SHOULD create the queried entity if it desires.
+During this process, the application service is blocking the homeserver until the
+entity is created and configured. If the homeserver does not receive a response
+to this request, the homeserver should retry several times before timing out. This
+should result in an HTTP status 408 "Request Timeout" on the client which initiated
+this request (e.g. to join a room alias).
+
+.. admonition:: Rationale
+
+  Blocking the homeserver and expecting the application service to create the entity
+  using the client-server API is simpler and more flexible than alternative methods
+  such as returning an initial sync style JSON blob and get the HS to provision
+  the room/user. This also meant that there didn't need to be a "backchannel" to inform
+  the application service about information about the entity such as room ID to
+  room alias mappings.
 
- GET /rooms/$room_alias?access_token=$hs_token
- 
- Returns:
-   200 : Room is recognised.
-   404 : Room not found.
-   401 : Credentials need to be supplied.
-   403 : HS credentials rejected.
- 
- 
-   200 OK response format
- 
-   {}
 
 Pushing
 +++++++
@@ -187,38 +151,9 @@ Inputs:
 Output:
  - None. 
 
-Data flows:
-
-::
-
- Typical
- HS ---> AS : Home server sends events with transaction ID T.
-    <---    : AS sends back 200 OK.
-    
- AS ACK Lost
- HS ---> AS : Home server sends events with transaction ID T.
-    <-/-    : AS 200 OK is lost.
- HS ---> AS : Home server retries with the same transaction ID of T.
-    <---    : AS sends back 200 OK. If the AS had processed these events 
-              already, it can NO-OP this request (and it knows if it is the same
-              events based on the transacton ID).
-            
-
-Retry notes:
- - If the HS fails to pass on the events to the AS, it must retry the request.
- - Since ASes by definition cannot alter the traffic being passed to it (unlike
-   say, a Policy Server), these requests can be done in parallel to general HS
-   processing; the HS doesn't need to block whilst doing this.
- - Home servers should use exponential backoff as their retry algorithm.
- - Home servers MUST NOT alter (e.g. add more) events they were going to 
-   send within that transaction ID on retries, as the AS may have already 
-   processed the events.
     
 Ordering notes:
- - The events sent to the AS should be linearised, as they are from the event
-   stream.
- - The home server will need to maintain a queue of transactions to send to 
-   the AS.
+ - 
 
 ::
 
@@ -231,6 +166,14 @@ Ordering notes:
     ]
   }
 
+HTTP APIs
++++++++++
+
+This contains application service APIs which are used by the home server. All
+application services MUST implement these APIs. These APIs are defined below.
+
+{{application_service_http_api}}
+
 Client-Server v2 API Extensions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

From f20faa80e57a20b8c8512154bb267ef45d17fc60 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 14:58:00 +0100
Subject: [PATCH 343/467] Swagger validation

---
 api/client-server/v1/application_service.yaml | 2 +-
 templating/matrix_templates/units.py          | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v1/application_service.yaml b/api/client-server/v1/application_service.yaml
index 5e99562f..22b4dfc5 100644
--- a/api/client-server/v1/application_service.yaml
+++ b/api/client-server/v1/application_service.yaml
@@ -6,7 +6,7 @@ host: localhost:8008
 schemes:
   - https
   - http
-basePath: ""
+basePath: "/"
 consumes:
   - application/json
 produces:
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 7b871057..37d005c7 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -153,7 +153,7 @@ class MatrixUnits(Units):
         for path in api["paths"]:
             for method in api["paths"][path]:
                 single_api = api["paths"][path][method]
-                full_path = api.get("basePath", "") + path
+                full_path = api.get("basePath", "").rstrip("/") + path
                 endpoint = {
                     "title": single_api.get("summary", ""),
                     "desc": single_api.get("description", single_api.get("summary", "")),
@@ -299,7 +299,7 @@ class MatrixUnits(Units):
                     )
                 ]
                 if len(params_missing_examples) == 0:
-                    path_template = api.get("basePath", "") + path
+                    path_template = api.get("basePath", "").rstrip("/") + path
                     qps = {}
                     body = ""
                     for param in single_api.get("parameters", []):
@@ -398,7 +398,7 @@ class MatrixUnits(Units):
                         })
 
         return {
-            "base": api.get("basePath"),
+            "base": api.get("basePath").rstrip("/"),
             "group": group_name,
             "endpoints": endpoints,
         }

From 906c59385f19b7fab34135146663872e3d0ce15a Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 15:49:58 +0100
Subject: [PATCH 344/467] s/had/has/

---
 specification/modules/search.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 665709c8..785ce26a 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -19,7 +19,7 @@ Currently the only specified category is ``room_events``.
 ``room_events``
 ~~~~~~~~~~~~~~~
 
-This category covers all events in rooms that the user had joined. The search
+This category covers all events in rooms that the user has joined. The search
 is performed on certain keys of certain event types.
 
 The supported keys to search over are:

From cfca4a6c09a5ea31fb8193c70f61c6fa9ca055ec Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 15:51:02 +0100
Subject: [PATCH 345/467] s/creteria/criteria/

---
 api/client-server/v1/search.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index a40d49c0..cdb53cf2 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -46,7 +46,7 @@ paths:
                 type: object
                 title: "Categories"
                 description: Describes which categories to search in and
-                  their creteria.
+                  their criteria.
                 properties:
                   room_events:
                     type: object
@@ -76,7 +76,7 @@ paths:
                 type: object
                 title: Categories
                 description: Describes which categories to search in and
-                  their creteria.
+                  their criteria.
                 properties:
                   room_events:
                     type: object

From 770cfbc5dee292498a2aaffc5144f70cc7f2f456 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 16:08:24 +0100
Subject: [PATCH 346/467] Be explicit about the events the search is performed
 over

---
 specification/modules/search.rst | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 785ce26a..9b88cc69 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -3,8 +3,9 @@ Server Side Search
 
 .. _module:search:
 
-The search API allows clients to perform full text search across events in
-their rooms without having to download the entire histories.
+The search API allows clients to perform full text search across events in all
+rooms that the user has been in, including those that they have left. Only
+events that the user is allowed to see will be searched.
 
 Client behaviour
 ----------------
@@ -19,8 +20,9 @@ Currently the only specified category is ``room_events``.
 ``room_events``
 ~~~~~~~~~~~~~~~
 
-This category covers all events in rooms that the user has joined. The search
-is performed on certain keys of certain event types.
+This category covers all events that the user is allowed to see, including
+events in rooms that they have left. The search is performed on certain keys of
+certain event types.
 
 The supported keys to search over are:
 

From 24e36adbe3da45f35113d5712f9ee9fa71ccefb4 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 16:09:11 +0100
Subject: [PATCH 347/467] Mention e2e encryption

---
 specification/modules/search.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 9b88cc69..62c78e8f 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -30,6 +30,8 @@ The supported keys to search over are:
 - ``content.name`` in ``m.room.name``
 - ``content.topic`` in ``m.room.topic``
 
+The search will *not* include rooms that are end to end encrypted.
+
 Security considerations
 -----------------------
 The server must only return results that the user has permission to see.

From eca98af89693ec09eb68c8ab7f230d9852606367 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Thu, 15 Oct 2015 16:38:12 +0100
Subject: [PATCH 348/467] Swaggerify /register endpoint

Need to move registration/login/auth sections around once #94 lands.
---
 api/client-server/v2_alpha/registration.yaml | 95 ++++++++++++++++++++
 specification/1-client_server_api.rst        | 49 ++--------
 2 files changed, 102 insertions(+), 42 deletions(-)
 create mode 100644 api/client-server/v2_alpha/registration.yaml

diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml
new file mode 100644
index 00000000..ad591206
--- /dev/null
+++ b/api/client-server/v2_alpha/registration.yaml
@@ -0,0 +1,95 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v2 Registration API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v2_alpha
+consumes:
+  - application/json
+produces:
+  - application/json
+paths:
+  "/register":
+    post:
+      summary: Register for an account on this homeserver.
+      description: |-
+        Register for an account on this homeserver.
+      parameters:
+        - in: body
+          name: body
+          schema:
+            type: object
+            example: |-
+              {
+                "username": "cheeky_monkey",
+                "password": "ilovebananas",
+                "bind_email": false
+              }
+            properties:
+              bind_email:
+                type: boolean
+                description: |-
+                  If true, the server binds the email used for authentication to
+                  the Matrix ID with the ID Server.
+              username:
+                type: string
+                description: |-
+                  The local part of the desired Matrix ID. If omitted,
+                  the homeserver MUST generate a Matrix ID local part.
+              password:
+                type: string
+                description: The desired password for the account.
+            required: ["password"]
+      responses:
+        200:
+          description: The account has been registered.
+          examples:
+            application/json: |-
+              {
+                "user_id": "@cheeky_monkey:matrix.org",
+                "access_token": "abc123",
+                "home_server": "matrix.org",
+                "refresh_token": "def456"
+              }
+          schema:
+            type: object
+            properties:
+              user_id:
+                type: string
+                description: The fully-qualified Matrix ID that has been registered.
+              access_token:
+                type: string
+                description: |-
+                  An access token for the account.
+                  This access token can then be used to authorize other requests.
+                  The access token may expire at some point, and if so, it SHOULD come with a ``refresh_token``.
+                  There is no specific error message to indicate that a request has failed because
+                  an access token has expired; instead, if a client has reason to believe its
+                  access token is valid, and it receives an auth error, they should attempt to
+                  refresh for a new token on failure, and retry the request with the new token.
+              refresh_token:
+                type: string
+                # TODO: Work out how to linkify /tokenrefresh
+                description: |-
+                  (optional) A ``refresh_token`` may be exchanged for a new ``access_token`` using the /tokenrefresh API endpoint.
+              home_server:
+                type: string
+                description: The hostname of the Home Server on which the account has been registered.
+        400:
+          description: |-
+            Part of the request was invalid. For example, this user ID may be taken. This may
+            be returned at any stage of the registration process, including after authentication
+            if the requested user ID was registered whilst the client was performing authentication.
+          examples:
+            application/json: |-
+              {
+                  "errcode": "M_USER_IN_USE",
+                  "error": "Desired user ID is already taken."
+              }
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index a63c114a..84de17da 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1001,52 +1001,17 @@ member's state, by making a request to
   }
 
 
-Registration
-------------
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Registering for a user account is done using the request::
-
-  POST $V2PREFIX/register
+Account operations
+------------------
 
+Registration
+~~~~~~~~~~~~
 This API endpoint uses the User-Interactive Authentication API.
-This API endpoint does not require an access token.
 
-The body of the POST request is a JSON object containing:
+{{v2_registration_http_api}}
 
-username
-  Optional. This is the local part of the desired Matrix ID. If omitted, the
-  Home Server must generate a Matrix ID local part.
-password
-  Required. The desired password for the account.
-bind_email
-  Optional. If ``true``, the server binds the email used for authentication to
-  the Matrix ID with the ID Server.
-
-On success, this returns a JSON object with keys:
-
-user_id
-  The fully-qualified Matrix ID that has been registered.
-access_token
-  An access token for the new account.
-home_server
-  The hostname of the Home Server on which the account has been registered.
-
-This endpoint may also return the following error codes:
-
-M_USER_IN_USE
-  If the Matrix ID is already in use
-M_EXCLUSIVE
-  If the requested Matrix ID is in the exclusive namespace of an application
-  service.
-
-Home Servers MUST perform the relevant checks and return these codes before
-performing User-Interactive Authentication, although they may also return
-them after authentication is completed if, for example, the requested user ID
-was registered whilst the client was performing authentication.
-
-Old V1 API docs: |register|_
+Login
+~~~~~
 
 {{login_http_api}}
 

From a6cb2e4339a9661dc822057666f92edc4adc5008 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Thu, 15 Oct 2015 16:40:21 +0100
Subject: [PATCH 349/467] Mention rank

---
 specification/modules/search.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 62c78e8f..493c2b7c 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -32,6 +32,9 @@ The supported keys to search over are:
 
 The search will *not* include rooms that are end to end encrypted.
 
+The results include a ``rank`` key that can be used to sort the results by
+revelancy. The higher the ``rank`` the more relevant the result is.
+
 Security considerations
 -----------------------
 The server must only return results that the user has permission to see.

From eb59b8e9d1ac096865bf433c803b69d1e25c9419 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 16 Oct 2015 10:16:40 +0100
Subject: [PATCH 350/467] Mention count

---
 specification/modules/search.rst | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 493c2b7c..03d2475f 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -35,6 +35,10 @@ The search will *not* include rooms that are end to end encrypted.
 The results include a ``rank`` key that can be used to sort the results by
 revelancy. The higher the ``rank`` the more relevant the result is.
 
+The value of ``count`` may not match the number of results. For example due to
+the search query matching 1000s of results and the server truncating the
+response.
+
 Security considerations
 -----------------------
 The server must only return results that the user has permission to see.

From 716c5b7a8b6d4a4109e0fee16929a88fae04bda4 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 16 Oct 2015 10:26:14 +0100
Subject: [PATCH 351/467] Add 400 and 429 response codes

---
 api/client-server/v1/search.yaml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index cdb53cf2..885e62b3 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -132,3 +132,9 @@ paths:
                     }
                   }
                 }
+        400:
+          description: Part of the request was invalid.
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"

From 20b11281ea822f672f7f24d139a7e170efb4cbf3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 16 Oct 2015 13:10:19 +0100
Subject: [PATCH 352/467] Remove lies about OAuth which doesn't exist

---
 specification/3-application_service_api.rst | 11 +----------
 1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index 8d0efdef..23fc126f 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -249,11 +249,8 @@ additional permissions granting the AS permission to masquerade as a matrix user
 
 Inputs:
  - Application service token (``access_token``)
+ - User ID in the AS namespace to act as.
 
- Either:
-   - User ID in the AS namespace to act as.
- Or:
-   - OAuth2 token of real user (which may end up being an access token) 
 Notes:
  - This will apply on all aspects of the CS API, except for Account Management.
  - The ``as_token`` is inserted into ``access_token`` which is usually where the
@@ -268,12 +265,6 @@ Notes:
    access_token: The application service token
    user_id: The desired user ID to act as.
    
- /path?access_token=$token&user_token=$token
-
- Query Parameters:
-   access_token: The application service token
-   user_token: The token granted to the AS by the real user
-
 Timestamp massaging
 +++++++++++++++++++
 The application service may want to inject events at a certain time (reflecting

From 6770d6b2d64530113525729061387ce64090f4eb Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 14:05:18 +0100
Subject: [PATCH 353/467] Review comments

---
 api/client-server/v1/application_service.yaml |  4 +-
 specification/3-application_service_api.rst   | 37 +++----------------
 2 files changed, 7 insertions(+), 34 deletions(-)

diff --git a/api/client-server/v1/application_service.yaml b/api/client-server/v1/application_service.yaml
index 22b4dfc5..a7400bd4 100644
--- a/api/client-server/v1/application_service.yaml
+++ b/api/client-server/v1/application_service.yaml
@@ -87,7 +87,7 @@ paths:
         the existence of a given room alias. The homeserver will only query room
         aliases inside the application service's ``aliases`` namespace. The
         homeserver will send this request when it receives a request to join a
-        room alias within the application services's namespace.
+        room alias within the application service's namespace.
       parameters:
         - in: path
           name: roomAlias
@@ -101,7 +101,7 @@ paths:
             The application service indicates that this room alias exists. The
             application service MUST have created a room and associated it with
             the queried room alias using the client-server API. Additional
-            information about the room such as its' name and topic can be set
+            information about the room such as its name and topic can be set
             before responding.
           examples:
             application/json: |-
diff --git a/specification/3-application_service_api.rst b/specification/3-application_service_api.rst
index c4bedf3f..decb6f81 100644
--- a/specification/3-application_service_api.rst
+++ b/specification/3-application_service_api.rst
@@ -12,10 +12,10 @@ irrespective of the underlying homeserver implementation.
   Add in Client-Server services? Overview of bots? Seems weird to be in the spec
   given it is VERY implementation specific.
 
-Passive Application Services
-----------------------------
-"Passive" application services can only observe events from a given home server,
-and inject events into a room they are participating in.
+Application Services
+--------------------
+Application services are passive and can only observe events from a given
+homeserver. They can inject events into rooms they are participating in.
 They cannot prevent events from being sent, nor can they modify the content of
 the event being sent. In order to observe events from a homeserver, the
 homeserver needs to be configured to pass certain types of traffic to the
@@ -139,33 +139,6 @@ this request (e.g. to join a room alias).
   room alias mappings.
 
 
-Pushing
-+++++++
-This API is called by the HS when the HS wants to push an event (or batch of 
-events) to the AS.
-
-Inputs:
- - HS Credentials
- - Event(s) to give to the AS
- - HS-generated transaction ID
-Output:
- - None. 
-
-    
-Ordering notes:
- - 
-
-::
-
-  PUT /transactions/$transaction_id?access_token=$hs_token
- 
-  Request format
-  {
-    events: [
-      ...
-    ]
-  }
-
 HTTP APIs
 +++++++++
 
@@ -177,7 +150,7 @@ application services MUST implement these APIs. These APIs are defined below.
 Client-Server v2 API Extensions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Passive application services can utilise a more powerful version of the 
+Application services can utilise a more powerful version of the 
 client-server API by identifying itself as an application service to the
 home server.
 

From e0fe3c42c8cb58c994396310279a18b9fa8b9f01 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 14:19:00 +0100
Subject: [PATCH 354/467] Review comments

---
 api/client-server/v2_alpha/registration.yaml | 12 +++++++++---
 specification/1-client_server_api.rst        |  2 +-
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml
index ad591206..9973d0b4 100644
--- a/api/client-server/v2_alpha/registration.yaml
+++ b/api/client-server/v2_alpha/registration.yaml
@@ -80,9 +80,15 @@ paths:
                 description: The hostname of the Home Server on which the account has been registered.
         400:
           description: |-
-            Part of the request was invalid. For example, this user ID may be taken. This may
-            be returned at any stage of the registration process, including after authentication
-            if the requested user ID was registered whilst the client was performing authentication.
+            Part of the request was invalid. This may include one of the following error codes:
+
+              * ``M_USER_IN_USE`` : The desired user ID is already taken.
+              * ``M_EXCLUSIVE`` : The desired user ID is in the exclusive namespace
+                claimed by an application service.
+
+            This errors may be returned at any stage of the registration process,
+            including after authentication if the requested user ID was registered
+            whilst the client was performing authentication.
           examples:
             application/json: |-
               {
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index 84de17da..331f951a 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -1006,7 +1006,7 @@ Account operations
 
 Registration
 ~~~~~~~~~~~~
-This API endpoint uses the User-Interactive Authentication API.
+This API endpoint uses the `User-Interactive Authentication API`_.
 
 {{v2_registration_http_api}}
 

From 4bb042daeb3b6bce59f75c308f059910517bd9b0 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 15:22:50 +0100
Subject: [PATCH 355/467] Review comments round II

---
 api/client-server/v1/sync.yaml        |  30 ++--
 specification/1-client_server_api.rst | 210 +++++++++++++-------------
 2 files changed, 122 insertions(+), 118 deletions(-)

diff --git a/api/client-server/v1/sync.yaml b/api/client-server/v1/sync.yaml
index 50af8546..d07e9399 100644
--- a/api/client-server/v1/sync.yaml
+++ b/api/client-server/v1/sync.yaml
@@ -41,16 +41,6 @@ paths:
           description: The maximum time in milliseconds to wait for an event.
           required: false
           x-example: "35000"
-        - in: query
-          type: boolean
-          name: archived
-          description: |-
-            Whether to include rooms that the user has left. If ``false`` then
-            only rooms that the user has been invited to or has joined are
-            included. If set to ``true`` then rooms that the user has left are
-            included as well. By default this is ``false``.
-          required: false
-          x-example: "true"
       responses:
         200:
           description: "The events received, which may be none."
@@ -80,19 +70,19 @@ paths:
               start:
                 type: string
                 description: |-
-                  A token which correlates to the first value in ``chunk``. Used
-                  for pagination.
+                  A token which correlates to the first value in ``chunk``. This
+                  is usually the same token supplied to ``from=``.
               end:
                 type: string
                 description: |-
-                  A token which correlates to the last value in ``chunk``. Used
-                  for pagination.
+                  A token which correlates to the last value in ``chunk``. This
+                  token should be used in the next request to ``/events``.
               chunk:
                 type: array
                 description: "An array of events."
                 items:
                   type: object
-                  title: RoomEvent
+                  title: Event
                   allOf:
                    - "$ref": "core-event-schema/room_event.json"
         400:
@@ -112,6 +102,16 @@ paths:
           description: The maximum number of messages to return for each room.
           required: false
           x-example: "2"
+        - in: query
+          type: boolean
+          name: archived
+          description: |-
+            Whether to include rooms that the user has left. If ``false`` then
+            only rooms that the user has been invited to or has joined are
+            included. If set to ``true`` then rooms that the user has left are
+            included as well. By default this is ``false``.
+          required: false
+          x-example: "true"
       responses:
         200:
           description: The user's current state.
diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst
index b96b4bb3..96f62c9b 100644
--- a/specification/1-client_server_api.rst
+++ b/specification/1-client_server_api.rst
@@ -491,6 +491,100 @@ medium
 address
   The textual address of the 3pid, eg. the email address
 
+Pagination
+----------
+
+.. NOTE::
+  The paths referred to in this section are not actual endpoints. They only
+  serve as examples to explain how pagination functions.
+
+Pagination is the process of dividing a dataset into multiple discrete pages.
+Matrix makes use of pagination to allow clients to view extremely large datasets.
+These datasets are not limited to events in a room (for example clients may want
+to paginate a list of rooms in addition to events within those rooms). Regardless
+of *what* is being paginated, there is a common underlying API which is used to
+to give clients a consistent way of selecting subsets of a potentially changing
+dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
+which describe where to read from the stream. ``from`` and ``to`` are opaque
+textual 'stream tokens' which describe the current position in the dataset.
+The ``dir`` parameter is an enum representing the direction of events to return:
+either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and
+``end`` stream token values which can then be passed to subsequent requests to
+continue pagination. Not all endpoints will make use of all the parameters
+outlined here: see the specific endpoint in question for more information.
+
+Pagination Request Query Parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+       
+Query parameters:
+  from:
+    $streamtoken - The opaque token to start streaming from.
+  to:
+    $streamtoken - The opaque token to end streaming at. Typically,
+    clients will not know the item of data to end at, so this will usually be 
+    omitted.
+  limit:
+    integer - An integer representing the maximum number of items to 
+    return.
+  dir:
+    f|b - The direction to return events in. Typically this is ``b`` to paginate
+    backwards in time.
+
+'START' and 'END' are placeholder values used in these examples to describe the
+start and end of the dataset respectively.
+
+Unless specified, the default pagination parameters are ``from=START``,
+``to=END``, without a limit set.
+
+For example, if an endpoint had events E1 -> E15. The client wants the last 5 
+events and doesn't know any previous events::
+
+    S                                                    E
+    |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
+    |                               |                    |
+    |                          _____|  <--backwards--    |
+    |__________________       |         |        ________|
+                       |      |         |        |
+     GET /somepath?to=START&limit=5&dir=b&from=END
+     Returns:
+       E15,E14,E13,E12,E11
+
+
+Another example: a public room list has rooms R1 -> R17. The client is showing 5 
+rooms at a time on screen, and is on page 2. They want to
+now show page 3 (rooms R11 -> 15)::
+
+    S                                                           E
+    |  0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 | stream token
+    |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room
+                      |____________| |________________|
+                            |                |
+                        Currently            |
+                        viewing              |
+                                             |
+                             GET /roomslist?from=9&to=END&limit=5
+                             Returns: R11,R12,R13,R14,R15
+                         
+Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 
+token from the initial request was '9' which corresponded to R10. When the 2nd
+request was made, R10 did not appear again, even though from=9 was specified. If
+you know the token, you already have the data.
+
+Pagination Response
+~~~~~~~~~~~~~~~~~~~
+
+Responses to pagination requests MUST follow the format::
+
+  {
+    "chunk": [ ... , Responses , ... ],
+    "start" : $streamtoken,
+    "end" : $streamtoken
+  }
+
+Where $streamtoken is an opaque token which can be used in another query to
+get the next set of results. The "start" and "end" keys can only be omitted if
+the complete dataset is provided in "chunk".
+
 Events
 ------
 
@@ -563,8 +657,14 @@ returning early if an event occurs. Only the events API supports long-polling.
 All events which are visible to the client will appear in the
 events API. When the request returns, an ``end`` token is included in the
 response. This token can be used in the next request to continue where the
-last request left off. Multiple events can be returned per long-poll. All events
-must be de-duplicated based on their event ID.
+last request left off. Multiple events can be returned per long-poll.
+
+.. Warning::
+  Events are ordered in this API according to the arrival time of the event on
+  the homeserver. This can conflict with other APIs which order events based on
+  their partial ordering in the event graph. This can result in duplicate events
+  being received (once per API call). Clients SHOULD de-duplicate events based
+  on the event ID when this happens.
 
 .. TODO
   is deduplication actually a hard requirement in CS v2?
@@ -573,8 +673,8 @@ must be de-duplicated based on their event ID.
   Do we ever support streaming requests? Why not websockets?
 
 When the client first logs in, they will need to initially synchronise with
-their home server. This is achieved via the initial sync API. This API also
-returns an ``end`` token which can be used with the event stream.
+their home server. This is achieved via the initial sync API described below.
+This API also returns an ``end`` token which can be used with the event stream.
 
 {{sync_http_api}}
 
@@ -667,6 +767,9 @@ There are several APIs provided to ``GET`` events for a room:
 
 {{rooms_http_api}}
 
+
+{{message_pagination_http_api}}
+
 Redactions
 ~~~~~~~~~~
 Since events are extensible it is possible for malicious users and/or servers
@@ -712,105 +815,6 @@ The redaction event should be added under the key ``redacted_because``. When a
 client receives a redaction event it should change the redacted event
 in the same way a server does.
 
-Pagination
-----------
-
-.. NOTE::
-  The paths referred to in this section are not actual endpoints. They only
-  serve as examples to explain how pagination functions.
-
-Pagination is the process of dividing a dataset into multiple discrete pages.
-Matrix makes use of pagination to allow clients to view extremely large datasets.
-These datasets are not limited to events in a room (for example clients may want
-to paginate a list of rooms in addition to events within those rooms). Regardless
-of *what* is being paginated, there is a common underlying API which is used to
-to give clients a consistent way of selecting subsets of a potentially changing
-dataset. Requests pass in ``from``, ``to``, ``dir`` and ``limit`` parameters
-which describe where to read from the stream. ``from`` and ``to`` are opaque
-textual 'stream tokens' which describe the current position in the dataset.
-The ``dir`` parameter is an enum representing the direction of events to return:
-either ``f`` orwards or ``b`` ackwards. The response returns new ``start`` and
-``end`` stream token values which can then be passed to subsequent requests to
-continue pagination. Not all endpoints will make use of all the parameters
-outlined here: see the specific endpoint in question for more information.
-
-Pagination Request Query Parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-       
-Query parameters:
-  from:
-    $streamtoken - The opaque token to start streaming from.
-  to:
-    $streamtoken - The opaque token to end streaming at. Typically,
-    clients will not know the item of data to end at, so this will usually be 
-    omitted.
-  limit:
-    integer - An integer representing the maximum number of items to 
-    return.
-  dir:
-    f|b - The direction to return events in. Typically this is ``b`` to paginate
-    backwards in time.
-
-'START' and 'END' are placeholder values used in these examples to describe the
-start and end of the dataset respectively.
-
-Unless specified, the default pagination parameters are ``from=START``,
-``to=END``, without a limit set.
-
-For example, if an endpoint had events E1 -> E15. The client wants the last 5 
-events and doesn't know any previous events::
-
-    S                                                    E
-    |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-|
-    |                               |                    |
-    |                          _____|  <--backwards--    |
-    |__________________       |         |        ________|
-                       |      |         |        |
-     GET /somepath?to=START&limit=5&dir=b&from=END
-     Returns:
-       E15,E14,E13,E12,E11
-
-
-Another example: a public room list has rooms R1 -> R17. The client is showing 5 
-rooms at a time on screen, and is on page 2. They want to
-now show page 3 (rooms R11 -> 15)::
-
-    S                                                           E
-    |  0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 | stream token
-    |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room
-                      |____________| |________________|
-                            |                |
-                        Currently            |
-                        viewing              |
-                                             |
-                             GET /roomslist?from=9&to=END&limit=5
-                             Returns: R11,R12,R13,R14,R15
-                         
-Note that tokens are treated in an *exclusive*, not inclusive, manner. The end 
-token from the initial request was '9' which corresponded to R10. When the 2nd
-request was made, R10 did not appear again, even though from=9 was specified. If
-you know the token, you already have the data.
-
-Pagination Response
-~~~~~~~~~~~~~~~~~~~
-
-Responses to pagination requests MUST follow the format::
-
-  {
-    "chunk": [ ... , Responses , ... ],
-    "start" : $streamtoken,
-    "end" : $streamtoken
-  }
-
-Where $streamtoken is an opaque token which can be used in another query to
-get the next set of results. The "start" and "end" keys can only be omitted if
-the complete dataset is provided in "chunk".
-
-Pagination APIs
-~~~~~~~~~~~~~~~
-
-{{message_pagination_http_api}}
-
 Rooms
 -----
 

From 62d53b4f3321d27cb0b3bc95e5b7625c1efe454d Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 15:58:11 +0100
Subject: [PATCH 356/467] Review comments round III

---
 specification/client_server_api.rst | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 96f62c9b..ee70d9d3 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -663,11 +663,8 @@ last request left off. Multiple events can be returned per long-poll.
   Events are ordered in this API according to the arrival time of the event on
   the homeserver. This can conflict with other APIs which order events based on
   their partial ordering in the event graph. This can result in duplicate events
-  being received (once per API call). Clients SHOULD de-duplicate events based
-  on the event ID when this happens.
-
-.. TODO
-  is deduplication actually a hard requirement in CS v2?
+  being received (once per distinct API called). Clients SHOULD de-duplicate
+  events based on the event ID when this happens.
 
 .. TODO-spec
   Do we ever support streaming requests? Why not websockets?

From d39a9082a0feabf926a94ac458f6e480bfcece11 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 16:43:33 +0100
Subject: [PATCH 357/467] Add invite_room_state to spec. Flesh out info.

---
 event-schemas/schema/v1/m.room.member           | 17 +++++++++++------
 .../matrix_templates/templates/events.tmpl      | 10 +++++-----
 templating/matrix_templates/units.py            |  8 ++++++++
 3 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 912f6cf3..9414a174 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "The current membership state of a user in the room.",
-    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. The ``third_party_invite`` property will be set if the invite was an ``m.room.third_party_invite`` event, and absent if the invite was an ``m.room.member`` event.",
+    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if the invite was an ``m.room.third_party_invite`` event, and absent if the invite was an ``m.room.member`` event.\n\nThis event also includes an ``invite_room_state`` key **outside the** ``content`` **key**. This contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
     "allOf": [{
         "$ref": "core-event-schema/state_event.json"
     }],
@@ -24,7 +24,7 @@
                 },
                 "third_party_invite": {
                     "type": "object",
-                    "title": "invite",
+                    "title": "Invite",
                     "properties": {
                         "token": {
                             "type": "string",
@@ -65,17 +65,22 @@
             "description": "A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``",
             "items": {
                 "type": "object",
-                "title": "StateEvent",
+                "title": "StrippedState",
                 "description": "A stripped down state event, with only the ``type``, ``state_key`` and ``content`` keys.",
                 "properties": {
                     "type": {
-                        "type": "string"
+                        "type": "string",
+                        "description": "The ``type`` for the event.",
+                        "enum": ["``m.room.join_rules``", "``m.room.canonical_alias``", "``m.room.avatar``", "``m.room.name``"]
                     },
                     "state_key": {
-                        "type": "string"
+                        "type": "string",
+                        "description": "The ``state_key`` for the event."
                     },
                     "content": {
-                        "type": "object"
+                        "title": "EventContent",
+                        "type": "object",
+                        "description": "The ``content`` for the event."
                     }
                 }
             }
diff --git a/templating/matrix_templates/templates/events.tmpl b/templating/matrix_templates/templates/events.tmpl
index ffae59e0..9dcc468e 100644
--- a/templating/matrix_templates/templates/events.tmpl
+++ b/templating/matrix_templates/templates/events.tmpl
@@ -7,18 +7,18 @@
 {% for table in event.content_fields -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-================== ================= ===========================================
-   {{table.title or "Content"}} Key          Type                 Description
-================== ================= ===========================================
+======================= ================= ===========================================
+   {{table.title or "Content"}} Key                Type                 Description
+======================= ================= ===========================================
 {% for row in table.rows -%}
 {# -#}
 {# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
 {# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
 {#      It also needs to then wrap inside the desc col (43 ch width) -#}
 {# -#}
-{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(37)}}
+{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(42)}}
 {% endfor -%}
-================== ================= ===========================================
+======================= ================= ===========================================
 
 {% endfor %}
 Example::
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 7b871057..33b5e6b4 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -532,6 +532,14 @@ class MatrixUnits(Units):
                     Units.prop(json_schema, "properties/content")
                 )
 
+                # this is horrible
+                if schema["type"] == "m.room.member":
+                    invite_room_state = get_json_schema_object_fields(
+                        json_schema["properties"]["invite_room_state"]["items"]
+                    )
+                    schema["content_fields"].extend(invite_room_state)
+
+
                 # grab msgtype if it is the right kind of event
                 msgtype = Units.prop(
                     json_schema, "properties/content/properties/msgtype/enum"

From a8d8412068c934acb19bfcd1b3423d35bb01c5bc Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 16:48:18 +0100
Subject: [PATCH 358/467] Add invite_room_state example.

---
 event-schemas/examples/v1/m.room.member | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/event-schemas/examples/v1/m.room.member b/event-schemas/examples/v1/m.room.member
index a5ab79b5..078cc5f1 100644
--- a/event-schemas/examples/v1/m.room.member
+++ b/event-schemas/examples/v1/m.room.member
@@ -12,6 +12,22 @@
         "sender": "@zun:zun.soft"
     }
   },
+  "invite_room_state": [
+    {
+      "type": "m.room.name",
+      "state_key": "",
+      "content": {
+        "name": "Forest of Magic"
+      }
+    },
+    {
+      "type": "m.room.join_rules",
+      "state_key": "",
+      "content": {
+        "join_rules": "invite"
+      }
+    }
+  ],
   "state_key": "@alice:localhost",
   "origin_server_ts": 1431961217939,
   "event_id": "$WLGTSEFSEF:localhost",

From 83d21484df4f01ee3ce8dd10820b7c7778686612 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Fri, 16 Oct 2015 16:54:40 +0100
Subject: [PATCH 359/467] Oopsie

---
 event-schemas/schema/v1/m.room.member | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 9414a174..54ae9100 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -71,7 +71,7 @@
                     "type": {
                         "type": "string",
                         "description": "The ``type`` for the event.",
-                        "enum": ["``m.room.join_rules``", "``m.room.canonical_alias``", "``m.room.avatar``", "``m.room.name``"]
+                        "enum": ["m.room.join_rules", "m.room.canonical_alias", "m.room.avatar", "m.room.name"]
                     },
                     "state_key": {
                         "type": "string",

From 6161a920b6b3b54b6806c01011acf374ca45f180 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 16 Oct 2015 18:29:20 +0100
Subject: [PATCH 360/467] Update 3pid invite section to reflect signed property

---
 event-schemas/examples/v1/m.room.member       | 10 +++++++++-
 event-schemas/schema/v1/m.room.member         | 20 ++++++++++++++++---
 specification/modules/third_party_invites.rst | 12 ++++++-----
 3 files changed, 33 insertions(+), 9 deletions(-)

diff --git a/event-schemas/examples/v1/m.room.member b/event-schemas/examples/v1/m.room.member
index a5ab79b5..4d7896f6 100644
--- a/event-schemas/examples/v1/m.room.member
+++ b/event-schemas/examples/v1/m.room.member
@@ -8,7 +8,15 @@
         "token": "pc98",
         "public_key": "abc123",
         "key_validity_url": "https://magic.forest/verifykey",
-        "signature": "q1w2e3",
+        "signed": {
+          "mxid": "@alice:localhost",
+          "token": "pc98",
+          "signatures": {
+            "magic.forest": {
+              "ed25519:0": "poi098"
+            }
+          }
+        },
         "sender": "@zun:zun.soft"
     }
   },
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 912f6cf3..205c426e 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -38,9 +38,23 @@
                             "type": "string",
                             "description": "A base64-encoded ed25519 key with which token must be signed."
                         },
-                        "signature": {
-                            "type": "string",
-                            "description": "A base64-encoded signature of token with public_key."
+                        "signed": {
+                            "type": "object",
+                            "title": "signed_third_party_invite",
+                            "properties": {
+                                "mxid": {
+                                    "type": "string",
+                                    "description": "The invited matrix user ID. Must be equal to the user_id property of the event."
+                                },
+                                "token": {
+                                    "type": "string",
+                                    "description": "The token property of the containing third_party_invite object.",
+                                },
+                                "signatures": {
+                                    "type": "object",
+                                    "description": "A single signature from the verifying server, in the format specified by the Signing Events section."
+                                }
+                            }
                         },
                         "sender": {
                             "type": "string",
diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index a9883db5..ebca59cf 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -36,7 +36,8 @@ A client asks a server to invite a user by their third party identifier.
 Server behaviour
 ----------------
 
-All homeservers MUST verify that sig(``token``, ``public_key``) = ``signature``.
+All homeservers MUST verify the signature in the ``signed`` property of the
+``third_party_invite`` property in the ``content`` the event.
 
 If a client of the current homeserver is joining by an
 ``m.room.third_party_invite``, that homesever MUST validate that the public
@@ -93,11 +94,12 @@ For example:
     When the third party user validates their identity, they are told about the
     invite, and ask their homeserver, H3, to join the room.
 
-    H3 validates that sign(``token``, ``public_key``) = ``signature``, and may check
-    ``key_validity_url``.
+    H3 validates that signature in the ``signed`` property of the
+    ``third_party_invite`` property of the ``content`` property of the  event,
+    and may check ``key_validity_url``.
 
-    H3 then asks H1 to join it to the room. H1 *must* validate that
-    sign(``token``, ``public_key``) = ``signature`` *and* check ``key_validity_url``.
+    H3 then asks H1 to join it to the room. H1 *must* validate the ``signed``
+    property *and* check ``key_validity_url``.
 
     Having validated these things, H1 writes the join event to the room, and H3
     begins participating in the room. H2 *must* accept this event.

From a38e0862cf0b325c7ab57b4624d9c9c90a337388 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 16 Oct 2015 18:56:07 +0100
Subject: [PATCH 361/467] Fix required field names

---
 event-schemas/schema/v1/m.room.member | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 205c426e..c52d0d4c 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -54,14 +54,15 @@
                                     "type": "object",
                                     "description": "A single signature from the verifying server, in the format specified by the Signing Events section."
                                 }
-                            }
+                            },
+                            "required": ["mxid", "signatures", "token"]
                         },
                         "sender": {
                             "type": "string",
                             "description": "The matrix user ID of the user who send the invite which is being used."
                         }
                     },
-                    "required": ["token", "key_validity_url", "public_key", "signature", "sender"]
+                    "required": ["token", "key_validity_url", "public_key", "sender", "signed"]
                 }
             },
             "required": ["membership"]

From 29720cd152bd12f08b155895fddfc0ad65eaca7a Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 16 Oct 2015 19:00:36 +0100
Subject: [PATCH 362/467] Initial attempt at describing the event redaction
 algorithm

---
 specification/event_signing.rst | 60 +++++++++++++++++++++++++++++++--
 1 file changed, 57 insertions(+), 3 deletions(-)

diff --git a/specification/event_signing.rst b/specification/event_signing.rst
index bde58f0c..2f61566e 100644
--- a/specification/event_signing.rst
+++ b/specification/event_signing.rst
@@ -190,9 +190,63 @@ in the event JSON in a ``hash`` object under a ``sha256`` key.
             event_json_object["unsigned"] = unsigned
         return event_json_object
 
-Then all non-essential keys are stripped from the event object, and the
-resulting object which included the ``hash`` key is signed using the JSON
-signing algorithm
+The event is then stripped of all non-essential keys both at the top level and
+within the ``content`` object. Essential top-level keys given below; any
+top-level keys not in this list are removed.
+
+.. code::
+
+    auth_events
+    depth
+    event_id
+    hashes
+    membership
+    origin
+    origin_server_ts
+    prev_events
+    prev_state
+    room_id
+    sender
+    signatures
+    state_key
+    type
+
+A new ``content`` object is constructed for the resulting event that contains
+only the essential keys of the original event. If the original event lacked a
+``content`` object at all, a new blank one is created for it.
+
+The keys that are considered essential for the ``content`` object depend on the
+the ``type`` of the event. These are:
+
+.. code::
+
+    type is "m.room.aliases":
+      aliases
+
+    type is "m.room.create":
+      creator
+
+    type is "m.room.history_visibility":
+      history_visibility
+
+    type is "m.room.join_rules":
+      join_rule
+
+    type is "m.room.member":
+      membership
+
+    type is "m.room.power_levels":
+      ban
+      events
+      events_default
+      kick
+      redact
+      state_default
+      users
+      users_default
+
+The resulting stripped object with the new ``content`` object and the original
+``hashes`` key is then signed using the JSON signing algorithm.
 
 .. code:: python
 

From c70067dd1e7eb26c5f1e17ccab5f3a7fbc786ae0 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 19 Oct 2015 10:44:26 +0100
Subject: [PATCH 363/467] Simplify language

---
 specification/modules/third_party_invites.rst | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index ebca59cf..85538c31 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -36,8 +36,8 @@ A client asks a server to invite a user by their third party identifier.
 Server behaviour
 ----------------
 
-All homeservers MUST verify the signature in the ``signed`` property of the
-``third_party_invite`` property in the ``content`` the event.
+All homeservers MUST verify the signature in the event's
+``content.third_party_invite.signed`` object.
 
 If a client of the current homeserver is joining by an
 ``m.room.third_party_invite``, that homesever MUST validate that the public
@@ -94,9 +94,8 @@ For example:
     When the third party user validates their identity, they are told about the
     invite, and ask their homeserver, H3, to join the room.
 
-    H3 validates that signature in the ``signed`` property of the
-    ``third_party_invite`` property of the ``content`` property of the  event,
-    and may check ``key_validity_url``.
+    H3 validates the signature in the event's
+    ``content.third_party_invite.signed`` object.
 
     H3 then asks H1 to join it to the room. H1 *must* validate the ``signed``
     property *and* check ``key_validity_url``.

From ae90d15b3c6955779356466a6edb6e7e6d48c105 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 19 Oct 2015 10:46:07 +0100
Subject: [PATCH 364/467] Add m.room.avatar

---
 event-schemas/examples/v1/m.room.avatar     | 18 ++++++
 event-schemas/schema/v1/m.room.avatar       | 64 +++++++++++++++++++++
 specification/modules/instant_messaging.rst |  2 +
 3 files changed, 84 insertions(+)
 create mode 100644 event-schemas/examples/v1/m.room.avatar
 create mode 100644 event-schemas/schema/v1/m.room.avatar

diff --git a/event-schemas/examples/v1/m.room.avatar b/event-schemas/examples/v1/m.room.avatar
new file mode 100644
index 00000000..9fb1189c
--- /dev/null
+++ b/event-schemas/examples/v1/m.room.avatar
@@ -0,0 +1,18 @@
+{
+  "age": 242352,
+  "content": {
+    "info": {
+        "h": 398,
+        "w": 394,
+        "mimetype": "image/jpeg",
+        "size": 31037
+    },
+    "url": "mxc://localhost/JWEIFJgwEIhweiWJE"
+  },
+  "origin_server_ts": 1431961217939,
+  "event_id": "$WLGTSEFSEF:localhost",
+  "type": "m.room.avatar",
+  "state_key": "",
+  "room_id": "!Cuyf34gef24t:localhost",
+  "user_id": "@example:localhost"
+}
diff --git a/event-schemas/schema/v1/m.room.avatar b/event-schemas/schema/v1/m.room.avatar
new file mode 100644
index 00000000..276c975c
--- /dev/null
+++ b/event-schemas/schema/v1/m.room.avatar
@@ -0,0 +1,64 @@
+{
+    "title": "RoomAvatar",
+    "description": "A picture that is associated with the room. This can be displayed alongside the room information.",
+    "type": "object",
+    "allOf": [{
+        "$ref": "core-event-schema/state_event.json"
+    }],
+    "properties": {
+        "content": {
+            "type": "object",
+            "properties": {
+                "url": {
+                    "type": "string",
+                    "description": "The URL to the image."
+                },
+                "thumbnail_url": {
+                    "type": "string",
+                    "description": "The URL to the thumbnail of the image."
+                },
+                "thumbnail_info": {
+                    "type": "object",
+                    "title": "ImageInfo",
+                    "description": "Metadata about the image referred to in ``thumbnail_url``.",
+                    "allOf": [{
+                        "$ref": "core-event-schema/msgtype_infos/image_info.json"
+                    }]
+                },
+                "info": {
+                    "type": "object",
+                    "title": "ImageInfo",
+                    "description": "Metadata about the image referred to in ``url``.",
+                    "properties": {
+                        "size": {
+                            "type": "integer",
+                            "description": "Size of the image in bytes."
+                        },
+                        "w": {
+                            "type": "integer",
+                            "description": "The width of the image in pixels."
+                        },
+                        "h": {
+                            "type": "integer",
+                            "description": "The height of the image in pixels."
+                        },
+                        "mimetype": {
+                            "type": "string",
+                            "description": "The mimetype of the image, e.g. ``image/jpeg``."
+                        }
+                    }
+                }
+            },
+            "required": ["url"]
+        },
+        "state_key": {
+            "type": "string",
+            "description": "A zero-length string.",
+            "pattern": "^$"
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.room.avatar"]
+        }
+    }
+}
diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 0b4e3e64..09fcb843 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -30,6 +30,8 @@ Usage of this event is discouraged for several reasons:
 
 {{m_room_topic_event}}
 
+{{m_room_avatar_event}}
+
 m.room.message msgtypes
 ~~~~~~~~~~~~~~~~~~~~~~~
 

From f51e0310668b32ec766f1d8a28076e4affe123c0 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 19 Oct 2015 12:36:10 +0100
Subject: [PATCH 365/467] Fix JSON

---
 event-schemas/schema/v1/m.room.member | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index c52d0d4c..e0ad673c 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -40,7 +40,7 @@
                         },
                         "signed": {
                             "type": "object",
-                            "title": "signed_third_party_invite",
+                            "title": "signed",
                             "properties": {
                                 "mxid": {
                                     "type": "string",
@@ -48,11 +48,12 @@
                                 },
                                 "token": {
                                     "type": "string",
-                                    "description": "The token property of the containing third_party_invite object.",
+                                    "description": "The token property of the containing third_party_invite object."
                                 },
                                 "signatures": {
                                     "type": "object",
-                                    "description": "A single signature from the verifying server, in the format specified by the Signing Events section."
+                                    "description": "A single signature from the verifying server, in the format specified by the Signing Events section.",
+                                    "title": "Signatures"
                                 }
                             },
                             "required": ["mxid", "signatures", "token"]

From b2aae762fc5740a6e471745e85d1f0287ddabb8b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 19 Oct 2015 12:36:49 +0100
Subject: [PATCH 366/467] Give useful error if z-schema is missing

---
 event-schemas/check.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/event-schemas/check.sh b/event-schemas/check.sh
index fac36cf8..a6d03b5a 100755
--- a/event-schemas/check.sh
+++ b/event-schemas/check.sh
@@ -1,5 +1,11 @@
 #!/bin/bash -e
 # Runs z-schema over all of the schema files (looking for matching examples)
+
+if ! which z-schema; then
+  echo >&2 "Need to install z-schema; run: sudo npm install -g z-schema"
+  exit 1
+fi
+
 find schema/v1/m.* | while read line
 do
     split_path=(${line///// })

From bbd3f8072cccf3d029448f43290c2326cfb6276b Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 19 Oct 2015 13:28:44 +0100
Subject: [PATCH 367/467] Review comments

---
 event-schemas/schema/v1/m.room.member | 1 +
 templating/matrix_templates/units.py  | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 54ae9100..121f144a 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -67,6 +67,7 @@
                 "type": "object",
                 "title": "StrippedState",
                 "description": "A stripped down state event, with only the ``type``, ``state_key`` and ``content`` keys.",
+                "required": ["type", "state_key", "content"],
                 "properties": {
                     "type": {
                         "type": "string",
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 33b5e6b4..9a17dd5b 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -532,7 +532,8 @@ class MatrixUnits(Units):
                     Units.prop(json_schema, "properties/content")
                 )
 
-                # this is horrible
+                # This is horrible because we're special casing a key on m.room.member.
+                # We need to do this because we want to document a non-content object.
                 if schema["type"] == "m.room.member":
                     invite_room_state = get_json_schema_object_fields(
                         json_schema["properties"]["invite_room_state"]["items"]

From f450fc1db6adb9cb76925e90aff8f3782081bb07 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 19 Oct 2015 13:32:43 +0100
Subject: [PATCH 368/467] Typo

---
 api/client-server/v2_alpha/registration.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml
index 9973d0b4..5222b3f3 100644
--- a/api/client-server/v2_alpha/registration.yaml
+++ b/api/client-server/v2_alpha/registration.yaml
@@ -86,7 +86,7 @@ paths:
               * ``M_EXCLUSIVE`` : The desired user ID is in the exclusive namespace
                 claimed by an application service.
 
-            This errors may be returned at any stage of the registration process,
+            These errors may be returned at any stage of the registration process,
             including after authentication if the requested user ID was registered
             whilst the client was performing authentication.
           examples:

From 111ca99519a32ba0412aed423517774250184567 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 19 Oct 2015 13:46:30 +0100
Subject: [PATCH 369/467] Clarify that this doesn't include events that
 occurred after you left the room.

---
 specification/modules/search.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 03d2475f..6a545aa0 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -5,7 +5,8 @@ Server Side Search
 
 The search API allows clients to perform full text search across events in all
 rooms that the user has been in, including those that they have left. Only
-events that the user is allowed to see will be searched.
+events that the user is allowed to see will be searched, e.g. it won't include
+evnets in rooms that happened after you left.
 
 Client behaviour
 ----------------

From bfa9937bdc1c7025cb5265da3bf5d22b3a78d152 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 19 Oct 2015 17:28:31 +0100
Subject: [PATCH 370/467] Minor wording updates

---
 specification/event_signing.rst | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/specification/event_signing.rst b/specification/event_signing.rst
index 2f61566e..9f33c796 100644
--- a/specification/event_signing.rst
+++ b/specification/event_signing.rst
@@ -191,8 +191,8 @@ in the event JSON in a ``hash`` object under a ``sha256`` key.
         return event_json_object
 
 The event is then stripped of all non-essential keys both at the top level and
-within the ``content`` object. Essential top-level keys given below; any
-top-level keys not in this list are removed.
+within the ``content`` object. Any top-level keys not in he following list MUST
+be removed:
 
 .. code::
 
@@ -212,8 +212,9 @@ top-level keys not in this list are removed.
     type
 
 A new ``content`` object is constructed for the resulting event that contains
-only the essential keys of the original event. If the original event lacked a
-``content`` object at all, a new blank one is created for it.
+only the essential keys of the original ``content`` object. If the original
+event lacked a ``content`` object at all, a new empty JSON object is created
+for it.
 
 The keys that are considered essential for the ``content`` object depend on the
 the ``type`` of the event. These are:
@@ -246,7 +247,7 @@ the ``type`` of the event. These are:
       users_default
 
 The resulting stripped object with the new ``content`` object and the original
-``hashes`` key is then signed using the JSON signing algorithm.
+``hashes`` key is then signed using the JSON signing algorithm outlined below:
 
 .. code:: python
 

From 9252b3e1a29c4c105aaa169e4c03a2ff6ac3a1d8 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 19 Oct 2015 17:44:09 +0100
Subject: [PATCH 371/467] Fix typo s/he/the/

---
 specification/event_signing.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/event_signing.rst b/specification/event_signing.rst
index 9f33c796..3ad3d8a7 100644
--- a/specification/event_signing.rst
+++ b/specification/event_signing.rst
@@ -191,8 +191,8 @@ in the event JSON in a ``hash`` object under a ``sha256`` key.
         return event_json_object
 
 The event is then stripped of all non-essential keys both at the top level and
-within the ``content`` object. Any top-level keys not in he following list MUST
-be removed:
+within the ``content`` object. Any top-level keys not in the following list
+MUST be removed:
 
 .. code::
 

From 228acc59aa2e21fcd74aeec189493e5f1e6ccfc2 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 19 Oct 2015 18:26:52 +0100
Subject: [PATCH 372/467] Capture cryptographic test vectors for JSON or event
 signing from source code

---
 specification/appendices.rst | 157 +++++++++++++++++++++++++++++++++++
 1 file changed, 157 insertions(+)

diff --git a/specification/appendices.rst b/specification/appendices.rst
index c45aa0a5..53119006 100644
--- a/specification/appendices.rst
+++ b/specification/appendices.rst
@@ -128,3 +128,160 @@ Threat: Disclosure to Servers Within Chatroom
 An attacker could take control of a server within a chatroom to expose message
 contents or metadata for messages in that room.
 
+
+Cryptographic Test Vectors
+--------------------------
+
+To assist in the development of compatible implementations, the following test
+values may be useful for verifying the cryptograph event signing code.
+
+Signing Key
+~~~~~~~~~~~
+
+The following test vectors all use the 32-byte value given by the following
+Base64-encoded string as the seed for generating the ``ed25519`` signing key:
+
+.. code::
+
+    SIGNING_KEY_SEED = decode_base64(
+        "YJDBA9Xnr2sVqXD9Vj7XVUnmFZcZrlw8Md7kMW+3XA1"
+    )
+
+In each case, the server name and key ID are as follows:
+
+.. code::
+
+    SERVER_NAME = "domain"
+
+    KEY_ID = "ed25519:1"
+
+JSON Signing
+~~~~~~~~~~~~
+
+Given an empty JSON object:
+
+.. code:: json
+
+    {}
+
+The JSON signing algorithm should emit the following signed data:
+
+.. code:: json
+
+    {
+        "signatures": {
+            "domain": {
+                "ed25519:1": "K8280/U9SSy9IVtjBuVeLr+HpOB4BQFWbg+UZaADMtTdGYI7Geitb76LTrr5QV/7Xg4ahLwYGYZzuHGZKM5ZAQ"
+            }
+        }
+    }
+
+Given the following JSON object with data values in it:
+
+.. code:: json
+
+    {
+        "one": 1,
+        "two": "Two"
+    }
+
+The JSON signing algorithm should emit the following signed JSON:
+
+.. code:: json
+
+    {
+        "one": 1,
+        "signatures": {
+            "domain": {
+                "ed25519:1": "KqmLSbO39/Bzb0QIYE82zqLwsA+PDzYIpIRA2sRQ4sL53+sN6/fpNSoqE7BP7vBZhG6kYdD13EIMJpvhJI+6Bw"
+            }
+        },
+        "two": "Two"
+    }
+
+Event Signing
+~~~~~~~~~~~~~
+
+Given the following minimally-sized event:
+
+.. code:: json
+
+    {
+        "event_id": "$0:domain",
+        "origin": "domain",
+        "origin_server_ts": 1000000,
+        "signatures": {},
+        "type": "X",
+        "unsigned": {
+            "age_ts": 1000000
+        }
+    }
+
+The event signing algorithm should emit the following signed event:
+
+.. code:: json
+
+    {
+        "event_id": "$0:domain",
+        "hashes": {
+            "sha256": "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI"
+        },
+        "origin": "domain",
+        "origin_server_ts": 1000000,
+        "signatures": {
+            "domain": {
+                "ed25519:1": "2Wptgo4CwmLo/Y8B8qinxApKaCkBG2fjTWB7AbP5Uy+aIbygsSdLOFzvdDjww8zUVKCmI02eP9xtyJxc/cLiBA"
+            }
+        },
+        "type": "X",
+        "unsigned": {
+            "age_ts": 1000000
+        }
+    }
+
+Given the following event containing redactable content:
+
+.. code:: json
+
+    {
+        "content": {
+            "body": "Here is the message content",
+        },
+        "event_id": "$0:domain",
+        "origin": "domain",
+        "origin_server_ts": 1000000,
+        "type": "m.room.message",
+        "room_id": "!r:domain",
+        "sender": "@u:domain",
+        "signatures": {},
+        "unsigned": {
+            "age_ts": 1000000
+        }
+    }
+
+The event signing algorithm should emit the following signed event:
+
+.. code:: json
+
+    {
+        "content": {
+            "body": "Here is the message content",
+        },
+        "event_id": "$0:domain",
+        "hashes": {
+            "sha256": "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"
+        },
+        "origin": "domain",
+        "origin_server_ts": 1000000,
+        "type": "m.room.message",
+        "room_id": "!r:domain",
+        "sender": "@u:domain",
+        "signatures": {
+            "domain": {
+                "ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA"
+            }
+        },
+        "unsigned": {
+            "age_ts": 1000000
+        }
+    }

From e38e4788f24e4f98bcf59592417f9fa6d37a6c0e Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 19 Oct 2015 18:29:37 +0100
Subject: [PATCH 373/467] Spell 'cryptographic' correctly

---
 specification/appendices.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/appendices.rst b/specification/appendices.rst
index 53119006..28682776 100644
--- a/specification/appendices.rst
+++ b/specification/appendices.rst
@@ -133,7 +133,7 @@ Cryptographic Test Vectors
 --------------------------
 
 To assist in the development of compatible implementations, the following test
-values may be useful for verifying the cryptograph event signing code.
+values may be useful for verifying the cryptographic event signing code.
 
 Signing Key
 ~~~~~~~~~~~

From 74d42f01423acefdb88e3727d61530f371b108bc Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 20 Oct 2015 11:08:22 +0100
Subject: [PATCH 374/467] Allow rejecting invitations

---
 specification/client_server_api.rst | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index c25248f8..791c1f4a 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -905,6 +905,11 @@ room. There are several states in which a user may be, in relation to a room:
  - Joined (the user can send and receive events in the room)
  - Banned (the user is not allowed to join the room)
 
+There is an exception to the requirement that a user join a room before sending
+events to it: users may send an ``m.room.member`` event to a room with
+``content.membership`` set to ``leave`` to reject an invitation if they have
+currently been invited to a room but have not joined it.
+
 Some rooms require that users be invited to it before they can join; others
 allow anyone to join. Whether a given room is an "invite-only" room is
 determined by the room config key ``m.room.join_rules``. It can have one of the
@@ -931,8 +936,11 @@ Leaving rooms
 
 
 A user can leave a room to stop receiving events for that room. A user must
-have joined the room before they are eligible to leave the room. If the room is
-an "invite-only" room, they will need to be re-invited before they can re-join
+have been invited to or have joined the room before they are eligible to leave
+the room. Leaving a room to which the user has been invited rejects the invite.
+
+Whether or not they actually joined the room, if the room is
+an "invite-only" room they will need to be re-invited before they can re-join
 the room.  To leave a room, a request should be made to
 |/rooms/<room_id>/leave|_ with::
 

From bd539c72b414864cd2e759c8dcbcebfc00d0dae2 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 21 Oct 2015 09:52:44 +0100
Subject: [PATCH 375/467] More review comments

---
 api/client-server/v1/application_service.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v1/application_service.yaml b/api/client-server/v1/application_service.yaml
index a7400bd4..557d5a70 100644
--- a/api/client-server/v1/application_service.yaml
+++ b/api/client-server/v1/application_service.yaml
@@ -115,7 +115,7 @@ paths:
           examples:
             application/json: |-
               {
-                "errcode": "ORG.MATRIX.MYAPPSERVICE_UNAUTHORIZED"
+                "errcode": "COM.EXAMPLE.MYAPPSERVICE_UNAUTHORIZED"
               }
           schema:
             type: object
@@ -147,7 +147,7 @@ paths:
         This endpoint is invoked by the homeserver on an application service to query
         the existence of a given user ID. The homeserver will only query user IDs
         inside the application service's ``users`` namespace. The homeserver will
-        send this request when it receives an *event* for an unknown user ID in
+        send this request when it receives an event for an unknown user ID in
         the application service's namespace.
       parameters:
         - in: path

From 4c3e1b9ed37d69b488d7c215b2ca287b6f0a2f94 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Wed, 21 Oct 2015 09:53:25 +0100
Subject: [PATCH 376/467] Find replace fail

---
 api/client-server/v1/application_service.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v1/application_service.yaml b/api/client-server/v1/application_service.yaml
index 557d5a70..1a5ec838 100644
--- a/api/client-server/v1/application_service.yaml
+++ b/api/client-server/v1/application_service.yaml
@@ -136,7 +136,7 @@ paths:
           examples:
             application/json: |-
               {
-                "errcode": "ORG.MATRIX.MYAPPSERVICE_NOT_FOUND"
+                "errcode": "COM.EXAMPLE.MYAPPSERVICE_NOT_FOUND"
               }
           schema:
             type: object
@@ -173,7 +173,7 @@ paths:
           examples:
             application/json: |-
               {
-                "errcode": "ORG.MATRIX.MYAPPSERVICE_UNAUTHORIZED"
+                "errcode": "COM.EXAMPLE.MYAPPSERVICE_UNAUTHORIZED"
               }
           schema:
             type: object
@@ -194,7 +194,7 @@ paths:
           examples:
             application/json: |-
               {
-                "errcode": "ORG.MATRIX.MYAPPSERVICE_NOT_FOUND"
+                "errcode": "COM.EXAMPLE.MYAPPSERVICE_NOT_FOUND"
               }
           schema:
             type: object

From 82d4ea199f507e7dbdc9a960d50ab831e69f26b9 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 22 Oct 2015 16:09:01 +0100
Subject: [PATCH 377/467] Trust Leo

---
 scripts/speculator/main.go | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index dab624df..e6992d4a 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -365,6 +365,7 @@ func main() {
 		"Kegsay":        true,
 		"NegativeMjark": true,
 		"richvdh":       true,
+		"leonerd":       true,
 	}
 	rand.Seed(time.Now().Unix())
 	masterCloneDir, err := gitClone(matrixDocCloneURL, false)

From 4a558ad63b0724a3b4f715df864cf698fbbcb1b3 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Fri, 23 Oct 2015 10:19:25 +0100
Subject: [PATCH 378/467] Typo

---
 specification/modules/search.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/modules/search.rst b/specification/modules/search.rst
index 6a545aa0..089953d2 100644
--- a/specification/modules/search.rst
+++ b/specification/modules/search.rst
@@ -6,7 +6,7 @@ Server Side Search
 The search API allows clients to perform full text search across events in all
 rooms that the user has been in, including those that they have left. Only
 events that the user is allowed to see will be searched, e.g. it won't include
-evnets in rooms that happened after you left.
+events in rooms that happened after you left.
 
 Client behaviour
 ----------------

From e3e8026025c3dbc02092958f957f1731b14f2853 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 23 Oct 2015 10:51:31 +0100
Subject: [PATCH 379/467] Fix list formatting so that we aren't including
 everything in blockquotes

---
 api/client-server/v1/membership.yaml         | 20 ++---
 api/client-server/v2_alpha/registration.yaml |  6 +-
 specification/client_server_api.rst          | 48 ++++++------
 specification/events.rst                     | 10 +--
 specification/modules/content_repo.rst       | 25 +++----
 specification/modules/presence.rst           | 32 ++++----
 specification/modules/push.rst               | 78 ++++++++++----------
 specification/modules/voip_events.rst        | 20 ++---
 8 files changed, 120 insertions(+), 119 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index efc82578..fb0737e5 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -52,8 +52,8 @@ paths:
         403:
           description: |-
             You do not have permission to join the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejection are:
-              - The room is invite-only and the user was not invited.
-              - The user has been banned from the room.
+            - The room is invite-only and the user was not invited.
+            - The user has been banned from the room.
           examples:
             application/json: |-
               {"errcode": "M_FORBIDDEN", "error": "You are not invited to this room."}
@@ -119,10 +119,10 @@ paths:
         403:
           description: |-
             You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
-              - The invitee has been banned from the room.
-              - The invitee is already a member of the room.
-              - The inviter is not currently in the room.
-              - The inviter's power level is insufficient to invite users to the room.
+            - The invitee has been banned from the room.
+            - The invitee is already a member of the room.
+            - The inviter is not currently in the room.
+            - The inviter's power level is insufficient to invite users to the room.
           examples:
             application/json: |-
               {"errcode": "M_FORBIDDEN", "error": "@cheeky_monkey:matrix.org is banned from the room"}
@@ -164,13 +164,13 @@ paths:
         user ID, the identity server will give the user a list of pending
         invitations, each containing:
 
-          - The room ID to which they were invited
+        - The room ID to which they were invited
 
-          - The token given to the homeserver
+        - The token given to the homeserver
 
-          - A signature of the token, signed with the identity server's private key
+        - A signature of the token, signed with the identity server's private key
 
-          - The matrix user ID who invited them to the room
+        - The matrix user ID who invited them to the room
 
         If a token is requested from the identity server, the home server will
         append a ``m.room.third_party_invite`` event to the room.
diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml
index 5222b3f3..b531c6cf 100644
--- a/api/client-server/v2_alpha/registration.yaml
+++ b/api/client-server/v2_alpha/registration.yaml
@@ -82,9 +82,9 @@ paths:
           description: |-
             Part of the request was invalid. This may include one of the following error codes:
 
-              * ``M_USER_IN_USE`` : The desired user ID is already taken.
-              * ``M_EXCLUSIVE`` : The desired user ID is in the exclusive namespace
-                claimed by an application service.
+            * ``M_USER_IN_USE`` : The desired user ID is already taken.
+            * ``M_EXCLUSIVE`` : The desired user ID is in the exclusive namespace
+              claimed by an application service.
 
             These errors may be returned at any stage of the registration process,
             including after authentication if the requested user ID was registered
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 791c1f4a..fcbc58a9 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -787,23 +787,23 @@ it to be redacted, which may include a reason.
 Upon receipt of a redaction event, the server should strip off any keys not in
 the following list:
 
- - ``event_id``
- - ``type``
- - ``room_id``
- - ``user_id``
- - ``state_key``
- - ``prev_state``
- - ``content``
+- ``event_id``
+- ``type``
+- ``room_id``
+- ``user_id``
+- ``state_key``
+- ``prev_state``
+- ``content``
 
 The content object should also be stripped of all keys, unless it is one of
 one of the following event types:
 
- - ``m.room.member`` allows key ``membership``
- - ``m.room.create`` allows key ``creator``
- - ``m.room.join_rules`` allows key ``join_rule``
- - ``m.room.power_levels`` allows keys ``ban``, ``events``, ``events_default``,
+- ``m.room.member`` allows key ``membership``
+- ``m.room.create`` allows key ``creator``
+- ``m.room.join_rules`` allows key ``join_rule``
+- ``m.room.power_levels`` allows keys ``ban``, ``events``, ``events_default``,
    ``kick``, ``redact``, ``state_default``, ``users``, ``users_default``.
- - ``m.room.aliases`` allows key ``aliases``
+- ``m.room.aliases`` allows key ``aliases``
 
 .. TODO
   Need to update m.room.power_levels to reflect new power levels formatting
@@ -823,9 +823,9 @@ which serves as the root of the event graph for this room. This event also has a
 generate several other events in order to manage permissions in this room. This
 includes:
 
- - ``m.room.power_levels`` : Sets the power levels of users and required power
+- ``m.room.power_levels`` : Sets the power levels of users and required power
    levels for various actions within the room such as sending events.
- - ``m.room.join_rules`` : Whether the room is "invite-only" or not.
+- ``m.room.join_rules`` : Whether the room is "invite-only" or not.
 
 See `Room Events`_ for more information on these events. To create a room, a
 client has to use the the following API.
@@ -899,11 +899,11 @@ Joining rooms
 Users need to be a member of a room in order to send and receive events in that
 room. There are several states in which a user may be, in relation to a room:
 
- - Unrelated (the user cannot send or receive events in the room)
- - Invited (the user has been invited to participate in the room, but is not
-   yet participating)
- - Joined (the user can send and receive events in the room)
- - Banned (the user is not allowed to join the room)
+- Unrelated (the user cannot send or receive events in the room)
+- Invited (the user has been invited to participate in the room, but is not
+  yet participating)
+- Joined (the user can send and receive events in the room)
+- Banned (the user is not allowed to join the room)
 
 There is an exception to the requirement that a user join a room before sending
 events to it: users may send an ``m.room.member`` event to a room with
@@ -1095,11 +1095,11 @@ many places of a client's display, changes to these fields cause an automatic
 propagation event to occur, informing likely-interested parties of the new
 values. This change is conveyed using two separate mechanisms:
 
- - a ``m.room.member`` event is sent to every room the user is a member of,
-   to update the ``displayname`` and ``avatar_url``.
- - a ``m.presence`` presence status update is sent, again containing the new
-   values of the ``displayname`` and ``avatar_url`` keys, in addition to the
-   required ``presence`` key containing the current presence state of the user.
+- a ``m.room.member`` event is sent to every room the user is a member of,
+  to update the ``displayname`` and ``avatar_url``.
+- a ``m.presence`` presence status update is sent, again containing the new
+  values of the ``displayname`` and ``avatar_url`` keys, in addition to the
+  required ``presence`` key containing the current presence state of the user.
 
 Both of these should be done automatically by the home server when a user
 successfully changes their display name or avatar URL fields.
diff --git a/specification/events.rst b/specification/events.rst
index 0eb4d933..7d64882b 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -18,11 +18,11 @@ Size limits
 The total size of any event MUST NOT exceed 65 KB. There are additional
 restrictions on sizes per key:
 
- - ``user_id`` MUST NOT exceed 255 bytes (including domain).
- - ``room_id`` MUST NOT exceed 255 bytes.
- - ``state_key`` MUST NOT exceed 255 bytes.
- - ``type`` MUST NOT exceed 255 bytes.
- - ``event_id`` MUST NOT exceed 255 bytes.
+- ``user_id`` MUST NOT exceed 255 bytes (including domain).
+- ``room_id`` MUST NOT exceed 255 bytes.
+- ``state_key`` MUST NOT exceed 255 bytes.
+- ``type`` MUST NOT exceed 255 bytes.
+- ``event_id`` MUST NOT exceed 255 bytes.
 
 Some event types have additional size restrictions which are specified in
 the description of the event. Additional keys have no limit other than that
diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index 52937ead..29dae059 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -64,21 +64,20 @@ UTF-8 encoded traversals, etc).
 
 Homeservers have additional content-specific concerns:
 
- - Clients may try to upload very large files. Homeservers should not store files
-   that are too large and should not serve them to clients.
+- Clients may try to upload very large files. Homeservers should not store files
+  that are too large and should not serve them to clients.
 
- - Clients may try to upload very large images. Homeservers should not attempt to
-   generate thumbnails for images that are too large.
+- Clients may try to upload very large images. Homeservers should not attempt to
+  generate thumbnails for images that are too large.
 
- - Remote homeservers may host very large files or images. Homeservers should not
-   proxy or thumbnail large files or images from remote homeservers.
+- Remote homeservers may host very large files or images. Homeservers should not
+  proxy or thumbnail large files or images from remote homeservers.
 
- - Clients may try to upload a large number of files. Homeservers should limit the
-   number and total size of media that can be uploaded by clients.
+- Clients may try to upload a large number of files. Homeservers should limit the
+  number and total size of media that can be uploaded by clients.
 
- - Clients may try to access a large number of remote files through a homeserver.
-   Homeservers should restrict the number and size of remote files that it caches.
-
- - Clients or remote homeservers may try to upload malicious files targeting
-   vulnerabilities in either the homeserver thumbnailing or the client decoders.
+- Clients may try to access a large number of remote files through a homeserver.
+  Homeservers should restrict the number and size of remote files that it caches.
 
+- Clients or remote homeservers may try to upload malicious files targeting
+  vulnerabilities in either the homeserver thumbnailing or the client decoders.
diff --git a/specification/modules/presence.rst b/specification/modules/presence.rst
index 79151c4f..3602d105 100644
--- a/specification/modules/presence.rst
+++ b/specification/modules/presence.rst
@@ -5,10 +5,10 @@ Presence
 
 Each user has the concept of presence information. This encodes:
 
- * Whether the user is currently online
- * How recently the user was last active (as seen by the server)
- * Whether a given client considers the user to be currently idle
- * Arbitrary information about the user's current status (e.g. "in a meeting").
+* Whether the user is currently online
+* How recently the user was last active (as seen by the server)
+* Whether a given client considers the user to be currently idle
+* Arbitrary information about the user's current status (e.g. "in a meeting").
 
 This information is collated from both per-device (``online``, ``idle``,
 ``last_active``) and per-user (status) data, aggregated by the user's homeserver
@@ -24,14 +24,14 @@ who must accept the invitation.
 User's presence state is represented by the ``presence`` key, which is an enum
 of one of the following:
 
-      - ``online`` : The default state when the user is connected to an event
-        stream.
-      - ``unavailable`` : The user is not reachable at this time e.g. they are
-        idle.
-      - ``offline`` : The user is not connected to an event stream or is
-        explicitly suppressing their profile information from being sent.
-      - ``free_for_chat`` : The user is generally willing to receive messages
-        moreso than default.
+- ``online`` : The default state when the user is connected to an event
+  stream.
+- ``unavailable`` : The user is not reachable at this time e.g. they are
+  idle.
+- ``offline`` : The user is not connected to an event stream or is
+  explicitly suppressing their profile information from being sent.
+- ``free_for_chat`` : The user is generally willing to receive messages
+  moreso than default.
 
 Events
 ------
@@ -94,10 +94,10 @@ pro-active event from the user. A pro-active event may be sending a message to a
 room or changing presence state to a higher level of availability. Levels of
 availability are defined from low to high as follows:
 
-      - ``offline``
-      - ``unavailable``
-      - ``online``
-      - ``free_for_chat``
+- ``offline``
+- ``unavailable``
+- ``online``
+- ``free_for_chat``
 
 Based on this list, changing state from ``unavailable`` to ``online`` counts as
 a pro-active event, whereas ``online`` to ``unavailable`` does not. This
diff --git a/specification/modules/push.rst b/specification/modules/push.rst
index 525e730d..893e9481 100644
--- a/specification/modules/push.rst
+++ b/specification/modules/push.rst
@@ -37,19 +37,19 @@ The above diagram shows the flow of push notifications being sent to a handset
 where push notifications are submitted via the handset vendor, such as Apple's
 APNS or Google's GCM. This happens as follows:
 
- 1. The client app signs in to a homeserver.
- 2. The client app registers with its vendor's Push Provider and
-    obtains a routing token of some kind.
- 3. The mobile app uses the Client/Server API to add a 'pusher', providing the
-    URL of a specific Push Gateway which is configured for that
-    application. It also provides the routing token it has acquired from the
-    Push Provider.
- 4. The homeserver starts sending HTTP requests to the Push Gateway using the
-    supplied URL. The Push Gateway relays this notification to
-    the Push Provider, passing the routing token along with any
-    necessary private credentials the provider requires to send push
-    notifications.
- 5. The Push Provider sends the notification to the device.
+1. The client app signs in to a homeserver.
+2. The client app registers with its vendor's Push Provider and
+   obtains a routing token of some kind.
+3. The mobile app uses the Client/Server API to add a 'pusher', providing the
+   URL of a specific Push Gateway which is configured for that
+   application. It also provides the routing token it has acquired from the
+   Push Provider.
+4. The homeserver starts sending HTTP requests to the Push Gateway using the
+   supplied URL. The Push Gateway relays this notification to
+   the Push Provider, passing the routing token along with any
+   necessary private credentials the provider requires to send push
+   notifications.
+5. The Push Provider sends the notification to the device.
 
 Definitions for terms used in this section are below:
 
@@ -127,16 +127,16 @@ affect delivery of notifications via pushers with a matching ``profile_tag``.
 All device-specific rules have a higher priority than global rules. This means
 that the full list of rule kinds, in descending priority order, is as follows:
 
- * Device-specific Override
- * Device-specific Content
- * Device-specific Room
- * Device-specific Sender
- * Device-specific Underride
- * Global Override
- * Global Content
- * Global Room
- * Global Sender
- * Global Underride
+* Device-specific Override
+* Device-specific Content
+* Device-specific Room
+* Device-specific Sender
+* Device-specific Underride
+* Global Override
+* Global Content
+* Global Room
+* Global Sender
+* Global Underride
 
 Rules with the same ``kind`` can specify an ordering priority. This determines
 which rule is selected in the event of multiple matches. For example, a rule
@@ -315,16 +315,17 @@ rule determines its behaviour. The following conditions are defined:
 
 ``event_match``
   This is a glob pattern match on a field of the event. Parameters:
-   * ``key``: The dot-separated field of the event to match, e.g. ``content.body``
-   * ``pattern``: The glob-style pattern to match against. Patterns with no
-     special glob characters should be treated as having asterisks
-     prepended and appended when testing the condition.
+
+  * ``key``: The dot-separated field of the event to match, e.g. ``content.body``
+  * ``pattern``: The glob-style pattern to match against. Patterns with no
+    special glob characters should be treated as having asterisks
+    prepended and appended when testing the condition.
 
 ``profile_tag``
   Matches the ``profile_tag`` of the device that the notification would be
   delivered to. Parameters:
 
-   * ``profile_tag``: The profile_tag to match with.
+  * ``profile_tag``: The profile_tag to match with.
 
 ``contains_display_name``
   This matches unencrypted messages where ``content.body`` contains the owner's
@@ -334,10 +335,11 @@ rule determines its behaviour. The following conditions are defined:
 
 ``room_member_count``
   This matches the current number of members in the room. Parameters:
-   * ``is``: A decimal integer optionally prefixed by one of, ``==``, ``<``,
-     ``>``, ``>=`` or ``<=``. A prefix of ``<`` matches rooms where the member
-     count is strictly less than the given number and so forth. If no prefix is
-     present, this parameter defaults to ``==``.
+
+  * ``is``: A decimal integer optionally prefixed by one of, ``==``, ``<``,
+    ``>``, ``>=`` or ``<=``. A prefix of ``<`` matches rooms where the member
+    count is strictly less than the given number and so forth. If no prefix is
+    present, this parameter defaults to ``==``.
 
 Push Rules: API
 ~~~~~~~~~~~~~~~
@@ -418,12 +420,12 @@ client app and its' push gateway to agree on. As APNS requires that the sender
 has a private key owned by the app developer, each app must have its own push
 gateway. It is recommended that:
 
- * The APNS token be base64 encoded and used as the pushkey.
- * A different app_id be used for apps on the production and sandbox
-   APS environments.
- * APNS push gateways do not attempt to wait for errors from the APNS
-   gateway before returning and instead to store failures and return
-   'rejected' responses next time that pushkey is used.
+* The APNS token be base64 encoded and used as the pushkey.
+* A different app_id be used for apps on the production and sandbox
+  APS environments.
+* APNS push gateways do not attempt to wait for errors from the APNS
+  gateway before returning and instead to store failures and return
+  'rejected' responses next time that pushkey is used.
 
 Security considerations
 -----------------------
diff --git a/specification/modules/voip_events.rst b/specification/modules/voip_events.rst
index a7b02538..6945a09e 100644
--- a/specification/modules/voip_events.rst
+++ b/specification/modules/voip_events.rst
@@ -62,19 +62,19 @@ As calls are "placed" to rooms rather than users, the glare resolution algorithm
 outlined below is only considered for calls which are to the same room. The
 algorithm is as follows:
 
- - If an ``m.call.invite`` to a room is received whilst the client is
-   **preparing to send** an ``m.call.invite`` to the same room:
+- If an ``m.call.invite`` to a room is received whilst the client is
+  **preparing to send** an ``m.call.invite`` to the same room:
 
-    * the client should cancel its outgoing call and instead
-      automatically accept the incoming call on behalf of the user.
+  * the client should cancel its outgoing call and instead
+    automatically accept the incoming call on behalf of the user.
 
- - If an ``m.call.invite`` to a room is received **after the client has sent**
-   an ``m.call.invite`` to the same room and is waiting for a response:
+- If an ``m.call.invite`` to a room is received **after the client has sent**
+  an ``m.call.invite`` to the same room and is waiting for a response:
 
-    * the client should perform a lexicographical comparison of the call IDs of
-      the two calls and use the *lesser* of the two calls, aborting the
-      greater. If the incoming call is the lesser, the client should accept
-      this call on behalf of the user.
+  * the client should perform a lexicographical comparison of the call IDs of
+    the two calls and use the *lesser* of the two calls, aborting the
+    greater. If the incoming call is the lesser, the client should accept
+    this call on behalf of the user.
 
 
 The call setup should appear seamless to the user as if they had simply placed

From cc146209329e7369ecf164bf4108ed4863250e01 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 23 Oct 2015 11:36:07 +0100
Subject: [PATCH 380/467] Add a newline before the list

---
 api/client-server/v1/membership.yaml | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index fb0737e5..13bfb6bb 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -119,6 +119,7 @@ paths:
         403:
           description: |-
             You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
+
             - The invitee has been banned from the room.
             - The invitee is already a member of the room.
             - The inviter is not currently in the room.
@@ -221,10 +222,11 @@ paths:
         403:
           description: |-
             You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
-              - The invitee has been banned from the room.
-              - The invitee is already a member of the room.
-              - The inviter is not currently in the room.
-              - The inviter's power level is insufficient to invite users to the room.
+
+            - The invitee has been banned from the room.
+            - The invitee is already a member of the room.
+            - The inviter is not currently in the room.
+            - The inviter's power level is insufficient to invite users to the room.
           examples:
             application/json: |-
               {"errcode": "M_FORBIDDEN", "error": "@cheeky_monkey:matrix.org is banned from the room"}

From 01064369a450210ea7780fb1c4b4819125aaa160 Mon Sep 17 00:00:00 2001
From: David Baker <dave@matrix.org>
Date: Fri, 23 Oct 2015 11:57:15 +0100
Subject: [PATCH 381/467] Add some clarification on the difference between
 'scale' and 'crop' resize methods

---
 specification/modules/content_repo.rst | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst
index 52937ead..fd797b2c 100644
--- a/specification/modules/content_repo.rst
+++ b/specification/modules/content_repo.rst
@@ -36,6 +36,10 @@ width and height are close to the requested size and the aspect matches
 the requested size. The client should scale the image if it needs to fit
 within a given rectangle.
 
+In summary:
+ * "scale" maintains the original aspect ratio of the image
+ * "crop" provides an image in the aspect ratio of the sizes given in the request
+
 Server behaviour
 ----------------
 

From df443c10ac973686a57f302629d5b8151f4aa324 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 23 Oct 2015 11:58:06 +0100
Subject: [PATCH 382/467] Move css into a css subdirectory.

Add css to put a bar next to blockquotes.
---
 scripts/{ => css}/basic.css         | 0
 scripts/css/blockquote.css          | 5 +++++
 scripts/{ => css}/codehighlight.css | 0
 scripts/{ => css}/nature.css        | 0
 scripts/gendoc.py                   | 2 +-
 5 files changed, 6 insertions(+), 1 deletion(-)
 rename scripts/{ => css}/basic.css (100%)
 create mode 100644 scripts/css/blockquote.css
 rename scripts/{ => css}/codehighlight.css (100%)
 rename scripts/{ => css}/nature.css (100%)

diff --git a/scripts/basic.css b/scripts/css/basic.css
similarity index 100%
rename from scripts/basic.css
rename to scripts/css/basic.css
diff --git a/scripts/css/blockquote.css b/scripts/css/blockquote.css
new file mode 100644
index 00000000..151d3bce
--- /dev/null
+++ b/scripts/css/blockquote.css
@@ -0,0 +1,5 @@
+blockquote {
+    margin: 20px 0 30px;
+    border-left: 5px solid;
+    padding-left: 20px;
+}
diff --git a/scripts/codehighlight.css b/scripts/css/codehighlight.css
similarity index 100%
rename from scripts/codehighlight.css
rename to scripts/css/codehighlight.css
diff --git a/scripts/nature.css b/scripts/css/nature.css
similarity index 100%
rename from scripts/nature.css
rename to scripts/css/nature.css
diff --git a/scripts/gendoc.py b/scripts/gendoc.py
index 74a6c1da..ed36a5a7 100755
--- a/scripts/gendoc.py
+++ b/scripts/gendoc.py
@@ -15,7 +15,7 @@ import yaml
 os.chdir(os.path.dirname(os.path.abspath(__file__)))
 
 stylesheets = {
-    "stylesheet_path": ["basic.css", "nature.css", "codehighlight.css"]
+    "stylesheet_path": glob.glob("css/*.css"),
 }
 
 VERBOSE = False

From d18d406c41e0658b376555d666dc3690749fab5e Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Fri, 23 Oct 2015 12:49:17 +0100
Subject: [PATCH 383/467] Add the room send and state APIs to the spec

---
 api/client-server/v1/room_send.yaml  | 125 +++++++++++++++++++++++++++
 api/client-server/v1/room_state.yaml |  78 +++++++++++++++++
 specification/client_server_api.rst  |  64 +++++---------
 3 files changed, 227 insertions(+), 40 deletions(-)
 create mode 100644 api/client-server/v1/room_send.yaml
 create mode 100644 api/client-server/v1/room_state.yaml

diff --git a/api/client-server/v1/room_send.yaml b/api/client-server/v1/room_send.yaml
new file mode 100644
index 00000000..9c9273df
--- /dev/null
+++ b/api/client-server/v1/room_send.yaml
@@ -0,0 +1,125 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 message event send API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/send/{eventType}/{txnId}":
+    put:
+      summary: Send a message event to the given room.
+      description: |-
+        This endpoint is used to send a message event to a room. Message events
+        allow access to historical events and pagination, making them suited
+        for "once-off" activity in a room.
+
+        The body of the request should be the content object of the event; the
+        fields in this object will vary depending on the type of event. See
+        `Room Events`_ for the m. event specification.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to send the event to.
+          required: true
+          x-example: "!636q39766251:example.com"
+        - in: path
+          type: string
+          name: eventType
+          description: The type of event to send.
+          required: true
+          x-example: "m.room.message"
+        - in: path
+          name: txnId
+          type: string
+          description: |-
+            The transaction ID for this event. Clients should generate a
+            unique ID; it will be used by the server to ensure idempotency of requests.
+          required: true
+          x-example: "35"
+        - in: body
+          name: body
+          schema:
+            type: object
+            example: |-
+              {
+                "msgtype": "m.text",
+                "body": "hello"
+              }
+      responses:
+        200:
+          description: "An ID for the sent event."
+          examples:
+            application/json: |-
+              {
+                  "event_id": "YUwRidLecu"
+              }
+          schema:
+            type: object
+            properties:
+              event_id:
+                type: string
+                description: |-
+                  A unique identifier for the event.
+  "/rooms/{roomId}/send/{eventType}":
+    post:
+      summary: Send a message event to the given room.
+      description: |-
+        This endpoint can be used to send a message event to a room; however
+        the lack of a transaction ID means that it is possible to cause message
+        duplication if events are resent on error, so it is preferable to use
+        `PUT /_matrix/client/api/v1/rooms/{roomId}/send/{eventType}/{txnId}`_.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to send the event to.
+          required: true
+          x-example: "!636q39766251:example.com"
+        - in: path
+          type: string
+          name: eventType
+          description: The type of event to send.
+          required: true
+          x-example: "m.room.message"
+        - in: body
+          name: body
+          schema:
+            type: object
+            example: |-
+              {
+                "msgtype": "m.text",
+                "body": "hello"
+              }
+      responses:
+        200:
+          description: "An ID for the sent event."
+          examples:
+            application/json: |-
+              {
+                  "event_id": "YUwRidLecu"
+              }
+          schema:
+            type: object
+            properties:
+              event_id:
+                type: string
+                description: |-
+                  A unique identifier for the event.
diff --git a/api/client-server/v1/room_state.yaml b/api/client-server/v1/room_state.yaml
new file mode 100644
index 00000000..0c0e76e6
--- /dev/null
+++ b/api/client-server/v1/room_state.yaml
@@ -0,0 +1,78 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 state event send API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/state/{eventType}/{stateKey}":
+    put:
+      summary: Send a message event to the given room.
+      description: |
+        State events can be sent using this endpoint.  These events will be
+        overwritten if ``<room id>``, ``<event type>`` and ``<state key>`` all
+        match.  If the state event has no ``state_key``, it can be omitted from
+        the path. These requests **cannot use transaction IDs** like other
+        ``PUT`` paths because they cannot be differentiated from the
+        ``state_key``. Furthermore, ``POST`` is unsupported on state paths.
+
+        The body of the request should be the content object of the event; the
+        fields in this object will vary depending on the type of event. See
+        `Room Events`_ for the ``m.`` event specification.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room to set the state in
+          required: true
+          x-example: "!636q39766251:example.com"
+        - in: path
+          type: string
+          name: eventType
+          description: The type of event to send.
+          required: true
+          x-example: "m.room.name"
+        - in: path
+          type: string
+          name: stateKey
+          description: The state_key for the state to send. Defaults to the empty string.
+          required: false
+          x-example: ""
+        - in: body
+          name: body
+          schema:
+            type: object
+            example: |
+              {
+                  "name": "The room name"
+              }
+      responses:
+        200:
+          description: "An ID for the sent event."
+          examples:
+            application/json: |-
+              {
+                  "event_id": "YUwRidLecu"
+              }
+          schema:
+            type: object
+            properties:
+              event_id:
+                type: string
+                description: |-
+                  A unique identifier for the event.
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 791c1f4a..4df821e0 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -697,22 +697,32 @@ events are added, the event ``type`` key SHOULD follow the Java package naming
 convention, e.g. ``com.example.myapp.event``.  This ensures event types are
 suitably namespaced for each application and reduces the risk of clashes.
 
-State events
-++++++++++++
+Getting events for a room
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are several APIs provided to ``GET`` events for a room:
+
+{{rooms_http_api}}
+
+
+{{message_pagination_http_api}}
+
+
+Sending events to a room
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+{{room_state_http_api}}
+
+
+**Examples**
 
-State events can be sent by ``PUT`` ing to
-|/rooms/<room_id>/state/<event_type>/<state_key>|_.  These events will be
-overwritten if ``<room id>``, ``<event type>`` and ``<state key>`` all match.
-If the state event has no ``state_key``, it can be omitted from the path. These
-requests **cannot use transaction IDs** like other ``PUT`` paths because they
-cannot be differentiated from the ``state_key``. Furthermore, ``POST`` is
-unsupported on state paths. Valid requests look like::
+Valid requests look like::
 
-  PUT /rooms/!roomid:domain/state/m.example.event
-  { "key" : "without a state key" }
+    PUT /rooms/!roomid:domain/state/m.example.event
+    { "key" : "without a state key" }
 
-  PUT /rooms/!roomid:domain/state/m.another.example.event/foo
-  { "key" : "with 'foo' as the state key" }
+    PUT /rooms/!roomid:domain/state/m.another.example.event/foo
+    { "key" : "with 'foo' as the state key" }
 
 In contrast, these requests are invalid::
 
@@ -738,34 +748,8 @@ In some cases, there may be no need for a ``state_key``, so it can be omitted::
   PUT /rooms/!roomid:domain/state/m.room.bgd.color
   { "color": "red", "hex": "#ff0000" }
 
-See `Room Events`_ for the ``m.`` event specification.
+{{room_send_http_api}}
 
-Message events
-++++++++++++++
-
-Message events can be sent by sending a request to
-|/rooms/<room_id>/send/<event_type>|_.  These requests *can* use transaction
-IDs and ``PUT``/``POST`` methods. Message events allow access to historical
-events and pagination, making it best suited for sending messages.  For
-example::
-
-  POST /rooms/!roomid:domain/send/m.custom.example.message
-  { "text": "Hello world!" }
-
-  PUT /rooms/!roomid:domain/send/m.custom.example.message/11
-  { "text": "Goodbye world!" }
-
-See `Room Events`_ for the ``m.`` event specification.
-
-Getting events for a room
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-There are several APIs provided to ``GET`` events for a room:
-
-{{rooms_http_api}}
-
-
-{{message_pagination_http_api}}
 
 Redactions
 ~~~~~~~~~~

From d2bbc461e4b2bb473c4a190aacbf4ecf4796da4b Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Fri, 23 Oct 2015 13:25:12 +0100
Subject: [PATCH 384/467] mark stateKey as required in room/{id}/state

Swagger insists that path params be mandatory.
---
 api/client-server/v1/room_state.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/api/client-server/v1/room_state.yaml b/api/client-server/v1/room_state.yaml
index 0c0e76e6..bd8e2946 100644
--- a/api/client-server/v1/room_state.yaml
+++ b/api/client-server/v1/room_state.yaml
@@ -51,15 +51,15 @@ paths:
           type: string
           name: stateKey
           description: The state_key for the state to send. Defaults to the empty string.
-          required: false
+          required: true
           x-example: ""
         - in: body
           name: body
           schema:
             type: object
-            example: |
+            example: |-
               {
-                  "name": "The room name"
+                  "name": "New name for the room"
               }
       responses:
         200:

From 67880686d588eb54ef2b3f3a75d4ba4f984977b5 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 23 Oct 2015 14:02:53 +0100
Subject: [PATCH 385/467] Allow history to be noted as world-readable

SPEC-237
---
 event-schemas/schema/v1/m.room.history_visibility |  4 ++--
 specification/modules/history_visibility.rst      | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/event-schemas/schema/v1/m.room.history_visibility b/event-schemas/schema/v1/m.room.history_visibility
index bc33f4b3..9abcf5b8 100644
--- a/event-schemas/schema/v1/m.room.history_visibility
+++ b/event-schemas/schema/v1/m.room.history_visibility
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "Controls visibility of history.",
-    "description": "This event controls whether a member of a room can see the events that happened in a room from before they joined.",
+    "description": "This event controls whether a user can see the events that happened in a room from before they joined.",
     "allOf": [{
         "$ref": "core-event-schema/state_event.json"
     }],
@@ -12,7 +12,7 @@
                 "history_visibility": {
                     "type": "string",
                     "description": "Who can see the room history.",
-                    "enum": ["invited","joined","shared"]
+                    "enum": ["invited","joined","shared","world_readable"]
                 }
             },
             "required": ["history_visibility"]
diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst
index f26d1b59..112e2e21 100644
--- a/specification/modules/history_visibility.rst
+++ b/specification/modules/history_visibility.rst
@@ -6,14 +6,14 @@ Room History Visibility
 This module adds support for controlling the visibility of previous events in a
 room.
 
-In all cases, a user needs to join a room to view events in that room. Once they
+In all cases except ``world_readable``, a user needs to join a room to view events in that room. Once they
 have joined a room, they will gain access to a subset of events in the room. How
 this subset is chosen is controlled by the ``m.room.history_visibility`` event
 outlined below. After a user has left a room, they may seen any events which they
 were allowed to see before they left the room, but no events received after they
 left.
 
-The three options for the ``m.room.history_visibility`` event are:
+The four options for the ``m.room.history_visibility`` event are:
 
 - ``shared`` - Previous events are always accessible to newly joined members. All
   events in the room are accessible, even those sent when the member was not a part
@@ -24,6 +24,9 @@ The three options for the ``m.room.history_visibility`` event are:
 - ``joined`` - Previous events are accessible to newly joined members from the point
   they joined the room onwards. Events stop being accessible when the member's state
   changes to something other than ``join``.
+- ``world_readable`` - All events while this is the ``m.room.history_visibility`` value
+  may be shared by any participating homeserver with anyone, regardless of whether
+  they have ever joined the room.
 
 .. WARNING::
   These options are applied at the point an event is *sent*. Checks are
@@ -43,10 +46,13 @@ Client behaviour
 Clients that implement this module MUST present to the user the possible options
 for setting history visibility when creating a room. 
 
+Clients may want to display a notice that their events may be read by non-joined
+people if the value is set to ``world_readable``.
+
 Server behaviour
 ----------------
 
-By default if no ``history_visibility`` is set the visibility is assumed to be
+By default if no ``history_visibility`` is set, or if the value is not understood, the visibility is assumed to be
 ``shared``. The rules governing whether a user is allowed to see an event depend
 solely on the state of the room *at that event*:
 

From 28ab643a4e767f11d5afaacc10e689eb9e9ceca9 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Fri, 23 Oct 2015 15:54:19 +0100
Subject: [PATCH 386/467] Remove duplicated registration/login APIs

Currently the spec duplicates all of the account-management APIs. There's still
work to be done here, but the complete duplication is confusing.
---
 api/client-server/v2_alpha/registration.yaml |   5 +
 specification/client_server_api.rst          | 161 +------------------
 2 files changed, 9 insertions(+), 157 deletions(-)

diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml
index 5222b3f3..a53d4f41 100644
--- a/api/client-server/v2_alpha/registration.yaml
+++ b/api/client-server/v2_alpha/registration.yaml
@@ -89,6 +89,11 @@ paths:
             These errors may be returned at any stage of the registration process,
             including after authentication if the requested user ID was registered
             whilst the client was performing authentication.
+
+            Home Servers MUST perform the relevant checks and return these codes before
+            performing `User-Interactive Authentication`_, although they may also return
+            them after authentication is completed if, for example, the requested user ID
+            was registered whilst the client was performing authentication.
           examples:
             application/json: |-
               {
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 791c1f4a..89b6666a 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -349,67 +349,21 @@ This MUST return an HTML page which can perform this authentication stage. This
 page must attempt to call the JavaScript function ``window.onAuthDone`` when
 the authentication has been completed.
 
-Registration
-~~~~~~~~~~~~
+API calls using the User-Interactive Authentication mechanism
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 This section refers to API Version 2. These API calls currently use the prefix
 ``/_matrix/client/v2_alpha``.
 
-Registering for a user account is done using the request::
-
-  POST $V2PREFIX/register
-
-This API endpoint uses the `User-Interactive Authentication API`_.
-This API endpoint does not require an access token.
-
-.. _User-Interactive Authentication API: `sect:auth-api`_
-
-The body of the POST request is a JSON object containing:
-
-username
-  Optional. This is the local part of the desired Matrix ID. If omitted, the
-  Home Server must generate a Matrix ID local part.
-password
-  Required. The desired password for the account.
-bind_email
-  Optional. If ``true``, the server binds the email used for authentication to
-  the Matrix ID with the ID Server.
-
-On success, this returns a JSON object with keys:
-
-user_id
-  The fully-qualified Matrix ID that has been registered.
-access_token
-  An access token for the new account.
-home_server
-  The hostname of the Home Server on which the account has been registered.
-refresh_token
-  A token that may be exchanged for a new ``access_token`` using the
-  ``/tokenrefresh`` API endpoint.
-
-This endpoint may also return the following error codes:
-
-M_USER_IN_USE
-  If the Matrix ID is already in use
-M_EXCLUSIVE
-  If the requested Matrix ID is in the exclusive namespace of an application
-  service.
-
-Home Servers MUST perform the relevant checks and return these codes before
-performing `User-Interactive Authentication`_, although they may also return
-them after authentication is completed if, for example, the requested user ID
-was registered whilst the client was performing authentication.
-
 .. _User-Interactive Authentication: `sect:auth-api`_
 
+{{v2_registration_http_api}}
+
 Old V1 API docs: |register|_
 
 {{login_http_api}}
 
 Changing Password
 +++++++++++++++++
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
 Request::
 
   POST $V2PREFIX/account/password
@@ -432,9 +386,6 @@ database.
 
 Adding Account Administrative Contact Information
 +++++++++++++++++++++++++++++++++++++++++++++++++
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
 Request::
 
   POST $V2PREFIX/account/3pid
@@ -470,9 +421,6 @@ M_THREEPID_AUTH_FAILED
 
 Fetching Currently Associated Contact Information
 +++++++++++++++++++++++++++++++++++++++++++++++++
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
 Request::
 
   GET $V2PREFIX/account/3pid
@@ -982,107 +930,6 @@ member's state, by making a request to
     "membership": "ban"
   }
 
-Account operations
-------------------
-
-Registration
-~~~~~~~~~~~~
-This API endpoint uses the `User-Interactive Authentication API`_.
-
-{{v2_registration_http_api}}
-
-Login
-~~~~~
-
-{{login_http_api}}
-
-Changing Password
-~~~~~~~~~~~~~~~~~
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Request::
-
-  POST $V2PREFIX/account/password
-
-This API endpoint uses the User-Interactive Authentication API. An access token
-should be submitted to this endpoint if the client has an active session. The
-Home Server may change the flows available depending on whether a valid access
-token is provided.
-
-The body of the POST request is a JSON object containing:
-
-new_password
-  The new password for the account.
-
-On success, an empty JSON object is returned.
-
-The error code M_NOT_FOUND is returned if the user authenticated with a third
-party identifier but the Home Server could not find a matching account in its
-database.
-
-Adding a Third Party Identifier
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Request::
-
-  POST $V2PREFIX/account/3pid
-
-Used to add a third party identifier to the user's account.
-
-The body of the POST request is a JSON object containing:
-
-threePidCreds
-  An object containing third party identifier credentials.
-bind
-  Optional. A boolean indicating whether the Home Server should also bind this
-  third party identifier to the account's matrix ID with the Identity Server. If
-  supplied and true, the Home Server must bind the 3pid accordingly.
-
-The third party identifier credentials object comprises:
-
-id_server
-  The colon-separated hostname and port of the Identity Server used to
-  authenticate the third party identifier. If the port is the default, it and the
-  colon should be omitted.
-sid
-  The session ID given by the Identity Server
-client_secret
-  The client secret used in the session with the Identity Server.
-
-On success, the empty JSON object is returned.
-
-May also return error codes:
-
-M_THREEPID_AUTH_FAILED
-  If the credentials provided could not be verified with the ID Server.
-
-Fetching Currently Associated Third Party Identifiers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This section refers to API Version 2. These API calls currently use the prefix
-``/_matrix/client/v2_alpha``.
-
-Request::
-
-  GET $V2PREFIX/account/3pid
-
-This returns a list of third party identifiers that the Home Server has
-associated with the user's account. This is *not* the same as the list of third
-party identifiers bound to the user's Matrix ID in Identity Servers. Identifiers
-in this list may be used by the Home Server as, for example, identifiers that it
-will accept to reset the user's account password.
-
-Returns a JSON object with the key ``threepids`` whose contents is an array of
-objects with the following keys:
-
-medium
-  The medium of the 3pid (eg, ``email``)
-address
-  The textual address of the 3pid, eg. the email address
-
-
 Profiles
 --------
 

From 637718108ef8ddadef779b904a9042d8d1c2d525 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 23 Oct 2015 16:52:37 +0100
Subject: [PATCH 387/467] Add newline before list

---
 api/client-server/v1/membership.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index 13bfb6bb..f8dfdea5 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -52,6 +52,7 @@ paths:
         403:
           description: |-
             You do not have permission to join the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejection are:
+
             - The room is invite-only and the user was not invited.
             - The user has been banned from the room.
           examples:

From 19456974567dbb52189ea247406c71fd64f111b2 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Fri, 23 Oct 2015 16:58:55 +0100
Subject: [PATCH 388/467] Further tweaks to the room send and state APIs

- fix confusion re empty/absent state_keys
- move 'types of room events' section earlier in the 'Events' section
- remove some redundant anchors
---
 api/client-server/v1/room_state.yaml |  8 +++--
 specification/client_server_api.rst  | 51 +++++++++++++---------------
 2 files changed, 29 insertions(+), 30 deletions(-)

diff --git a/api/client-server/v1/room_state.yaml b/api/client-server/v1/room_state.yaml
index bd8e2946..ec30fd34 100644
--- a/api/client-server/v1/room_state.yaml
+++ b/api/client-server/v1/room_state.yaml
@@ -24,9 +24,11 @@ paths:
       description: |
         State events can be sent using this endpoint.  These events will be
         overwritten if ``<room id>``, ``<event type>`` and ``<state key>`` all
-        match.  If the state event has no ``state_key``, it can be omitted from
-        the path. These requests **cannot use transaction IDs** like other
-        ``PUT`` paths because they cannot be differentiated from the
+        match.  If the state event has an empty ``state_key``, it can be
+        omitted from the path.
+
+        Requests to this endpoint **cannot use transaction IDs**
+        like other ``PUT`` paths because they cannot be differentiated from the
         ``state_key``. Furthermore, ``POST`` is unsupported on state paths.
 
         The body of the request should be the content object of the event; the
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 4df821e0..9e89ecf4 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -647,6 +647,30 @@ To continue paginating backwards, one calls the /messages API again, supplying
 the new ``start`` value as the ``from`` parameter.
 
 
+Types of room events
+~~~~~~~~~~~~~~~~~~~~
+
+Room events are split into two categories:
+
+:State Events:
+  These are events which update the metadata state of the room (e.g. room topic,
+  room membership etc). State is keyed by a tuple of event ``type`` and a
+  ``state_key``. State in the room with the same key-tuple will be overwritten.
+
+:Message events:
+  These are events which describe transient "once-off" activity in a room:
+  typically communication such as sending an instant message or setting up a
+  VoIP call.
+
+This specification outlines several events, all with the event type prefix
+``m.``. (See `Room Events`_ for the m. event specification.) However,
+applications may wish to add their own type of event, and this can be achieved
+using the REST API detailed in the following sections. If new events are added,
+the event ``type`` key SHOULD follow the Java package naming convention,
+e.g. ``com.example.myapp.event``.  This ensures event types are suitably
+namespaced for each application and reduces the risk of clashes.
+
+
 Syncing
 ~~~~~~~
 
@@ -675,27 +699,6 @@ This API also returns an ``end`` token which can be used with the event stream.
 
 {{sync_http_api}}
 
-Types of room events
-~~~~~~~~~~~~~~~~~~~~
-
-Room events are split into two categories:
-
-:State Events:
-  These are events which update the metadata state of the room (e.g. room topic,
-  room membership etc). State is keyed by a tuple of event ``type`` and a
-  ``state_key``. State in the room with the same key-tuple will be overwritten.
-
-:Message events:
-  These are events which describe transient "once-off" activity in a room:
-  typically communication such as sending an instant message or setting up a
-  VoIP call.
-
-This specification outlines several events, all with the event type prefix
-``m.``. However, applications may wish to add their own type of event, and this
-can be achieved using the REST API detailed in the following sections. If new
-events are added, the event ``type`` key SHOULD follow the Java package naming
-convention, e.g. ``com.example.myapp.event``.  This ensures event types are
-suitably namespaced for each application and reduces the risk of clashes.
 
 Getting events for a room
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1144,12 +1147,6 @@ have to wait in milliseconds before they can try again.
 .. |/rooms/<room_id>/state| replace:: ``/rooms/<room_id>/state``
 .. _/rooms/<room_id>/state: /docs/api/client-server/#!/-rooms/get_state_events
 
-.. |/rooms/<room_id>/send/<event_type>| replace:: ``/rooms/<room_id>/send/<event_type>``
-.. _/rooms/<room_id>/send/<event_type>: /docs/api/client-server/#!/-rooms/send_non_state_event
-
-.. |/rooms/<room_id>/state/<event_type>/<state_key>| replace:: ``/rooms/<room_id>/state/<event_type>/<state_key>``
-.. _/rooms/<room_id>/state/<event_type>/<state_key>: /docs/api/client-server/#!/-rooms/send_state_event
-
 .. |/rooms/<room_id>/invite| replace:: ``/rooms/<room_id>/invite``
 .. _/rooms/<room_id>/invite: /docs/api/client-server/#!/-rooms/invite
 

From 36634df3555c72456ac7aa2b8968e96a5921f896 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 10:18:40 +0000
Subject: [PATCH 389/467] Add filter param

---
 api/client-server/v1/search.yaml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/api/client-server/v1/search.yaml b/api/client-server/v1/search.yaml
index 885e62b3..cc299056 100644
--- a/api/client-server/v1/search.yaml
+++ b/api/client-server/v1/search.yaml
@@ -62,6 +62,12 @@ paths:
                           type: string
                           enum: ["content.body", "content.name", "content.topic"]
                         description: The keys to search. Defaults to all.
+                      filter:
+                        type: object
+                        title: Filter
+                        description: |-
+                            The filter to apply to search results.
+                            This has the same format as v2 filter API.
                     required: ["search_term"]
             required: ["search_categories"]
       responses:

From 1aeddac3199e045a69ff163cbfc1bd4c4fc553a4 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 26 Oct 2015 10:28:14 +0000
Subject: [PATCH 390/467] Cache spec output

Does not cache inability to generate spec. Does not cache diffs or
anything fancy. Just the raw spec generation.
---
 scripts/speculator/README  |  8 ++++++--
 scripts/speculator/main.go | 17 +++++++++++++++++
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/scripts/speculator/README b/scripts/speculator/README
index c54e0f2b..82dc3d36 100644
--- a/scripts/speculator/README
+++ b/scripts/speculator/README
@@ -6,9 +6,13 @@ It serves the following HTTP endpoints:
  - /diff/rst/123 which gives a diff of the spec's rst at pull request 123.
  - /diff/html/123 which gives a diff of the spec's HTML at pull request 123.
 
-To run it, you must install the `go` tool, and run:
+The build or run, you need a working `go` installation.
+Then fetch dependencies:
+ ` go get github.com/hashicorp/golang-lru`
+
+To run it, then run:
  `go run main.go`
 
 To build the binary (which is necessary for deployment to the matrix.org
-servers), you must again install `go`, and then run:
+servers), you must again install `go` and dependencies, and then run:
  `go build`
diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index e6992d4a..85fb2596 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -25,6 +25,8 @@ import (
 	"strings"
 	"syscall"
 	"time"
+
+	"github.com/hashicorp/golang-lru"
 )
 
 type PullRequest struct {
@@ -53,6 +55,7 @@ type User struct {
 var (
 	port           = flag.Int("port", 9000, "Port on which to listen for HTTP")
 	allowedMembers map[string]bool
+	specCache      *lru.Cache // string -> []byte
 )
 
 func (u *User) IsTrusted() bool {
@@ -196,6 +199,10 @@ func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 		}
 		sha = pr.Head.SHA
 	}
+	if cached, ok := specCache.Get(sha); ok {
+		w.Write(cached.([]byte))
+		return
+	}
 
 	dst, err := s.generateAt(sha)
 	defer os.RemoveAll(dst)
@@ -210,6 +217,7 @@ func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 		return
 	}
 	w.Write(b)
+	specCache.Add(sha, b)
 }
 
 func checkAuth(pr *PullRequest) error {
@@ -367,6 +375,9 @@ func main() {
 		"richvdh":       true,
 		"leonerd":       true,
 	}
+	if err := initCache(); err != nil {
+		log.Fatal(err)
+	}
 	rand.Seed(time.Now().Unix())
 	masterCloneDir, err := gitClone(matrixDocCloneURL, false)
 	if err != nil {
@@ -388,3 +399,9 @@ func serveText(s string) func(http.ResponseWriter, *http.Request) {
 		io.WriteString(w, s)
 	}
 }
+
+func initCache() error {
+	c, err := lru.New(50) // Evict after 50 entries (i.e. 50 sha1s)
+	specCache = c
+	return err
+}

From 3b7585cbda5e588bb1de47230d8d9a39a40eb1cd Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 26 Oct 2015 11:17:19 +0000
Subject: [PATCH 391/467] Make nested request objects display correctly (search
 API)

This now displays search_categories.room_events.filter and co correctly.
Also make arrays of enums display correctly.
---
 .../matrix_templates/templates/http-api.tmpl  | 12 +++++-----
 templating/matrix_templates/units.py          | 24 ++++++++++++++++++-
 2 files changed, 29 insertions(+), 7 deletions(-)

diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index d1633513..2812cf7b 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -13,17 +13,17 @@
 
 Request format:
 
-=================================== ================= ===========================================
-   Parameter                             Value                        Description
-=================================== ================= ===========================================
+=========================================== ================= ===========================================
+   Parameter                                 Value                        Description
+=========================================== ================= ===========================================
 {% for loc in endpoint.req_param_by_loc -%}
 *{{loc}} parameters*
-------------------------------------------------------------------------------------------------- 
+--------------------------------------------------------------------------------------------------------- 
 {% for param in endpoint.req_param_by_loc[loc] -%}
-{{param.key}}{{param.type|indent(36-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(54)}}
+{{param.key}}{{param.type|indent(44-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(62)}}
 {% endfor -%}
 {% endfor -%}
-=================================== ================= ===========================================
+=========================================== ================= ===========================================
 
 {% if endpoint.res_tables|length > 0 -%}
 Response format:
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 78d1b367..c5cf0044 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -121,6 +121,17 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                 tables += nested_object
             else:
                 value_type = "[%s]" % props[key_name]["items"]["type"]
+                array_enums = props[key_name]["items"].get("enum")
+                if array_enums:
+                    if len(array_enums) > 1:
+                        value_type = "[enum]"
+                        desc += (
+                            " One of: %s" % json.dumps(array_enums)
+                        )
+                    else:
+                        desc += (
+                            " Must be '%s'." % array_enums[0]
+                        )
         else:
             value_type = props[key_name]["type"]
             if props[key_name].get("enum"):
@@ -252,13 +263,24 @@ class MatrixUnits(Units):
                                 req_obj = req_obj["items"]
 
                             req_tables = get_json_schema_object_fields(req_obj)
+
+                            if req_tables > 1:
+                                for table in req_tables[1:]:
+                                    nested_key_name = [
+                                        s["key"] for s in req_tables[0]["rows"] if
+                                        s["type"] == ("{%s}" % table["title"])
+                                    ][0]
+                                    for row in table["rows"]:
+                                        row["key"] = "%s.%s" % (nested_key_name, row["key"])
+
                             key_sep = "[0]." if is_array else "."
                             for table in req_tables:
                                 if table.get("no-table"):
                                     continue
                                 for row in table["rows"]:
+                                    nested_key = key + key_sep + row["key"]
                                     endpoint["req_params"].append({
-                                        "key": key + key_sep + row["key"],
+                                        "key": nested_key,
                                         "loc": "JSON body",
                                         "type": row["type"],
                                         "desc": row["req_str"] + row["desc"]

From 88c459d5490e3c74118d5cfd34038869418d3d98 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 13:40:24 +0000
Subject: [PATCH 392/467] Add back in login v1 fallback API

---
 specification/client_server_api.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 9e230334..9266002f 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -362,6 +362,18 @@ Old V1 API docs: |register|_
 
 {{login_http_api}}
 
+Fallback
+++++++++
+
+Clients cannot be expected to be able to know how to process every single login
+type. If a client determines it does not know how to handle a given login type,
+it should request a login fallback page:
+
+    GET /_matrix/client/api/v1/login/fallback
+
+This returns an HTML and JavaScript page which can perform the entire login
+process.
+
 Changing Password
 +++++++++++++++++
 Request::

From 0529b4a9a2018feb509459403d2c0f2a26029b40 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 13:45:36 +0000
Subject: [PATCH 393/467] Reword

---
 specification/client_server_api.rst | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 9266002f..da6f13a7 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -365,14 +365,14 @@ Old V1 API docs: |register|_
 Fallback
 ++++++++
 
-Clients cannot be expected to be able to know how to process every single login
-type. If a client determines it does not know how to handle a given login type,
-it should request a login fallback page:
+If a client does not recognize all or any login flows, it can use the fallback
+login API::
 
     GET /_matrix/client/api/v1/login/fallback
 
 This returns an HTML and JavaScript page which can perform the entire login
-process.
+process. The page will attempt to call the JavaScript function
+``window.onLogin`` when login has been successfully completed.
 
 Changing Password
 +++++++++++++++++

From 98dbf4d60b7e79d7b29fb0aa17e7aaa9bf7d0273 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 13:46:41 +0000
Subject: [PATCH 394/467] Better flow

---
 specification/client_server_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index da6f13a7..fac2e8ff 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -365,7 +365,7 @@ Old V1 API docs: |register|_
 Fallback
 ++++++++
 
-If a client does not recognize all or any login flows, it can use the fallback
+If a client does not recognize any or all login flows it can use the fallback
 login API::
 
     GET /_matrix/client/api/v1/login/fallback

From 9a5bd6d21f9306ff1673155a41cc055de6024665 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 13:56:07 +0000
Subject: [PATCH 395/467] Use static

---
 specification/client_server_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index fac2e8ff..9f942c94 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -368,7 +368,7 @@ Fallback
 If a client does not recognize any or all login flows it can use the fallback
 login API::
 
-    GET /_matrix/client/api/v1/login/fallback
+    GET /_matrix/static/client/login/
 
 This returns an HTML and JavaScript page which can perform the entire login
 process. The page will attempt to call the JavaScript function

From 30d46a19d51e8ed64ef9743ee7f6f451c034a723 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 26 Oct 2015 14:01:27 +0000
Subject: [PATCH 396/467] Review comments

---
 templating/matrix_templates/units.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index c5cf0044..c4342141 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -268,7 +268,7 @@ class MatrixUnits(Units):
                                 for table in req_tables[1:]:
                                     nested_key_name = [
                                         s["key"] for s in req_tables[0]["rows"] if
-                                        s["type"] == ("{%s}" % table["title"])
+                                        s["type"] == ("{%s}" % (table["title"],))
                                     ][0]
                                     for row in table["rows"]:
                                         row["key"] = "%s.%s" % (nested_key_name, row["key"])

From 3cf431cac1e3ed81578ba821f894ca6146487bf6 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 14:10:02 +0000
Subject: [PATCH 397/467] Add link

---
 specification/client_server_api.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 9f942c94..8988acb4 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -365,6 +365,8 @@ Old V1 API docs: |register|_
 Fallback
 ++++++++
 
+.. _sect:login_fallback:
+
 If a client does not recognize any or all login flows it can use the fallback
 login API::
 

From 696f75d80b07fc16b01e238f3a377d345e02ca6e Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 26 Oct 2015 14:14:13 +0000
Subject: [PATCH 398/467] Rename section

---
 specification/client_server_api.rst | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 8988acb4..09c6765c 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -362,10 +362,8 @@ Old V1 API docs: |register|_
 
 {{login_http_api}}
 
-Fallback
-++++++++
-
-.. _sect:login_fallback:
+Login Fallback
+++++++++++++++
 
 If a client does not recognize any or all login flows it can use the fallback
 login API::

From 13eddd456f8277c9aec1743f5541d43f38b016a1 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Mon, 26 Oct 2015 15:52:45 +0000
Subject: [PATCH 399/467] Proposal for adding full_state param to v2 sync

---
 api/client-server/v2_alpha/sync.yaml | 20 ++++++++++++++++++--
 specification/client_server_api.rst  |  2 ++
 2 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 266f27bc..5acbf583 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -22,7 +22,7 @@ paths:
       summary: Synchronise the client's state and receive new messages.
       description: |-
         Synchronise the client's state with the latest state on the server.
-        Client's use this API when they first log in to get an initial snapshot
+        Clients use this API when they first log in to get an initial snapshot
         of the state on the server, and then continue to call this API to get
         incremental deltas to the state, and to receive new messages.
       security:
@@ -40,6 +40,22 @@ paths:
           description: |-
             A point in time to continue a sync from.
           x-example: "s72594_4483_1934"
+        - in: query
+          name: full_state
+          type: boolean
+          description: |-
+            Controls whether to include the full state for all rooms the user
+            is a member of.
+
+            If this is set to ``true``, then all state events will be returned,
+            even if ``since`` is non-empty. The timeline will still be limited
+            by the ``since`` parameter.
+
+            If ``false``, and ``since`` is non-empty, only state which has
+            changed since the point indicated by ``since`` will be returned.
+
+            By default, this is ``false``.
+          x-example: "false"
         - in: query
           name: set_presence
           type: string
@@ -135,7 +151,7 @@ paths:
                           type: object
                           description: |-
                             The state of a room that the user has been invited
-                            to. These state events may only have the `sender``,
+                            to. These state events may only have the ``sender``,
                             ``type``, ``state_key`` and ``content`` keys
                             present. These events do not replace any state that
                             the client already has for the room, for example if
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 09c6765c..b02dbf28 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -659,6 +659,8 @@ This API also returns an ``end`` token which can be used with the event stream.
 
 {{sync_http_api}}
 
+{{v2_sync_http_api}}
+
 
 Getting events for a room
 ~~~~~~~~~~~~~~~~~~~~~~~~~

From 3f1cbb22bb5a143b7318794429d7e5dce6c0e024 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 26 Oct 2015 16:27:57 +0000
Subject: [PATCH 400/467] Enable syntax highlighting for the JSON event
 examples

---
 templating/matrix_templates/templates/events.tmpl | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/templates/events.tmpl b/templating/matrix_templates/templates/events.tmpl
index 9dcc468e..fb876440 100644
--- a/templating/matrix_templates/templates/events.tmpl
+++ b/templating/matrix_templates/templates/events.tmpl
@@ -21,6 +21,8 @@
 ======================= ================= ===========================================
 
 {% endfor %}
-Example::
+Example:
+
+.. code:: json
 
     {{example | jsonify(4, 4)}}

From 71874870c8b66aa1f71247b74d9bb6453880e015 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 26 Oct 2015 17:25:33 +0000
Subject: [PATCH 401/467] Enable syntax highlighting for example http requests

---
 scripts/css/codehighlight.css                      | 14 ++++++++++++--
 .../matrix_templates/templates/http-api.tmpl       |  4 +++-
 templating/matrix_templates/units.py               | 12 +++++++++---
 3 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/scripts/css/codehighlight.css b/scripts/css/codehighlight.css
index 5c9b0c36..fafc43f4 100644
--- a/scripts/css/codehighlight.css
+++ b/scripts/css/codehighlight.css
@@ -1,6 +1,16 @@
 pre.code .comment, code .comment { color: green }
 pre.code .keyword, code .keyword { color: darkred; font-weight: bold }
 pre.code .name.builtin, code .name.builtin { color: darkred; font-weight: bold }
-pre.code .literal.number, code .literal.number { color: blue }
 pre.code .name.tag, code .name.tag { color: darkgreen }
-pre.code .literal.string, code .literal.string { color: darkblue }
+pre.code .literal, code .literal { color: darkblue }
+pre.code .literal.number, code .literal.number { color: blue }
+
+
+/* HTTP Methods have class "name function" */
+pre.code.http .name.function,  code.http .name.function { color: black; font-weight: bold }
+/* HTTP Paths have class "name namespace" */
+pre.code.http .name.namespace, code.http .name.namespace { color: darkgreen }
+/* HTTP "HTTP" strings have class "keyword reserved" */
+pre.code.http .keyword.reserved, code.http .keyword.reserved { color: black; font-weight: bold }
+/* HTTP Header names have class "name attribute" */
+pre.code.http .name.attribute, code.http .name.attribute { color: black; font-weight: bold }
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 2812cf7b..86eacb14 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -47,7 +47,9 @@ Response format:
 {% endfor %}
 {% endif -%}
 
-Example request::
+Example request:
+
+.. code:: http
 
   {{endpoint.example.req | indent_block(2)}}
 
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index c4342141..be113b9c 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -336,9 +336,15 @@ class MatrixUnits(Units):
                         elif param["in"] == "query":
                             qps[param["name"]] = param["x-example"]
                     query_string = "" if len(qps) == 0 else "?"+urllib.urlencode(qps)
-                    endpoint["example"]["req"] = "%s %s%s\n%s" % (
-                        method.upper(), path_template, query_string, body
-                    )
+                    if body:
+                        endpoint["example"]["req"] = "%s %s%s HTTP/1.1\nContent-Type: application/json\n\n%s" % (
+                            method.upper(), path_template, query_string, body
+                        )
+                    else:
+                        endpoint["example"]["req"] = "%s %s%s HTTP/1.1\n\n" % (
+                            method.upper(), path_template, query_string
+                        )
+
                 else:
                     self.log(
                         "The following parameters are missing examples :( \n %s" %

From ece42688d089efbbe69e4d6c7b07bb2d878ec7b3 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Mon, 26 Oct 2015 19:20:58 +0000
Subject: [PATCH 402/467] Clarify the interaction between full_state and
 timeout.

---
 api/client-server/v2_alpha/sync.yaml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 5acbf583..01308405 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -49,7 +49,9 @@ paths:
 
             If this is set to ``true``, then all state events will be returned,
             even if ``since`` is non-empty. The timeline will still be limited
-            by the ``since`` parameter.
+            by the ``since`` parameter. In this case, the ``timeout`` parameter
+            will be ignored and the query will return immediately, possibly with
+            an empty timeline.
 
             If ``false``, and ``since`` is non-empty, only state which has
             changed since the point indicated by ``since`` will be returned.

From 745e60757768b296c3136b761b244e1167c32a2b Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Wed, 28 Oct 2015 14:49:50 +0000
Subject: [PATCH 403/467] Use 'room' or 'room ID' instead of 'context' when
 describing federation protocol

---
 specification/server_server_api.rst | 32 ++++++++++++++---------------
 1 file changed, 16 insertions(+), 16 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 66367cb0..26e040ca 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -15,9 +15,9 @@ There are three main kinds of communication that occur between home servers:
 
 Persisted Data Units (PDUs):
     These events are broadcast from one home server to any others that have
-    joined the same "context" (namely, a Room ID). They are persisted in
+    joined the same room (identified by Room ID). They are persisted in
     long-term storage and record the history of messages and state for a
-    context.
+    room.
 
     Like email, it is the responsibility of the originating server of a PDU
     to deliver that event to its recipient servers. However PDUs are signed
@@ -26,7 +26,7 @@ Persisted Data Units (PDUs):
 
 Ephemeral Data Units (EDUs):
     These events are pushed between pairs of home servers. They are not
-    persisted and are not part of the history of a "context", nor does the
+    persisted and are not part of the history of a room, nor does the
     receiving home server have to reply to them.
 
 Queries:
@@ -338,11 +338,11 @@ PDUs
 
 All PDUs have:
 
-- An ID
-- A context
+- An ID to identify the PDU itself
+- A room ID that it relates to
 - A declaration of their type
-- A list of other PDU IDs that have been seen recently on that context
-  (regardless of which origin sent them)
+- A list of other PDU IDs that have been seen recently in that room (regardless
+  of which origin sent them)
 
 
 Required PDU Fields
@@ -351,7 +351,7 @@ Required PDU Fields
 ==================== ================== =======================================
  Key                  Type               Description
 ==================== ================== =======================================
-``context``          String             Event context identifier
+``context``          String             Room identifier
 ``user_id``          String             The ID of the user sending the PDU
 ``origin``           String             DNS name of homeserver that created
                                         this PDU
@@ -363,7 +363,7 @@ Required PDU Fields
 ``content``          Object             The content of the PDU.
 ``prev_pdus``        List of (String,   The originating homeserver, PDU ids and
                      String, Object)    hashes of the most recent PDUs the
-                     Triplets           homeserver was aware of for the context
+                     Triplets           homeserver was aware of for the room
                                         when it made this PDU
 ``depth``            Integer            The maximum depth of the previous PDUs
                                         plus one
@@ -440,7 +440,7 @@ keys exist to support this:
 EDUs
 ----
 
-EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
+EDUs, by comparison to PDUs, do not have an ID, a room ID, or a list of
 "previous" IDs. The only mandatory fields for these are the type, origin and
 destination home server names, and the actual nested content.
 
@@ -491,23 +491,23 @@ Retrieves a given PDU from the server. The response will contain a single new
 Transaction, inside which will be the requested PDU.
 
 
-To fetch all the state of a given context::
+To fetch all the state of a given room::
 
-  GET .../state/<context>/
+  GET .../state/<room_id>/
     Response: JSON encoding of a single Transaction containing multiple PDUs
 
-Retrieves a snapshot of the entire current state of the given context. The
+Retrieves a snapshot of the entire current state of the given room. The
 response will contain a single Transaction, inside which will be a list of PDUs
 that encode the state.
 
-To backfill events on a given context::
+To backfill events on a given room::
 
-  GET .../backfill/<context>/
+  GET .../backfill/<room_id>/
     Query args: v, limit
     Response: JSON encoding of a single Transaction containing multiple PDUs
 
 Retrieves a sliding-window history of previous PDUs that occurred on the given
-context. Starting from the PDU ID(s) given in the "v" argument, the PDUs that
+room. Starting from the PDU ID(s) given in the "v" argument, the PDUs that
 preceded it are retrieved, up to a total number given by the "limit" argument.
 These are then returned in a new Transaction containing all of the PDUs.
 

From 5b6f858802332e8fcee368e9db6cdfac7533c995 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Wed, 28 Oct 2015 20:28:49 +0000
Subject: [PATCH 404/467] Some initial notes by way of the remote join
 handshake; with several TODOs and unanswered questions

---
 specification/server_server_api.rst | 149 +++++++++++++++++++++++++++-
 1 file changed, 147 insertions(+), 2 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 26e040ca..f0239128 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -533,6 +533,150 @@ part of the path specifies the kind of query being made, and its query
 arguments have a meaning specific to that kind of query. The response is a
 JSON-encoded object whose meaning also depends on the kind of query.
 
+
+To join a room::
+
+  GET .../make_join/<room_id>/<user_id>
+    Response: JSON encoding of a join proto-event
+
+  PUT .../send_join/<room_id>/<event_id>
+    Response: JSON encoding of the state of the room at the time of the event
+
+Performs the room join handshake. For more information, see "Joining Rooms"
+below.
+
+Joining Rooms
+-------------
+
+When a new user wishes to join room that the user's homeserver already knows
+about, the homeserver can immediately determine if this is allowable by
+inspecting the state of the room, and if it is acceptable, it can generate,
+sign, and emit a new ``m.room.member`` state event adding the user into that
+room. When the homeserver does not yet know about the room it cannot do this
+directly. Instead, it must take a longer multi-stage handshaking process by
+which it first selects a remote homeserver which is already participating in
+that room, and uses it to assist in the joining procss. This is the remote join
+handshake.
+
+This handshake involves the homeserver of the new member wishing to join
+(referred to here as the "joining" server), the directory server hosting the
+room alias the user is requesting to join with, and a homeserver where existing
+room members are already present (referred to as the "resident" server).
+
+In summary, the remote join handshake consists of the joining server querying
+the directory server for information about the room alias; receiving a room ID
+and a list of join candidates. The joining server then requests information
+about the current state of the room from one of the residents. It uses this
+information to construct a ``m.room.member`` event which it finally sends to
+a resident server.
+
+Conceptually these are three different roles of homeserver. In practice the
+directory server is likely to be resident in the room, and so may be selected
+by the joining server to be the assisting resident. Likewise, it is likely that
+the joining server picks the same candidate resident for both phases of event
+construction, though in principle any valid candidate may be used at each time.
+Thus, any join handshake can potentially involve anywhere from two to four
+homeservers, though most in practice will use just two.
+
+.. TODO-doc
+  - Consider drawing a request/response diagram here
+
+The first part of the handshake involves using the directory server to request
+the room ID and join candidates. This is covered in more detail on the
+directory server documentation, below.
+
+Once the joining server has the room ID and the join candidates, it then needs
+to obtain enough of the current state of the room to fill in the required
+fields of the ``m.room.member`` event. It obtains this by selecting a resident
+from the candidate list, and requesting the ``make_join`` endpoint using a
+``GET`` request, specifying the room ID and the user ID of the new member who
+is attempting to join.
+
+The resident server replies to this request with a JSON-encoded object having a
+single key called ``event``; within this is an object whose fields contain some
+of the information that the joining server will need. Despite its name, this
+object is not a full object; notably it does not need to be hashed or signed by
+the assisting resident. The required fields are:
+
+==================== ======== ============
+ Key                  Type     Description
+==================== ======== ============
+``type``             String   The value ``m.room.member``
+``auth_events``      String   An event-reference list containing the
+                              authorization events that would allow this member
+                              to join
+``content``          Object   The event content
+``depth``            Integer  (this field must be present but is ignored; it
+                              may be 0)
+``event_id``         String   A new event ID specified by the assisting
+                              resident
+``origin``           String   The name of the assisting resident homeserver
+``origin_server_ts`` Integer  A timestamp added by the resident homeserver
+``prev_events``      List     (TODO(paul): ? - I notice these can be blank)
+``prev_state``       List     (TODO(paul): ? - I notice these can be blank)
+``room_id``          String   The room ID of the room
+``sender``           String   The user ID of the joining member
+``state_key``        String   The user ID of the joining member
+==================== ======== ============
+
+The ``content`` field itself must be an object, containing:
+
+============== ====== ============
+ Key            Type   Description
+============== ====== ============
+``membership`` String The value ``join``
+============== ====== ============
+
+The joining server now has sufficient information to construct the real join
+event from these protoevent fields. It copies the values of most of them,
+adding (or replacing) the following fields:
+
+==================== ======= ============
+ Key                  Type    Description
+==================== ======= ============
+``event_id``         String  A new event ID specified by the joining homeserver
+``origin``           String  The name of the joining homeserver
+``origin_server_ts`` Integer A timestamp added by the joining homeserver
+==================== ======= ============
+
+.. TODO-spec
+  - Why does the protoevent have an event_id, only for the real event to ignore
+    it and specify a different one? We should definitely pick one or the other.
+
+This will be a true event, so the joining server should apply the event-signing
+algorithm to it, resulting in the addition of the ``hashes`` and ``signatures``
+fields.
+
+To complete the join handshake, the joining server must now submit this new
+event to an assisting resident, by using the ``send_join`` endpoint. This is
+invoked using the room ID and the event ID of the new member event.
+
+The assisting resident then accepts this event into the room's event graph, and
+responds to the joining server with the full set of state for the newly-joined
+room. This is returned as a two-element list, whose first element is the
+integer 200, and whose second element contains the following keys:
+
+.. TODO-spec
+  - This is likely an implementation bug; see SYN-490. This should probably
+    actually just return the object directly
+
+============== ===== ============
+ Key            Type  Description
+============== ===== ============
+``auth_chain`` List  A list of events giving the authorization chain for this
+                     join event
+``state``      List  A complete list of the prevailing state events at the
+                     instant just before accepting the new ``m.room.member``
+                     event
+============== ===== ============
+
+.. TODO-spec
+  - (paul) I don't really understand why the full auth_chain events are given
+    here. What purpose does it serve expanding them out in full, when surely
+    they'll appear in the state anyway?
+  - (paul) the state seems to be entirely ignored by synapse, so I'm not really
+    sure what ought to be there.
+
 Backfilling
 -----------
 .. NOTE::
@@ -763,6 +907,7 @@ Querying directory information::
     servers: list of strings giving the join candidates
 
 The list of join candidates is a list of server names that are likely to hold
-the given room; these are servers that the requesting server may wish to try
-joining with. This list may or may not include the server answering the query.
+the given room; these are servers that the requesting server may wish to use as
+assisting resident servers as part of the remote join handshake. This list may
+or may not include the server answering the query.
 

From 6c1df04b4b99e027ba0750af785a335b08133fc8 Mon Sep 17 00:00:00 2001
From: Matthew Hodgson <matthew@matrix.org>
Date: Thu, 29 Oct 2015 02:11:47 +0000
Subject: [PATCH 405/467] typo

---
 api/client-server/v2_alpha/filter.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v2_alpha/filter.yaml b/api/client-server/v2_alpha/filter.yaml
index 37a0a3aa..0c2761b7 100644
--- a/api/client-server/v2_alpha/filter.yaml
+++ b/api/client-server/v2_alpha/filter.yaml
@@ -56,7 +56,7 @@ paths:
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]
                   },
-                  "emphemeral": {
+                  "ephemeral": {
                     "types": ["m.receipt", "m.typing"],
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]
@@ -120,7 +120,7 @@ paths:
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]
                   },
-                  "emphemeral": {
+                  "ephemeral": {
                     "types": ["m.receipt", "m.typing"],
                     "not_rooms": ["!726s6s6q:example.com"],
                     "not_senders": ["@spam:example.com"]

From d067e50af5990177199a6f3adb0617c2a8ee8836 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 29 Oct 2015 18:38:33 +0000
Subject: [PATCH 406/467] Document the differences in event formatting between
 the v1 and v2 client APIs

---
 specification/events.rst | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/specification/events.rst b/specification/events.rst
index 7d64882b..d2428b6b 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -12,6 +12,41 @@ server-server and application-service APIs, and are described below.
 {{common_state_event_fields}}
 
 
+Differences between /v1 and /v2 events
+--------------------------------------
+
+There are a few differences between how events are formatted for sending
+between servers over federation and how they are formatted for sending between
+a server and its clients.
+
+Additionally there are a few differences between the format of events in the
+responses to client APIs with a /v1 prefix and responses APIs with a /v2
+prefix.
+
+Events in responses for APIs with the /v2 prefix are generated from an event
+formatted for federation by:
+
+* Removing the following keys:
+  ``auth_events``, ``prev_events``, ``hashes``, ``signatures``, ``depth``,
+  ``origin``, ``prev_state``.
+* Adding an ``age`` to the ``unsigned`` object which gives the time in
+  milliseconds that has ellapsed since the event was sent.
+* Adding a ``prev_content`` to the ``unsigned`` object if the event is
+  a ``state event`` which gives previous content of that state key.
+* Adding a ``redacted_because`` to the ``unsigned`` object if the event was
+  redacted which gives the event that redacted it.
+* Adding a ``transaction_id`` if the event was sent by the client requesting it.
+
+Events in responses for APIs with the /v1 prefix are generated from an event
+formatted for the /v2 prefix by:
+
+* Moving the folling keys from the ``unsigned`` object to the top level event
+  object: ``age``, ``redacted_because``, ``replaces_state``, ``prev_content``.
+* Removing the ``unsigned`` object.
+* If the event was an ``m.room.member`` with ``membership`` set to ``invite``
+  then adding a ``invite_room_state`` key to the top level event object.
+
+
 Size limits
 -----------
 

From 451801bf38159aacc5ee2e05d0230a5b89382712 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 29 Oct 2015 18:40:05 +0000
Subject: [PATCH 407/467] Add an example of ``prev_content`` in ``unsigned`` to
 v2 /sync

---
 api/client-server/v2_alpha/sync.yaml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 01308405..a2d5a2b8 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -241,6 +241,9 @@ paths:
                           "type": "m.room.member",
                           "state_key": "@bob:example.com",
                           "content": {"membership": "join"},
+                          "unsigned": {
+                            "prev_content": {"membership": "invite"}
+                          },
                           "origin_server_ts": 1417731086795
                         },
                         "$74686972643033:example.com": {

From d297d83151cf44b785ca5d671ddf4bc8de5f4974 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Thu, 29 Oct 2015 18:45:53 +0000
Subject: [PATCH 408/467] Mention that sender is renamed to user_id in v1

---
 specification/events.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/specification/events.rst b/specification/events.rst
index d2428b6b..a1aece1c 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -43,6 +43,7 @@ formatted for the /v2 prefix by:
 * Moving the folling keys from the ``unsigned`` object to the top level event
   object: ``age``, ``redacted_because``, ``replaces_state``, ``prev_content``.
 * Removing the ``unsigned`` object.
+* Rename the ``sender`` key to ``user_id``.
 * If the event was an ``m.room.member`` with ``membership`` set to ``invite``
   then adding a ``invite_room_state`` key to the top level event object.
 

From 91eb25b76d12b7936583e7a3c482d5b3c066245e Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Fri, 30 Oct 2015 15:45:46 +0000
Subject: [PATCH 409/467] Include the full schema for an http API in the docs
 by resolving references to other files

---
 templating/matrix_templates/units.py | 66 +++++++++++++++++++++-------
 1 file changed, 51 insertions(+), 15 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index be113b9c..74aee26c 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -28,7 +28,25 @@ ROOM_EVENT = "core-event-schema/room_event.json"
 STATE_EVENT = "core-event-schema/state_event.json"
 
 
-def get_json_schema_object_fields(obj, enforce_title=False):
+def resolve_references(path, schema):
+    if isinstance(schema, dict):
+        result = {}
+        for key, value in schema.items():
+            if key == "$ref":
+                path = os.path.join(os.path.dirname(path), value)
+                with open(path) as f:
+                    schema = json.load(f)
+                return resolve_references(path, schema)
+            else:
+                result[key] = resolve_references(path, value)
+        return result
+    elif isinstance(schema, list):
+        return [resolve_references(path, value) for value in schema]
+    else:
+        return schema
+
+
+def get_json_schema_object_fields(obj, enforce_title=False, include_parents=False):
     # Algorithm:
     # f.e. property => add field info (if field is object then recurse)
     if obj.get("type") != "object":
@@ -36,9 +54,9 @@ def get_json_schema_object_fields(obj, enforce_title=False):
             "get_json_schema_object_fields: Object %s isn't an object." % obj
         )
     if enforce_title and not obj.get("title"):
-        raise Exception(
-            "get_json_schema_object_fields: Nested object %s doesn't have a title." % obj
-        )
+        # Force a default titile of "NO_TITLE" to make it obvious in the
+        # specification output which parts of the schema are missing a title
+        obj["title"] = 'NO_TITLE'
 
     required_keys = obj.get("required")
     if not required_keys:
@@ -73,9 +91,15 @@ def get_json_schema_object_fields(obj, enforce_title=False):
             "Object %s has no properties or parents." % obj
         )
     if not props:  # parents only
+        if include_parents:
+            if obj["title"] == "NO_TITLE" and parents[0].get("title"):
+                obj["title"] = parents[0].get("title")
+            props = parents[0].get("properties")
+
+    if not props:
         return [{
             "title": obj["title"],
-            "parent": parents[0]["$ref"],
+            "parent": parents[0].get("$ref"),
             "no-table": True
         }]
 
@@ -91,7 +115,8 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                 if prop_val == "object":
                     nested_object = get_json_schema_object_fields(
                         props[key_name]["additionalProperties"],
-                        enforce_title=True
+                        enforce_title=True,
+                        include_parents=include_parents,
                     )
                     key = props[key_name]["additionalProperties"].get(
                         "x-pattern", "string"
@@ -103,8 +128,9 @@ def get_json_schema_object_fields(obj, enforce_title=False):
                     value_type = "{string: %s}" % prop_val
             else:
                 nested_object = get_json_schema_object_fields(
-                    props[key_name], 
-                    enforce_title=True
+                    props[key_name],
+                    enforce_title=True,
+                    include_parents=include_parents,
                 )
                 value_type = "{%s}" % nested_object[0]["title"]
 
@@ -114,8 +140,9 @@ def get_json_schema_object_fields(obj, enforce_title=False):
             # if the items of the array are objects then recurse
             if props[key_name]["items"]["type"] == "object":
                 nested_object = get_json_schema_object_fields(
-                    props[key_name]["items"], 
-                    enforce_title=True
+                    props[key_name]["items"],
+                    enforce_title=True,
+                    include_parents=include_parents,
                 )
                 value_type = "[%s]" % nested_object[0]["title"]
                 tables += nested_object
@@ -159,7 +186,7 @@ def get_json_schema_object_fields(obj, enforce_title=False):
 
 class MatrixUnits(Units):
 
-    def _load_swagger_meta(self, api, group_name):
+    def _load_swagger_meta(self, filepath, api, group_name):
         endpoints = []
         for path in api["paths"]:
             for method in api["paths"][path]:
@@ -262,7 +289,10 @@ class MatrixUnits(Units):
                             if is_array_of_objects:
                                 req_obj = req_obj["items"]
 
-                            req_tables = get_json_schema_object_fields(req_obj)
+                            req_tables = get_json_schema_object_fields(
+                                resolve_references(filepath, req_obj),
+                                include_parents=True,
+                            )
 
                             if req_tables > 1:
                                 for table in req_tables[1:]:
@@ -379,7 +409,10 @@ class MatrixUnits(Units):
                     elif res_type and Units.prop(good_response, "schema/properties"):
                         # response is an object:
                         schema = good_response["schema"]
-                        res_tables = get_json_schema_object_fields(schema)
+                        res_tables = get_json_schema_object_fields(
+                            resolve_references(filepath, schema),
+                            include_parents=True,
+                        )
                         for table in res_tables:
                             if "no-table" not in table:
                                 endpoint["res_tables"].append(table)
@@ -445,13 +478,16 @@ class MatrixUnits(Units):
                 if not filename.endswith(".yaml"):
                     continue
                 self.log("Reading swagger API: %s" % filename)
-                with open(os.path.join(path, filename), "r") as f:
+                filepath = os.path.join(path, filename)
+                with open(filepath, "r") as f:
                     # strip .yaml
                     group_name = filename[:-5].replace("-", "_")
                     if is_v2:
                         group_name = "v2_" + group_name
                     api = yaml.load(f.read())
-                    api["__meta"] = self._load_swagger_meta(api, group_name)
+                    api["__meta"] = self._load_swagger_meta(
+                        filepath, api, group_name
+                    )
                     apis[group_name] = api
         return apis
 

From 40fa339cf7790837f550cf1ac7fa580a8442ad80 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 2 Nov 2015 15:00:18 +0000
Subject: [PATCH 410/467] Draw a pretty (well at least I think it's pretty)
 ASCII diagram of the remote join handshake

---
 specification/server_server_api.rst | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index f0239128..66633376 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -578,8 +578,23 @@ construction, though in principle any valid candidate may be used at each time.
 Thus, any join handshake can potentially involve anywhere from two to four
 homeservers, though most in practice will use just two.
 
-.. TODO-doc
-  - Consider drawing a request/response diagram here
+::
+
+  Client         Joining                Directory       Resident
+                 Server                 Server          Server
+
+  join request -->
+                 |
+                 directory request ------->
+                 <---------- directory response
+                 |
+                 make_join request ----------------------->
+                 <------------------------------- make_join response
+                 |
+                 send_join request ----------------------->
+                 <------------------------------- send_join response
+                 |
+  <---------- join response
 
 The first part of the handshake involves using the directory server to request
 the room ID and join candidates. This is covered in more detail on the

From f6c55979e0cdf6881ca4211745b9fc5cd6f6fbb7 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 2 Nov 2015 15:17:18 +0000
Subject: [PATCH 411/467] Remove TODO comment about SYN-490 as it's unlikely to
 matter for v1; we'll fix it in v2

---
 specification/server_server_api.rst | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 66633376..d26cc26e 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -671,10 +671,6 @@ responds to the joining server with the full set of state for the newly-joined
 room. This is returned as a two-element list, whose first element is the
 integer 200, and whose second element contains the following keys:
 
-.. TODO-spec
-  - This is likely an implementation bug; see SYN-490. This should probably
-    actually just return the object directly
-
 ============== ===== ============
  Key            Type  Description
 ============== ===== ============

From 2e3a0b4e00798b672fe3282e24cc1e1c34cd8b0b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 2 Nov 2015 15:26:06 +0000
Subject: [PATCH 412/467] Specify guest accounts

---
 api/client-server/v2_alpha/registration.yaml | 17 +++++++++++++++++
 templating/build.py                          |  7 +++++++
 2 files changed, 24 insertions(+)

diff --git a/api/client-server/v2_alpha/registration.yaml b/api/client-server/v2_alpha/registration.yaml
index 2bd86e73..681654ae 100644
--- a/api/client-server/v2_alpha/registration.yaml
+++ b/api/client-server/v2_alpha/registration.yaml
@@ -17,7 +17,24 @@ paths:
       summary: Register for an account on this homeserver.
       description: |-
         Register for an account on this homeserver.
+
+        There are two kinds of user account:
+
+        - `user` accounts. These accounts may use the full API described in this specification.
+
+        - `guest` accounts. These accounts may have limited permissions and may not be supported by all servers.
+
       parameters:
+        - in: query
+          name: kind
+          type: string
+          x-example: guest
+          required: false
+          default: user
+          enum:
+            - guest
+            - user
+          description: The kind of account to register. Defaults to `user`.
         - in: body
           name: body
           schema:
diff --git a/templating/build.py b/templating/build.py
index b91e1da2..a35d8a08 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -84,6 +84,13 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
         input_lines = input.split('\n\n')
         wrapper = TextWrapper(initial_indent=initial_indent, width=wrap)
         output_lines = [wrapper.fill(line) for line in input_lines]
+
+        for i in range(len(output_lines)):
+            line = output_lines[i]
+            in_bullet = line.startswith("- ")
+            if in_bullet:
+                output_lines[i] = line.replace("\n", "\n  " + initial_indent)
+
         return '\n\n'.join(output_lines)
 
     # make Jinja aware of the templates and filters

From ebc02371095be8789051ac0a47de99803f961606 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 3 Nov 2015 19:35:44 +0000
Subject: [PATCH 413/467] Add the missing titles to the schema

---
 .../v1/definitions/push_rule.json             |  3 +-
 .../v1/definitions/push_ruleset.json          | 37 +++++++++++--------
 .../v2_alpha/definitions/event_batch.json     |  1 +
 .../schema/v1/core-event-schema/event.json    |  1 +
 .../v1/core-event-schema/state_event.json     |  1 +
 event-schemas/schema/v1/m.room.member         |  1 +
 6 files changed, 27 insertions(+), 17 deletions(-)

diff --git a/api/client-server/v1/definitions/push_rule.json b/api/client-server/v1/definitions/push_rule.json
index 4df93f67..d10f11f8 100644
--- a/api/client-server/v1/definitions/push_rule.json
+++ b/api/client-server/v1/definitions/push_rule.json
@@ -1,4 +1,5 @@
 {
+  "title": "PushRule",
   "type": "object", 
   "properties": {
     "default": {
@@ -17,4 +18,4 @@
       "type": "array"
     }
   }
-}
\ No newline at end of file
+}
diff --git a/api/client-server/v1/definitions/push_ruleset.json b/api/client-server/v1/definitions/push_ruleset.json
index e0372701..529ebeed 100644
--- a/api/client-server/v1/definitions/push_ruleset.json
+++ b/api/client-server/v1/definitions/push_ruleset.json
@@ -1,60 +1,65 @@
 {
-  "type": "object", 
+  "type": "object",
   "properties": {
     "content": {
       "items": {
-        "type": "object", 
+        "type": "object",
+        "title": "PushRule",
         "allOf": [
           {
             "$ref": "push_rule.json"
           }
         ]
-      }, 
+      },
       "type": "array"
-    }, 
+    },
     "override": {
       "items": {
-        "type": "object", 
+        "type": "object",
+        "title": "PushRule",
         "allOf": [
           {
             "$ref": "push_rule.json"
           }
         ]
-      }, 
+      },
       "type": "array"
-    }, 
+    },
     "sender": {
       "items": {
-        "type": "object", 
+        "type": "object",
+        "title": "PushRule",
         "allOf": [
           {
             "$ref": "push_rule.json"
           }
         ]
-      }, 
+      },
       "type": "array"
-    }, 
+    },
     "underride": {
       "items": {
-        "type": "object", 
+        "type": "object",
+        "title": "PushRule",
         "allOf": [
           {
             "$ref": "push_rule.json"
           }
         ]
-      }, 
+      },
       "type": "array"
-    }, 
+    },
     "room": {
       "items": {
-        "type": "object", 
+        "type": "object",
+        "title": "PushRule",
         "allOf": [
           {
             "$ref": "push_rule.json"
           }
         ]
-      }, 
+      },
       "type": "array"
     }
   }
-}
\ No newline at end of file
+}
diff --git a/api/client-server/v2_alpha/definitions/event_batch.json b/api/client-server/v2_alpha/definitions/event_batch.json
index 75762d75..395aed13 100644
--- a/api/client-server/v2_alpha/definitions/event_batch.json
+++ b/api/client-server/v2_alpha/definitions/event_batch.json
@@ -5,6 +5,7 @@
       "type": "array",
       "description": "List of events",
       "items": {
+        "title": "Event",
         "type": "object"
       }
     }
diff --git a/event-schemas/schema/v1/core-event-schema/event.json b/event-schemas/schema/v1/core-event-schema/event.json
index e73aec80..f9103715 100644
--- a/event-schemas/schema/v1/core-event-schema/event.json
+++ b/event-schemas/schema/v1/core-event-schema/event.json
@@ -5,6 +5,7 @@
   "properties": {
     "content": {
       "type": "object",
+      "title": "EventContent",
       "description": "The fields in this object will vary depending on the type of event. When interacting with the REST API, this is the HTTP body."
     },
     "type": {
diff --git a/event-schemas/schema/v1/core-event-schema/state_event.json b/event-schemas/schema/v1/core-event-schema/state_event.json
index 88b4900a..5809cf7f 100644
--- a/event-schemas/schema/v1/core-event-schema/state_event.json
+++ b/event-schemas/schema/v1/core-event-schema/state_event.json
@@ -11,6 +11,7 @@
       "description": "A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event. The key MUST NOT start with '_'."
     },
     "prev_content": {
+      "title": "EventContent",
       "type": "object",
       "description": "Optional. The previous ``content`` for this event. If there is no previous content, this key will be missing."
     }
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 45f2ad70..0bd8cd1e 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -8,6 +8,7 @@
     "properties": {
         "content": {
             "type": "object",
+            "title": "EventContent",
             "properties": {
                 "membership": {
                     "type": "string",

From 8322151661f8c8a7cb6cfa3fe1d22ab83ce55642 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Tue, 3 Nov 2015 19:42:49 +0000
Subject: [PATCH 414/467] Don't put a space when appending the "Must be"
 strings to the desciption if there isn't a description, otherwise it will
 mess up the indent

---
 templating/matrix_templates/units.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 74aee26c..9aa9489a 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -164,12 +164,16 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
             if props[key_name].get("enum"):
                 if len(props[key_name].get("enum")) > 1:
                     value_type = "enum"
+                    if desc:
+                        desc += " "
                     desc += (
-                        " One of: %s" % json.dumps(props[key_name]["enum"])
+                        "One of: %s" % json.dumps(props[key_name]["enum"])
                     )
                 else:
+                    if desc:
+                        desc += " "
                     desc += (
-                        " Must be '%s'." % props[key_name]["enum"][0]
+                        "Must be '%s'." % props[key_name]["enum"][0]
                     )
             if isinstance(value_type, list):
                 value_type = " or ".join(value_type)

From e49ea9015f4ee5f86b2d401190ec39517c5ae49b Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Wed, 4 Nov 2015 11:39:36 +0000
Subject: [PATCH 415/467] Deduplicate tables with the same title

---
 templating/matrix_templates/units.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 9aa9489a..9cc68f49 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -185,7 +185,17 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
             "desc": desc,
             "req_str": "**Required.** " if required else ""
         })
-    return tables
+
+    titles = set()
+    filtered = []
+    for table in tables:
+        if table.get("title") in titles:
+            continue
+
+        titles.add(table.get("title"))
+        filtered.append(table)
+
+    return filtered
 
 
 class MatrixUnits(Units):

From 8070489080afdc08179087c64b1f6a4a261c224d Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Wed, 4 Nov 2015 11:44:20 +0000
Subject: [PATCH 416/467] Handle lists of types in arrays

---
 templating/matrix_templates/units.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index 9cc68f49..cec5b83e 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -147,7 +147,10 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
                 value_type = "[%s]" % nested_object[0]["title"]
                 tables += nested_object
             else:
-                value_type = "[%s]" % props[key_name]["items"]["type"]
+                value_type = props[key_name]["items"]["type"]
+                if isinstance(value_type, list):
+                    value_type = " or ".join(value_type)
+                value_type = "[%s]" % value_type
                 array_enums = props[key_name]["items"].get("enum")
                 if array_enums:
                     if len(array_enums) > 1:

From c7199463704709ca4c0dc8a3b60572210f464c84 Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Wed, 4 Nov 2015 11:51:50 +0000
Subject: [PATCH 417/467] Enable syntax highlighting for message type examples

---
 templating/matrix_templates/templates/msgtypes.tmpl | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/templating/matrix_templates/templates/msgtypes.tmpl b/templating/matrix_templates/templates/msgtypes.tmpl
index f7862451..18d3492b 100644
--- a/templating/matrix_templates/templates/msgtypes.tmpl
+++ b/templating/matrix_templates/templates/msgtypes.tmpl
@@ -18,6 +18,8 @@
 ================== ================= ===========================================
 
 {% endfor %}
-Example::
+Example:
+
+.. code:: json
 
     {{example | jsonify(4, 4)}}

From 2fd5fc39a032dbce80a3aba4361f79f553cc5b85 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Thu, 5 Nov 2015 10:55:11 +0000
Subject: [PATCH 418/467] Add spec for calculating display names for rooms and
 users

Merged from https://github.com/matrix-org/matrix-doc/pull/145
---
 specification/modules/instant_messaging.rst | 125 +++++++++++++++++++-
 1 file changed, 124 insertions(+), 1 deletion(-)

diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 09fcb843..7a2142dd 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -116,12 +116,58 @@ of the client-server API will resolve this by attaching the transaction ID of th
 sending request to the event itself.
 
 
+Calculating the display name for a user
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Clients may wish to show the human-readable display name of a room member as
+part of a membership list, or when they send a message. However, different
+members may have conflicting display names. Display names MUST be disambiguated
+before showing them to the user, in order to prevent spoofing of other users.
+
+To ensure this is done consistently across clients, clients SHOULD use the
+following algorithm to calculate a disambiguated display name for a given user:
+
+1. Inspect the ``m.room.member`` state event for the relevant user id.
+2. If the ``m.room.member`` state event has no ``displayname`` field, or if
+   that field has a ``null`` value, use the raw user id as the display
+   name. Otherwise:
+3. If the ``m.room.member`` event has a ``displayname`` which is unique among
+   members of the room with ``membership: join`` or ``membership: invite``, use
+   the given ``displayname`` as the user-visible display name. Otherwise:
+4. The ``m.room.member`` event has a non-unique ``displayname``. This should be
+   disambiguated using the user id, for example "display name
+   (@id:homeserver.org)".
+
+Developers should take note of the following when implementing the above
+algorithm:
+
+* The user-visible display name of one member can be affected by changes in the
+  state of another member. For example, if ``@user1:matrix.org`` is present in
+  a room, with ``displayname: Alice``, then when ``@user2:example.com`` joins
+  the room, also with ``displayname: Alice``, *both* users must be given
+  disambiguated display names. Similarly, when one of the users then changes
+  their display name, there is no longer a clash, and *both* users can be given
+  their chosen display name. Clients should be alert to this possibility and
+  ensure that all affected users are correctly renamed.
+
+* The display name of a room may also be affected by changes in the membership
+  list. This is due to the room name sometimes being based on user display
+  names (see `Calculating the display name for a room`_).
+
+* If the entire membership list is searched for clashing display names, this
+  leads to an O(N^2) implementation for building the list of room members. This
+  will be very inefficient for rooms with large numbers of members. It is
+  recommended that client implementations maintain a hash table mapping from
+  ``displayname`` to a list of room members using that name. Such a table can
+  then be used for efficient calculation of whether disambiguation is needed.
+
+
 Displaying membership information with messages
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Clients may wish to show the display name and avatar URL of the room member who
 sent a message. This can be achieved by inspecting the ``m.room.member`` state
-event for that user ID.
+event for that user ID (see `Calculating the display name for a user`_).
 
 When a user paginates the message history, clients may wish to show the
 **historical** display name and avatar URL for a room member. This is possible
@@ -133,6 +179,83 @@ events update the old state. When paginated events are processed sequentially,
 the old state represents the state of the room *at the time the event was sent*.
 This can then be used to set the historical display name and avatar URL.
 
+
+Calculating the display name for a room
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Clients may wish to show a human-readable name for a room. There are a number
+of possibilities for choosing a useful name. To ensure that rooms are named
+consistently across clients, clients SHOULD use the following algorithm to
+choose a name:
+
+1. If the room has an `m.room.name`_ state event, use the name given by that
+   event.
+
+#. If the room has an `m.room.canonical_alias`_ state event, use the alias
+   given by that event.
+
+#. If neither of the above events are present, a name should be composed based
+   on the members of the room. Clients should consider `m.room.member`_ events
+   for users other than the logged-in user, with ``membership: join`` or
+   ``membership: invite``.
+
+   .. _active_members:
+
+   i. If there is only one such event, the display name for the room should be
+      the `disambiguated display name`_ of the corresponding user.
+
+   #. If there are two such events, they should be lexicographically sorted by
+      their ``state_key`` (i.e. the corresponding user IDs), and the display
+      name for the room should be the  `disambiguated display name`_ of both
+      users: "<user1> and <user2>", or a localised variant thereof.
+
+   #. If there are three or more such events, the display name for the room
+      should be based on the disambiguated display name of the user
+      corresponding to the first such event, under a lexicographical sorting
+      according to their ``state_key``. The display name should be in the
+      format "<user1> and <N> others" (or a localised variant thereof), where N
+      is the number of `m.room.member`_ events with ``membership: join`` or
+      ``membership: invite``, excluding the logged-in user and "user1".
+
+      For example, if Alice joins a room, where Bob (whose user id is
+      ``@superuser:example.com``), Carol (user id ``@carol:example.com``) and
+      Dan (user id ``@dan:matrix.org``) are in conversation, Alice's
+      client should show the room name as "Carol and 2 others".
+
+   .. TODO-spec
+     Sorting by user_id certainly isn't ideal, as IDs at the start of the
+     alphabet will end up dominating room names: they will all be called
+     "Arathorn and 15 others". Furthermore - user_ids are not necessarily
+     ASCII, which means we need to either specify a collation order, or specify
+     how to choose one.
+
+     Ideally we might sort by the time when the user was first invited to, or
+     first joined the room. But we don't have this information.
+
+#. If the room has no ``m.room.name`` or ``m.room.canonical_alias`` events, and
+   no active members other than the current user, clients should consider
+   ``m.room.member`` events with ``membership: leave``. If such events exist, a
+   display name such as "Empty room (was <user1> and <N> others)" (or a
+   localised variant thereof) should be used, following similar rules as for
+   active members (see `above <active_members_>`_).
+
+#. A complete absence of ``m.room.name``, ``m.room.canonical_alias``, and
+   ``m.room.member`` events is likely to indicate a problem with creating the
+   room or synchronising the state table; however clients should still handle
+   this situation. A display name such as "Empty room" (or a localised variant
+   thereof) should be used in this situation.
+
+.. _`disambiguated display name`: `Calculating the display name for a user`_
+
+Clients SHOULD NOT use `m.room.aliases`_ events as a source for room names, as
+it is difficult for clients to agree on the best alias to use, and aliases can
+change unexpectedly.
+
+.. TODO-spec
+  How can we make this less painful for clients to implement, without forcing
+  an English-language implementation on them all?
+
+
 Server behaviour
 ----------------
 

From 8cba11b1cd6f70906aa3f8ce03d4332d469af13f Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Thu, 5 Nov 2015 11:06:31 +0000
Subject: [PATCH 419/467] Add some links to spec bugs for display names

Just added a couple of TODO comments to useful jira bugs
---
 specification/modules/instant_messaging.rst | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst
index 7a2142dd..a58c762f 100644
--- a/specification/modules/instant_messaging.rst
+++ b/specification/modules/instant_messaging.rst
@@ -138,6 +138,11 @@ following algorithm to calculate a disambiguated display name for a given user:
    disambiguated using the user id, for example "display name
    (@id:homeserver.org)".
 
+   .. TODO-spec
+     what does it mean for a ``displayname`` to be 'unique'? Are we
+     case-sensitive?  Do we care about homograph attacks? See
+     https://matrix.org/jira/browse/SPEC-221.
+
 Developers should take note of the following when implementing the above
 algorithm:
 
@@ -232,6 +237,8 @@ choose a name:
      Ideally we might sort by the time when the user was first invited to, or
      first joined the room. But we don't have this information.
 
+     See https://matrix.org/jira/browse/SPEC-267 for further discussion.
+
 #. If the room has no ``m.room.name`` or ``m.room.canonical_alias`` events, and
    no active members other than the current user, clients should consider
    ``m.room.member`` events with ``membership: leave``. If such events exist, a

From 161441fa3ae83db985c51c9f1c8a4cb96494bd45 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 5 Nov 2015 18:11:20 +0000
Subject: [PATCH 420/467] Update 3pid spec based on new implementation

---
 api/client-server/v1/membership.yaml          | 110 +--------------
 .../v1/third_party_membership.yaml            | 127 ++++++++++++++++++
 event-schemas/examples/v1/m.room.member       |  17 +--
 event-schemas/schema/v1/m.room.member         |  20 +--
 specification/modules/anonymous_access.rst    |  50 +++++++
 specification/modules/third_party_invites.rst |  85 ++++++++----
 6 files changed, 246 insertions(+), 163 deletions(-)
 create mode 100644 api/client-server/v1/third_party_membership.yaml
 create mode 100644 specification/modules/anonymous_access.rst

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/membership.yaml
index f8dfdea5..c089e154 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/membership.yaml
@@ -73,9 +73,12 @@ paths:
     post:
       summary: Invite a user to participate in a particular room.
       description: |-
+        .. _invite-by-user-id-endpoint:
+
         *Note that there are two forms of this API, which are documented separately.
         This version of the API requires that the inviter knows the Matrix
-        identifier of the invitee.*
+        identifier of the invitee. The other is documented in the*
+        `third party invites section`_.
 
         This API invites a user to participate in a particular room.
         They do not start participating in the room until they actually join the
@@ -86,6 +89,8 @@ paths:
 
         If the user was invited to the room, the home server will append a
         ``m.room.member`` event to the room.
+
+        .. _third party invites section: `invite-by-third-party-id-endpoint`_
       security:
         - accessToken: []
       parameters:
@@ -132,106 +137,3 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
-
-  "/rooms/{roomId}/invite":
-    post:
-      summary: Invite a user to participate in a particular room.
-      description: |-
-        *Note that there are two forms of this API, which are documented separately.
-        This version of the API does not require that the inviter know the Matrix
-        identifier of the invitee, and instead relies on third party identifiers.
-        The homeserver uses an identity server to perform the mapping from
-        third party identifier to a Matrix identifier.*
-
-        This API invites a user to participate in a particular room.
-        They do not start participating in the room until they actually join the
-        room.
-
-        Only users currently in a particular room can invite other users to
-        join that room.
-
-        If the identity server did know the Matrix user identifier for the
-        third party identifier, the home server will append a ``m.room.member``
-        event to the room.
-
-        If the identity server does not know a Matrix user identifier for the
-        passed third party identifier, the homeserver will issue an invitation
-        which can be accepted upon providing proof of ownership of the third
-        party identifier. This is achieved by the identity server generating a
-        token, which it gives to the inviting homeserver. The homeserver will
-        add an ``m.room.third_party_invite`` event into the graph for the room,
-        containing that token.
-
-        When the invitee binds the invited third party identifier to a Matrix
-        user ID, the identity server will give the user a list of pending
-        invitations, each containing:
-
-        - The room ID to which they were invited
-
-        - The token given to the homeserver
-
-        - A signature of the token, signed with the identity server's private key
-
-        - The matrix user ID who invited them to the room
-
-        If a token is requested from the identity server, the home server will
-        append a ``m.room.third_party_invite`` event to the room.
-      security:
-        - accessToken: []
-      parameters:
-        - in: path
-          type: string
-          name: roomId
-          description: The room identifier (not alias) to which to invite the user.
-          required: true
-          x-example: "!d41d8cd:matrix.org"
-        - in: body
-          name: body
-          required: true
-          schema:
-            type: object
-            example: |-
-              {
-                "id_server": "matrix.org",
-                "medium": "email",
-                "address": "cheeky@monkey.com",
-                "display_name": "A very cheeky monkey"
-              }
-            properties:
-              id_server:
-                type: string
-                description: The hostname+port of the identity server which should be used for third party identifier lookups.
-              medium:
-                type: string
-                # TODO: Link to identity service spec when it eixsts
-                description: The kind of address being passed in the address field, for example ``email``.
-              address:
-                type: string
-                description: The invitee's third party identifier.
-              display_name:
-                type: string
-                description: A user-friendly string describing who has been invited. It should not contain the address of the invitee, to avoid leaking mappings between third party identities and matrix user IDs.
-            required: ["id_server", "medium", "address", "display_name"]
-      responses:
-        200:
-          description: The user has been invited to join the room.
-          examples:
-            application/json: |-
-              {}
-          schema:
-            type: object
-        403:
-          description: |-
-            You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
-
-            - The invitee has been banned from the room.
-            - The invitee is already a member of the room.
-            - The inviter is not currently in the room.
-            - The inviter's power level is insufficient to invite users to the room.
-          examples:
-            application/json: |-
-              {"errcode": "M_FORBIDDEN", "error": "@cheeky_monkey:matrix.org is banned from the room"}
-        429:
-          description: This request was rate-limited.
-          schema:
-            "$ref": "definitions/error.yaml"
diff --git a/api/client-server/v1/third_party_membership.yaml b/api/client-server/v1/third_party_membership.yaml
new file mode 100644
index 00000000..90b167b4
--- /dev/null
+++ b/api/client-server/v1/third_party_membership.yaml
@@ -0,0 +1,127 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Room Membership API for third party identifiers"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/invite":
+    post:
+      summary: Invite a user to participate in a particular room.
+      description: |-
+        .. _invite-by-third-party-id-endpoint:
+
+        *Note that there are two forms of this API, which are documented separately.
+        This version of the API does not require that the inviter know the Matrix
+        identifier of the invitee, and instead relies on third party identifiers.
+        The homeserver uses an identity server to perform the mapping from
+        third party identifier to a Matrix identifier. The other is documented in the*
+        `joining rooms section`_.
+
+        This API invites a user to participate in a particular room.
+        They do not start participating in the room until they actually join the
+        room.
+
+        Only users currently in a particular room can invite other users to
+        join that room.
+
+        If the identity server did know the Matrix user identifier for the
+        third party identifier, the home server will append a ``m.room.member``
+        event to the room.
+
+        If the identity server does not know a Matrix user identifier for the
+        passed third party identifier, the homeserver will issue an invitation
+        which can be accepted upon providing proof of ownership of the third
+        party identifier. This is achieved by the identity server generating a
+        token, which it gives to the inviting homeserver. The homeserver will
+        add an ``m.room.third_party_invite`` event into the graph for the room,
+        containing that token.
+
+        When the invitee binds the invited third party identifier to a Matrix
+        user ID, the identity server will give the user a list of pending
+        invitations, each containing:
+
+        - The room ID to which they were invited
+
+        - The token given to the homeserver
+
+        - A signature of the token, signed with the identity server's private key
+
+        - The matrix user ID who invited them to the room
+
+        If a token is requested from the identity server, the home server will
+        append a ``m.room.third_party_invite`` event to the room.
+
+        .. _joining rooms section: `invite-by-user-id-endpoint`_
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier (not alias) to which to invite the user.
+          required: true
+          x-example: "!d41d8cd:matrix.org"
+        - in: body
+          name: body
+          required: true
+          schema:
+            type: object
+            example: |-
+              {
+                "id_server": "matrix.org",
+                "medium": "email",
+                "address": "cheeky@monkey.com",
+                "display_name": "A very cheeky monkey"
+              }
+            properties:
+              id_server:
+                type: string
+                description: The hostname+port of the identity server which should be used for third party identifier lookups.
+              medium:
+                type: string
+                # TODO: Link to identity service spec when it eixsts
+                description: The kind of address being passed in the address field, for example ``email``.
+              address:
+                type: string
+                description: The invitee's third party identifier.
+              display_name:
+                type: string
+                description: A user-friendly string describing who has been invited. It should not contain the address of the invitee, to avoid leaking mappings between third party identities and matrix user IDs.
+            required: ["id_server", "medium", "address", "display_name"]
+      responses:
+        200:
+          description: The user has been invited to join the room.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
+        403:
+          description: |-
+            You do not have permission to invite the user to the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejections are:
+
+            - The invitee has been banned from the room.
+            - The invitee is already a member of the room.
+            - The inviter is not currently in the room.
+            - The inviter's power level is insufficient to invite users to the room.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN", "error": "@cheeky_monkey:matrix.org is banned from the room"}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
diff --git a/event-schemas/examples/v1/m.room.member b/event-schemas/examples/v1/m.room.member
index 2c174e8a..e2ca5668 100644
--- a/event-schemas/examples/v1/m.room.member
+++ b/event-schemas/examples/v1/m.room.member
@@ -3,22 +3,7 @@
   "content": {
     "membership": "join",
     "avatar_url": "mxc://localhost/SEsfnsuifSDFSSEF#auto",
-    "displayname": "Alice Margatroid",
-    "third_party_invite": {
-        "token": "pc98",
-        "public_key": "abc123",
-        "key_validity_url": "https://magic.forest/verifykey",
-        "signed": {
-          "mxid": "@alice:localhost",
-          "token": "pc98",
-          "signatures": {
-            "magic.forest": {
-              "ed25519:0": "poi098"
-            }
-          }
-        },
-        "sender": "@zun:zun.soft"
-    }
+    "displayname": "Alice Margatroid"
   },
   "invite_room_state": [
     {
diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 0bd8cd1e..81057049 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "The current membership state of a user in the room.",
-    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if the invite was an ``m.room.third_party_invite`` event, and absent if the invite was an ``m.room.member`` event.\n\nThis event also includes an ``invite_room_state`` key **outside the** ``content`` **key**. This contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
+    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event also includes an ``invite_room_state`` key **outside the** ``content`` **key**. This contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
     "allOf": [{
         "$ref": "core-event-schema/state_event.json"
     }],
@@ -27,18 +27,6 @@
                     "type": "object",
                     "title": "Invite",
                     "properties": {
-                        "token": {
-                            "type": "string",
-                            "description": "A token which must be correctly signed, in order to join the room."
-                        },
-                        "key_validity_url": {
-                            "type": "string",
-                            "description": "A URL which can be fetched, with querystring ``public_key=public_key``, to validate whether the key has been revoked. The URL must return a JSON object containing a boolean property named 'valid'."
-                        },
-                        "public_key": {
-                            "type": "string",
-                            "description": "A base64-encoded ed25519 key with which token must be signed."
-                        },
                         "signed": {
                             "type": "object",
                             "title": "signed",
@@ -58,13 +46,9 @@
                                 }
                             },
                             "required": ["mxid", "signatures", "token"]
-                        },
-                        "sender": {
-                            "type": "string",
-                            "description": "The matrix user ID of the user who send the invite which is being used."
                         }
                     },
-                    "required": ["token", "key_validity_url", "public_key", "sender", "signed"]
+                    "required": ["signed"]
                 }
             },
             "required": ["membership"]
diff --git a/specification/modules/anonymous_access.rst b/specification/modules/anonymous_access.rst
new file mode 100644
index 00000000..5a421187
--- /dev/null
+++ b/specification/modules/anonymous_access.rst
@@ -0,0 +1,50 @@
+Guest access
+================
+
+.. _module:guest-access:
+
+It may be desirable to allow users without a fully registered user account to
+ephemerally access Matrix rooms. This module specifies limited ways of doing so.
+
+Note that this is not currently a complete anonymous access solution; in
+particular, it only allows servers to provided anonymous access to rooms in
+which they are already participating, and relies on individual homeservers to
+adhere to the conventions which this module sets, rather than allowing all
+participating homeservers to enforce them.
+
+Events
+------
+
+{{m_room_guest_accessibility}}
+
+Client behaviour
+----------------
+A client can register for guest access using the FOO endpoint. From that point
+on, they can interact with a limited subset of the existing client-server API,
+as if they were a fully registered user, using the access token granted to them
+by the server.
+
+These users are only allowed to make calls in relation to rooms which have the
+``m.room.history_visibility`` event set to ``world_readable``.
+
+The APIs they are allowed to hit are:
+
+/rooms/{roomId}/messages
+/rooms/{roomId}/state
+/rooms/{roomId}/state/{eventType}/{stateKey}
+/events
+
+Server behaviour
+----------------
+Does the server need to handle any of the new events in a special way (e.g.
+typing timeouts, presence). Advice on how to persist events and/or requests are
+recommended to aid implementation. Federation-specific logic should be included
+here.
+
+Security considerations
+-----------------------
+This includes privacy leaks: for example leaking presence info. How do
+misbehaving clients or servers impact this module? This section should always be
+included, if only to say "we've thought about it but there isn't anything to do
+here".
+
diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index 85538c31..4d268631 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -1,27 +1,30 @@
 Third party invites
 ===================
 
-.. _module:third_party_invites:
+.. _module:third-party-invites:
 
 This module adds in support for inviting new members to a room where their
 Matrix user ID is not known, instead addressing them by a third party identifier
 such as an email address.
-
 There are two flows here; one if a Matrix user ID is known for the third party
 identifier, and one if not. Either way, the client calls ``/invite`` with the
 details of the third party identifier.
 
 The homeserver asks the identity server whether a Matrix user ID is known for
-that identifier. If it is, an invite is simply issued for that user.
+that identifier:
+
+- If it is, an invite is simply issued for that user.
 
-If it is not, the homeserver asks the identity server to record the details of
-the invitation, and to notify the client of this pending invitation if it gets
-a binding for this identifier in the future. The identity server returns a token
-and public key to the homeserver.
+- If it is not, the homeserver asks the identity server to record the details of
+  the invitation, and to notify the invitee's homeserver of this pending invitation if it gets
+  a binding for this identifier in the future. The identity server returns a token
+  and public key to the inviting homeserver.
 
-If a client then tries to join the room in the future, it will be allowed to if
-it presents both the token, and a signature of that token from the identity
-server which can be verified with the public key.
+When the invitee's homeserver receives the notification of the binding, it
+should insert an ``m.room.member`` event into the room's graph for that user,
+with ``content.membership`` = ``invite``, as well as a
+``content.third_party_invite`` property which contains proof that the invitee
+does indeed own that third party identifier.
 
 Events
 ------
@@ -33,15 +36,18 @@ Client behaviour
 
 A client asks a server to invite a user by their third party identifier.
 
+{{third_party_membership_http_api}}
+
 Server behaviour
 ----------------
 
 All homeservers MUST verify the signature in the event's
 ``content.third_party_invite.signed`` object.
 
-If a client of the current homeserver is joining by an
-``m.room.third_party_invite``, that homesever MUST validate that the public
-key used for signing is still valid, by checking ``key_validity_url``. It does
+When a homeserver inserts an ``m.room.member`` ``invite`` event into the graph
+because of an ``m.room.third_party_invite`` event,
+that homesever MUST validate that the public
+key used for signing is still valid, by checking ``key_validity_url`` from the ``m.room.third_party_invite``. It does
 this by making an HTTP GET request to ``key_validity_url``:
 
 .. TODO: Link to identity server spec when it exists
@@ -84,24 +90,24 @@ membership is questionable.
 
 For example:
 
-    If room R has two participating homeservers, H1, H2
+#. Room R has two participating homeservers, H1, H2
 
-    And user A on H1 invites a third party identifier to room R
+#. User A on H1 invites a third party identifier to room R
 
-    H1 asks the identity server for a binding to a Matrix user ID, and has none,
-    so issues an ``m.room.third_party_invite`` event to the room.
+#. H1 asks the identity server for a binding to a Matrix user ID, and has none,
+   so issues an ``m.room.third_party_invite`` event to the room.
 
-    When the third party user validates their identity, they are told about the
-    invite, and ask their homeserver, H3, to join the room.
+#. When the third party user validates their identity, their homeserver H3
+   is notified and attempts to issue an ``m.room.member`` event to participate
+   in the room.
 
-    H3 validates the signature in the event's
-    ``content.third_party_invite.signed`` object.
+#. H3 validates the signature given to it by the identity server.
 
-    H3 then asks H1 to join it to the room. H1 *must* validate the ``signed``
-    property *and* check ``key_validity_url``.
+#. H3 then asks H1 to join it to the room. H1 *must* validate the ``signed``
+   property *and* check ``key_validity_url``.
 
-    Having validated these things, H1 writes the join event to the room, and H3
-    begins participating in the room. H2 *must* accept this event.
+#. Having validated these things, H1 writes the invite event to the room, and H3
+   begins participating in the room. H2 *must* accept this event.
 
 The reason that no other homeserver may reject the event based on checking
 ``key_validity_url`` is that we must ensure event acceptance is deterministic.
@@ -112,3 +118,32 @@ This relies on participating servers trusting each other, but that trust is
 already implied by the server-server protocol. Also, the public key signature
 verification must still be performed, so the attack surface here is minimized.
 
+Security considerations
+-----------------------
+
+There are a number of privary and trust implications to this module.
+
+It is important for user privacy that leaking the mapping between a matrix user
+ID and a third party identifier is hard. In particular, being able to look up
+all third party identifiers from a matrix user ID (and accordingly, being able
+to link each third party identifier) should be avoided wherever possible.
+To this end, when implementing this API care should be taken to avoid
+adding links between these two identifiers as room events. This mapping can be
+unintentionally created by specifying the third party identifier in the
+``display_name`` field of the ``m.room.third_party_invite`` event, and then
+observing which matrix user ID joins the room using that invite. Clients SHOULD
+set ``display_name`` to a value other than the third party identifier, e.g. the
+invitee's common name.
+
+Homeservers are not required to trust any particular identity server(s). It is
+generally a client's responsibility to decide which identity servers it trusts,
+not a homeserver's. Accordingly, this API takes identity servers as input from
+end users, and doesn't have any specific trusted set. It is possible some
+homeservers may want to supply defaults, or reject some identity servers for
+*its* users, but no homeserver is allowed to dictate which identity servers
+*other* homeservers' users trust.
+
+There is some risk of denial of service attacks by flooding homeservers or
+identity servers with many requests, or much state to store. Defending against
+these is left to the implementer's discretion.
+

From 559747e77a1bac4cd104f9526bfcaa84217651fb Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 5 Nov 2015 19:18:28 +0000
Subject: [PATCH 421/467] speculator: Sent Content-Type: text/html header

Go is auto-detecting that this is XML (because for some reason we
generate XHTML), and serving it with a Content-Type header text/xml.

This causes the browser to render it as XHTML, which gives interesting
quirks like extra newlines.

This forces the browser to interpret it as HTML.

What we should probably do instead of stop generating XHTML and start
generating HTML. But in the mean time, this will fix the rendering
issues.
---
 scripts/speculator/main.go | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 85fb2596..ef37b931 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -384,9 +384,9 @@ func main() {
 		log.Fatal(err)
 	}
 	s := server{masterCloneDir}
-	http.HandleFunc("/spec/", s.serveSpec)
-	http.HandleFunc("/diff/rst/", s.serveRSTDiff)
-	http.HandleFunc("/diff/html/", s.serveHTMLDiff)
+	http.HandleFunc("/spec/", forceHTML(s.serveSpec))
+	http.HandleFunc("/diff/rst/", forceHTML(s.serveRSTDiff))
+	http.HandleFunc("/diff/html/", forceHTML(s.serveHTMLDiff))
 	http.HandleFunc("/healthz", serveText("ok"))
 	http.HandleFunc("/", listPulls)
 
@@ -394,6 +394,13 @@ func main() {
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))
 }
 
+func forceHTML(h func(w http.ResponseWriter, req *http.Request)) func(w http.ResponseWriter, req *http.Request) {
+	return func(w http.ResponseWriter, req *http.Request) {
+		w.Header().Set("Content-Type", "text/html")
+		h(w, req)
+	}
+}
+
 func serveText(s string) func(http.ResponseWriter, *http.Request) {
 	return func(w http.ResponseWriter, req *http.Request) {
 		io.WriteString(w, s)

From 937ff046d866b96d068c3e1424689ebb58de99f5 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 5 Nov 2015 19:21:16 +0000
Subject: [PATCH 422/467] Force / to be HTML too

---
 scripts/speculator/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index ef37b931..7f86bd62 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -388,7 +388,7 @@ func main() {
 	http.HandleFunc("/diff/rst/", forceHTML(s.serveRSTDiff))
 	http.HandleFunc("/diff/html/", forceHTML(s.serveHTMLDiff))
 	http.HandleFunc("/healthz", serveText("ok"))
-	http.HandleFunc("/", listPulls)
+	http.HandleFunc("/", forceHTML(listPulls))
 
 	fmt.Printf("Listening on port %d\n", *port)
 	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), nil))

From 7f6eafdce52972021643512d09e46eaef88c810d Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 6 Nov 2015 14:46:55 +0000
Subject: [PATCH 423/467] continuserv: set Content-Type header

---
 scripts/continuserv/main.go | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index 573c2c95..425a1af9 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -113,6 +113,7 @@ func filter(e fsnotify.Event) bool {
 func serve(w http.ResponseWriter, req *http.Request) {
 	wg.Wait()
 	b := toServe.Load().([]byte)
+	w.Header().Set("Content-Type", "text/html")
 	w.Write(b)
 }
 

From 1be5b856bd2447aa557ada4fcde8b8ddf63a3f70 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 6 Nov 2015 16:05:07 +0000
Subject: [PATCH 424/467] Preserve text/plain for errors

Newlines are nice
---
 scripts/continuserv/main.go | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index 425a1af9..4613437a 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -25,7 +25,7 @@ import (
 var (
 	port = flag.Int("port", 8000, "Port on which to serve HTTP")
 
-	toServe atomic.Value   // Always contains valid []byte to serve. May be stale unless wg is zero.
+	toServe atomic.Value   // Always contains valid bytesOrErr to serve. May be stale unless wg is zero.
 	wg      sync.WaitGroup // Indicates how many updates are pending.
 	mu      sync.Mutex     // Prevent multiple updates in parallel.
 )
@@ -112,9 +112,14 @@ func filter(e fsnotify.Event) bool {
 
 func serve(w http.ResponseWriter, req *http.Request) {
 	wg.Wait()
-	b := toServe.Load().([]byte)
-	w.Header().Set("Content-Type", "text/html")
-	w.Write(b)
+	b := toServe.Load().(bytesOrErr)
+	if b.err != nil {
+		w.Header().Set("Content-Type", "text/plain")
+		w.Write([]byte(b.err.Error()))
+	} else {
+		w.Header().Set("Content-Type", "text/html")
+		w.Write([]byte(b.bytes))
+	}
 }
 
 func populateOnce(dir string) {
@@ -127,15 +132,15 @@ func populateOnce(dir string) {
 	cmd.Stderr = &b
 	err := cmd.Run()
 	if err != nil {
-		toServe.Store([]byte(fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String()).Error()))
+		toServe.Store(bytesOrErr{nil, fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())})
 		return
 	}
 	specBytes, err := ioutil.ReadFile(path.Join(dir, "scripts", "gen", "specification.html"))
 	if err != nil {
-		toServe.Store([]byte(fmt.Errorf("error reading spec: %v", err).Error()))
+		toServe.Store(bytesOrErr{nil, fmt.Errorf("error reading spec: %v", err)})
 		return
 	}
-	toServe.Store(specBytes)
+	toServe.Store(bytesOrErr{specBytes, nil})
 }
 
 func doPopulate(ch chan struct{}, dir string) {
@@ -160,3 +165,8 @@ func exists(path string) bool {
 	_, err := os.Stat(path)
 	return !os.IsNotExist(err)
 }
+
+type bytesOrErr struct {
+	bytes []byte
+	err   error
+}

From bbf9e229a7735c1c1ac295169a9f81b83874cc91 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 6 Nov 2015 16:09:09 +0000
Subject: [PATCH 425/467] continuserv: guard concurrent accesses to wg

---
 scripts/continuserv/main.go | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go
index 573c2c95..2c6072a3 100644
--- a/scripts/continuserv/main.go
+++ b/scripts/continuserv/main.go
@@ -25,9 +25,11 @@ import (
 var (
 	port = flag.Int("port", 8000, "Port on which to serve HTTP")
 
-	toServe atomic.Value   // Always contains valid []byte to serve. May be stale unless wg is zero.
-	wg      sync.WaitGroup // Indicates how many updates are pending.
-	mu      sync.Mutex     // Prevent multiple updates in parallel.
+	mu      sync.Mutex   // Prevent multiple updates in parallel.
+	toServe atomic.Value // Always contains valid []byte to serve. May be stale unless wg is zero.
+
+	wgMu sync.Mutex     // Prevent multiple calls to wg.Wait() or wg.Add(positive number) in parallel.
+	wg   sync.WaitGroup // Indicates how many updates are pending.
 )
 
 func main() {
@@ -111,7 +113,9 @@ func filter(e fsnotify.Event) bool {
 }
 
 func serve(w http.ResponseWriter, req *http.Request) {
+	wgMu.Lock()
 	wg.Wait()
+	wgMu.Unlock()
 	b := toServe.Load().([]byte)
 	w.Write(b)
 }
@@ -143,7 +147,9 @@ func doPopulate(ch chan struct{}, dir string) {
 		select {
 		case <-ch:
 			if pending == 0 {
+				wgMu.Lock()
 				wg.Add(1)
+				wgMu.Unlock()
 			}
 			pending++
 		case <-time.After(10 * time.Millisecond):

From e72151f2c3f8e2825eb2719b3e4598672c134a03 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Fri, 6 Nov 2015 18:15:21 +0000
Subject: [PATCH 426/467] Specify guest room access

This was reviewed as PR #150 and merged from daniel/anonymousaccess
---
 api/client-server/v1/guest_events.yaml        | 103 ++++++++++++++++++
 event-schemas/examples/v1/m.room.guest_access |  12 ++
 event-schemas/schema/v1/m.room.guest_access   |  30 +++++
 specification/modules/guest_access.rst        |  81 ++++++++++++++
 specification/targets.yaml                    |   1 +
 templating/matrix_templates/units.py          |   9 +-
 6 files changed, 234 insertions(+), 2 deletions(-)
 create mode 100644 api/client-server/v1/guest_events.yaml
 create mode 100644 event-schemas/examples/v1/m.room.guest_access
 create mode 100644 event-schemas/schema/v1/m.room.guest_access
 create mode 100644 specification/modules/guest_access.rst

diff --git a/api/client-server/v1/guest_events.yaml b/api/client-server/v1/guest_events.yaml
new file mode 100644
index 00000000..bbb5799a
--- /dev/null
+++ b/api/client-server/v1/guest_events.yaml
@@ -0,0 +1,103 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Sync Guest API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/events":
+    get:
+      summary: Listen on the event stream.
+      description: |-
+        This will listen for new events related to a particular room and return
+        them to the caller. This will block until an event is received, or until
+        the ``timeout`` is reached.
+
+        This API is the same as the non-guest /events endpoint, but can be
+        called by guest users.
+      security:
+        - accessToken: []
+      parameters:
+        - in: query
+          type: string
+          name: from
+          description: |-
+            The token to stream from. This token is either from a previous
+            request to this API or from the initial sync API.
+          required: false
+          x-example: "s3456_9_0"
+        - in: query
+          type: integer
+          name: timeout
+          description: The maximum time in milliseconds to wait for an event.
+          required: false
+          x-example: "35000"
+        - in: query
+          type: array
+          items:
+            type: string
+          name: room_id
+          description: |-
+            The room IDs for which events should be returned.
+          x-example:
+            - "!somewhere:over"
+            - "!the:rainbow"
+      responses:
+        200:
+          description: "The events received, which may be none."
+          examples:
+            application/json: |-
+              {
+                "start": "s3456_9_0",
+                "end": "s3457_9_0",
+                "chunk": [
+                  {
+                    "age": 32,
+                    "content": {
+                        "body": "incoming message",
+                        "msgtype": "m.text"
+                    },
+                    "event_id": "$14328055551tzaee:localhost",
+                    "origin_server_ts": 1432804485886,
+                    "room_id": "!TmaZBKYIFrIPVGoUYp:localhost",
+                    "type": "m.room.message",
+                    "user_id": "@bob:localhost"
+                  }
+                ]
+              }
+          schema:
+            type: object
+            properties:
+              start:
+                type: string
+                description: |-
+                  A token which correlates to the first value in ``chunk``. This
+                  is usually the same token supplied to ``from=``.
+              end:
+                type: string
+                description: |-
+                  A token which correlates to the last value in ``chunk``. This
+                  token should be used in the next request to ``/events``.
+              chunk:
+                type: array
+                description: "An array of events."
+                items:
+                  type: object
+                  title: Event
+                  allOf:
+                   - "$ref": "core-event-schema/room_event.json"
+        400:
+          description: "Bad pagination ``from`` parameter."
diff --git a/event-schemas/examples/v1/m.room.guest_access b/event-schemas/examples/v1/m.room.guest_access
new file mode 100644
index 00000000..8ad4d294
--- /dev/null
+++ b/event-schemas/examples/v1/m.room.guest_access
@@ -0,0 +1,12 @@
+{
+  "age": 242353,
+  "content": {
+    "guest_access": "can_join"
+  },
+  "state_key": "",
+  "origin_server_ts": 1431961217938,
+  "event_id": "$WLGTSEFSEG:localhost",
+  "type": "m.room.guest_access",
+  "room_id": "!Cuyf34gef24u:localhost",
+  "user_id": "@example:localhost"
+}
diff --git a/event-schemas/schema/v1/m.room.guest_access b/event-schemas/schema/v1/m.room.guest_access
new file mode 100644
index 00000000..a0141f37
--- /dev/null
+++ b/event-schemas/schema/v1/m.room.guest_access
@@ -0,0 +1,30 @@
+{
+    "type": "object",
+    "title": "Controls whether guest users are allowed to join rooms.",
+    "description": "This event controls whether guest users are allowed to join rooms. If this event is absent, servers should act as if it is present and has the guest_access value \"forbidden\".",
+    "allOf": [{
+        "$ref": "core-event-schema/state_event.json"
+    }],
+    "properties": {
+        "content": {
+            "type": "object",
+            "properties": {
+                "guest_access": {
+                    "type": "string",
+                    "description": "Whether guests can join the room.",
+                    "enum": ["can_join", "forbidden"]
+                }
+            },
+            "required": ["guest_access"]
+        },
+        "state_key": {
+            "type": "string",
+            "description": "A zero-length string.",
+            "pattern": "^$"
+        },
+        "type": {
+            "type": "string",
+            "enum": ["m.room.guest_access"]
+        }
+    }
+}
diff --git a/specification/modules/guest_access.rst b/specification/modules/guest_access.rst
new file mode 100644
index 00000000..ac373af9
--- /dev/null
+++ b/specification/modules/guest_access.rst
@@ -0,0 +1,81 @@
+Guest access
+============
+
+.. _module:guest-access:
+
+There are times when it is desirable for clients to be able to interact with
+rooms without having to fully register for an account on a homeserver or join
+the room. This module specifies how these clients should interact with servers
+in order to participate in rooms as guests.
+
+Guest users retrieve access tokens from a homeserver using the ordinary
+`register endpoint <#post-matrix-client-api-v2-alpha-register>`_, specifying
+the ``kind`` parameter as ``guest``. They may then interact with the
+client-server API as any other user would, but will only have access to a subset
+of the API as described the Client behaviour subsection below.
+Homeservers may choose not to allow this access at all to their local users, but
+have no information about whether users on other homeservers are guests or not.
+
+This module does not fully factor in federation; it relies on individual
+homeservers properly adhering to the rules set out in this module, rather than
+allowing all homeservers to enforce the rules on each other.
+
+Events
+------
+{{m_room_guest_access_event}}
+
+Client behaviour
+----------------
+The following API endpoints are allowed to be accessed by guest accounts for
+retrieving events:
+
+* `GET /rooms/:room_id/state <#get-matrix-client-api-v1-rooms-roomid-state>`_
+* `GET /rooms/:room_id/state/:event_type/:state_key <#get-matrix-client-api-v1-rooms-roomid-state-eventtype-statekey>`_
+* `GET /rooms/:room_id/messages <#get-matrix-client-api-v1-rooms-roomid-messages>`_
+
+There is also a special version of the
+`GET /events <#get-matrix-client-api-v1-events>`_ endpoint:
+
+{{guest_events_http_api}}
+
+They will only return events which happened while the room state had the
+``m.room.history_visibility`` state event present with ``history_visibility``
+value ``world_readable``. Guest clients do not need to join rooms in order to
+receive events for them.
+
+The following API endpoints are allowed to be accessed by guest accounts for
+sending events:
+
+* `POST /rooms/:room_id/join <#post-matrix-client-api-v1-rooms-roomid-join>`_
+* `PUT /rooms/:room_id/send/m.room.message/:txn_id <#put-matrix-client-api-v1-rooms-roomid-send-eventtype-txnid>`_
+
+Guest clients *do* need to join rooms in order to send events to them.
+
+Server behaviour
+----------------
+Servers are required to only return events to guest accounts for rooms where
+the room state at the event had the  ``m.room.history_visibility`` state event
+present with ``history_visibility`` value ``world_readable``. These events may
+be returned even if the anonymous user is not joined to the room.
+
+Servers MUST only allow guest users to join rooms if the ``m.room.guest_access``
+state event is present on the room, and has the ``guest_access`` value
+``can_join``. If the ``m.room.guest_access`` event is changed to stop this from
+being the case, the server MUST set those users' ``m.room.member`` state to
+``leave``.
+
+Security considerations
+-----------------------
+Each homeserver manages its own guest accounts itself, and whether an account
+is a guest account or not is not information passed from server to server.
+Accordingly, any server participating in a room is trusted to properly enforce
+the permissions outlined in this section.
+
+Clients may wish to display to their users that rooms which are
+``world_readable`` *may* be showing messages to non-joined users. There is no
+way using this module to find out whether any non-joined guest users *do* see
+events in the room, or to list or count any guest users.
+
+Homeservers may want to enable protections such as captchas for guest
+registration to prevent spam, denial of service, and similar attacks.
+
diff --git a/specification/targets.yaml b/specification/targets.yaml
index 2482dcfd..8e6a2ce0 100644
--- a/specification/targets.yaml
+++ b/specification/targets.yaml
@@ -25,6 +25,7 @@ groups:  # reusable blobs of files when prefixed with 'group:'
     - modules/push.rst
     - modules/third_party_invites.rst
     - modules/search.rst
+    - modules/guest_access.rst
 
 
 title_styles: ["=", "-", "~", "+", "^", "`"]
diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index cec5b83e..adb7f427 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -369,7 +369,7 @@ class MatrixUnits(Units):
                 ]
                 if len(params_missing_examples) == 0:
                     path_template = api.get("basePath", "").rstrip("/") + path
-                    qps = {}
+                    qps = []
                     body = ""
                     for param in single_api.get("parameters", []):
                         if param["in"] == "path":
@@ -381,7 +381,12 @@ class MatrixUnits(Units):
                         elif param["in"] == "body":
                             body = param["schema"]["example"]
                         elif param["in"] == "query":
-                            qps[param["name"]] = param["x-example"]
+                            example = param["x-example"]
+                            if type(example) == list:
+                                for value in example:
+                                    qps.append((param["name"], value))
+                            else:
+                                qps.append((param["name"], example))
                     query_string = "" if len(qps) == 0 else "?"+urllib.urlencode(qps)
                     if body:
                         endpoint["example"]["req"] = "%s %s%s HTTP/1.1\nContent-Type: application/json\n\n%s" % (

From 436a35e9f647fcaa5ef27b48ef26b683440fcb40 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 9 Nov 2015 16:04:31 +0000
Subject: [PATCH 427/467] Document macaroon type=login

---
 drafts/macaroons_caveats.rst | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 93622c3d..a7c1b036 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -25,10 +25,13 @@ Specified caveats:
 | gen         | Generation of the macaroon caveat spec.          | 1                                                                                              |
 | user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                       |
 | type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                     |
-|                                                                   refresh - only used to authorize a token refresh                                              |
+|             |                                                  | refresh - only used to authorize a token refresh                                               |
+|             |                                                  | login - issued as a very short-lived token by third party login flows; proves that             |
+|             |                                                  | authentication has happened but doesn't grant any privileges other than being able to be       |
+|             |                                                  | exchanged for other tokens.                                                                    |
 | time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                    |
-|                                                                  Operator < means the macaroon is valid before the timestamp, as interpreted by the server.     |
-|                                                                  Operator > means the macaroon is valid after the timestamp, as interpreted by the server.      |
-|                                                                  Operator == means the macaroon is valid at exactly the timestamp, as interpreted by the server.|
-|                                                                  Note that exact equality of time is largely meaningless.                                       |
+|             |                                                  | Operator < means the macaroon is valid before the timestamp, as interpreted by the server.     |
+|             |                                                  | Operator > means the macaroon is valid after the timestamp, as interpreted by the server.      |
+|             |                                                  | Operator == means the macaroon is valid at exactly the timestamp, as interpreted by the server.|
+|             |                                                  | Note that exact equality of time is largely meaningless.                                       |
 +-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+

From 24c2036a35738ebf9c1ebe3b01cef1943a087702 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Mon, 9 Nov 2015 17:30:18 +0000
Subject: [PATCH 428/467] 3pid invites: remove mentions of display_name

---
 api/client-server/v1/third_party_membership.yaml |  8 ++------
 specification/modules/third_party_invites.rst    | 11 ++++-------
 2 files changed, 6 insertions(+), 13 deletions(-)

diff --git a/api/client-server/v1/third_party_membership.yaml b/api/client-server/v1/third_party_membership.yaml
index 90b167b4..a10d6167 100644
--- a/api/client-server/v1/third_party_membership.yaml
+++ b/api/client-server/v1/third_party_membership.yaml
@@ -84,8 +84,7 @@ paths:
               {
                 "id_server": "matrix.org",
                 "medium": "email",
-                "address": "cheeky@monkey.com",
-                "display_name": "A very cheeky monkey"
+                "address": "cheeky@monkey.com"
               }
             properties:
               id_server:
@@ -98,10 +97,7 @@ paths:
               address:
                 type: string
                 description: The invitee's third party identifier.
-              display_name:
-                type: string
-                description: A user-friendly string describing who has been invited. It should not contain the address of the invitee, to avoid leaking mappings between third party identities and matrix user IDs.
-            required: ["id_server", "medium", "address", "display_name"]
+            required: ["id_server", "medium", "address"]
       responses:
         200:
           description: The user has been invited to join the room.
diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst
index 4d268631..d8e3d4d9 100644
--- a/specification/modules/third_party_invites.rst
+++ b/specification/modules/third_party_invites.rst
@@ -127,13 +127,10 @@ It is important for user privacy that leaking the mapping between a matrix user
 ID and a third party identifier is hard. In particular, being able to look up
 all third party identifiers from a matrix user ID (and accordingly, being able
 to link each third party identifier) should be avoided wherever possible.
-To this end, when implementing this API care should be taken to avoid
-adding links between these two identifiers as room events. This mapping can be
-unintentionally created by specifying the third party identifier in the
-``display_name`` field of the ``m.room.third_party_invite`` event, and then
-observing which matrix user ID joins the room using that invite. Clients SHOULD
-set ``display_name`` to a value other than the third party identifier, e.g. the
-invitee's common name.
+To this end, the third party identifier is not put in any event, rather an
+opaque display name provided by the identity server is put into the events.
+Clients should not remember or display third party identifiers from invites,
+other than for the use of the inviter themself.
 
 Homeservers are not required to trust any particular identity server(s). It is
 generally a client's responsibility to decide which identity servers it trusts,

From c1866ebebc708d0e8a404edbc3f0bb9b0f3d4297 Mon Sep 17 00:00:00 2001
From: Kegsay <kegsay@gmail.com>
Date: Tue, 10 Nov 2015 11:26:06 +0000
Subject: [PATCH 429/467] Fix table formatting

---
 drafts/macaroons_caveats.rst | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index a7c1b036..71c4784e 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -21,14 +21,17 @@ Specified caveats:
 
 +-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
 | Caveat name | Description                                      | Legal Values                                                                                   |
-+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
++=============+==================================================+================================================================================================+
 | gen         | Generation of the macaroon caveat spec.          | 1                                                                                              |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
 | user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                       |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
 | type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                     |
 |             |                                                  | refresh - only used to authorize a token refresh                                               |
 |             |                                                  | login - issued as a very short-lived token by third party login flows; proves that             |
 |             |                                                  | authentication has happened but doesn't grant any privileges other than being able to be       |
 |             |                                                  | exchanged for other tokens.                                                                    |
++-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
 | time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                    |
 |             |                                                  | Operator < means the macaroon is valid before the timestamp, as interpreted by the server.     |
 |             |                                                  | Operator > means the macaroon is valid after the timestamp, as interpreted by the server.      |

From 51fe4a90b6f3e79428f1e1160b7c7df4f906adb9 Mon Sep 17 00:00:00 2001
From: Kegsay <kegsay@gmail.com>
Date: Tue, 10 Nov 2015 11:28:27 +0000
Subject: [PATCH 430/467] More formatting fixes

---
 drafts/macaroons_caveats.rst | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 71c4784e..11e36e59 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -9,12 +9,15 @@ Caveats can only be used for reducing the scope of a token, never for increasing
 
 Some caveats are specified in this specification, and must be understood by all servers. The use of non-standard caveats is allowed.
 
-All caveats must take the form:
+All caveats must take the form::
 
-`key` `operator` `value`
-where `key` is a non-empty string drawn from the character set [A-Za-z0-9_]
-`operator` is a non-empty string which does not contain whitespace
-`value` is a non-empty string
+  key operator value
+
+where:
+ - ``key`` is a non-empty string drawn from the character set [A-Za-z0-9_]
+ - ``operator`` is a non-empty string which does not contain whitespace
+ - ``value`` is a non-empty string
+ 
 And these are joined by single space characters.
 
 Specified caveats:

From c8f6ed11074f489df5562ce614e3113f48ea77d9 Mon Sep 17 00:00:00 2001
From: Kegsay <kegsay@gmail.com>
Date: Tue, 10 Nov 2015 11:31:31 +0000
Subject: [PATCH 431/467] More formatting..

---
 drafts/macaroons_caveats.rst | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drafts/macaroons_caveats.rst b/drafts/macaroons_caveats.rst
index 11e36e59..de5973fa 100644
--- a/drafts/macaroons_caveats.rst
+++ b/drafts/macaroons_caveats.rst
@@ -29,11 +29,11 @@ Specified caveats:
 +-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
 | user_id     | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =.                                                       |
 +-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
-| type        | The purpose of this macaroon.                    | access - used to authorize any action except token refresh                                     |
-|             |                                                  | refresh - only used to authorize a token refresh                                               |
-|             |                                                  | login - issued as a very short-lived token by third party login flows; proves that             |
-|             |                                                  | authentication has happened but doesn't grant any privileges other than being able to be       |
-|             |                                                  | exchanged for other tokens.                                                                    |
+| type        | The purpose of this macaroon.                    | - ``access``: used to authorize any action except token refresh                                |
+|             |                                                  | -  ``refresh``: only used to authorize a token refresh                                         |
+|             |                                                  | - ``login``: issued as a very short-lived token by third party login flows; proves that        |
+|             |                                                  |   authentication has happened but doesn't grant any privileges other than being able to be     |
+|             |                                                  |   exchanged for other tokens.                                                                  |
 +-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
 | time        | Time before/after which this macaroon is valid.  | A POSIX timestamp in milliseconds (in UTC).                                                    |
 |             |                                                  | Operator < means the macaroon is valid before the timestamp, as interpreted by the server.     |

From 27ffe7bacd15ae559cdfc7bce03a42d00699d7d4 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 10 Nov 2015 15:34:32 +0000
Subject: [PATCH 432/467] Don't serve rst diffs as HTML

---
 scripts/speculator/main.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index 7f86bd62..e1898d7e 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -385,7 +385,7 @@ func main() {
 	}
 	s := server{masterCloneDir}
 	http.HandleFunc("/spec/", forceHTML(s.serveSpec))
-	http.HandleFunc("/diff/rst/", forceHTML(s.serveRSTDiff))
+	http.HandleFunc("/diff/rst/", s.serveRSTDiff)
 	http.HandleFunc("/diff/html/", forceHTML(s.serveHTMLDiff))
 	http.HandleFunc("/healthz", serveText("ok"))
 	http.HandleFunc("/", forceHTML(listPulls))

From d7357ef9b75893631e4a8b1e5e40273885cd6709 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 11 Nov 2015 11:39:40 +0000
Subject: [PATCH 433/467] Specify /publicRooms

---
 api/client-server/v1/list_public_rooms.yaml   | 85 +++++++++++++++++++
 specification/client_server_api.rst           |  5 ++
 .../matrix_templates/templates/http-api.tmpl  |  5 +-
 3 files changed, 94 insertions(+), 1 deletion(-)
 create mode 100644 api/client-server/v1/list_public_rooms.yaml

diff --git a/api/client-server/v1/list_public_rooms.yaml b/api/client-server/v1/list_public_rooms.yaml
new file mode 100644
index 00000000..2f189d2e
--- /dev/null
+++ b/api/client-server/v1/list_public_rooms.yaml
@@ -0,0 +1,85 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Room Creation API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+paths:
+  "/publicRooms":
+    get:
+      summary: Lists the public rooms on the server.
+      description: |-
+        Lists the public rooms on the server.
+
+        This API returns paginated responses.
+      responses:
+        200:
+          description: A list of the rooms on the server.
+          schema:
+            type: object
+            description: A list of the rooms on the server.
+            properties:
+              chunk:
+                title: "PublicRoomsChunks"
+                type: array
+                description: |-
+                  A paginated chunk of public rooms.
+                items:
+                  type: object
+                  title: "PublicRoomsChunk"
+                  properties:
+                    aliases:
+                      type: array
+                      description: |-
+                        Aliases of the room. May be empty.
+                      items:
+                        type: string
+                    name:
+                      type: string
+                      description: |-
+                        The name of the room, if any. May be null.
+                    num_joined_members:
+                      type: number
+                      description: |-
+                        The number of members joined to the room.
+                    room_id:
+                      type: string
+                      description: |-
+                        The ID of the room.
+                    topic:
+                      type: string
+                      description: |-
+                        The topic of the room, if any. May be null.
+              start:
+                type: string
+                description: |-
+                  A pagination token for the response.
+              end:
+                type: string
+                description: |-
+                  A pagination token for the response.
+          examples:
+            application/json: |-
+              {
+                "chunk": [
+                  {
+                    "aliases": ["#murrays:cheese.bar"],
+                    "name": "CHEESE",
+                    "num_joined_members": 37,
+                    "room_id": "!ol19s:bleecker.street",
+                    "topic": "Tasty tasty cheese"
+                  }
+                ],
+                "start": "p190q",
+                "end": "p1902"
+              }
+        400:
+          description: >
+            The request body is malformed or the room alias specified is already taken.
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index b02dbf28..05b6ff3c 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -931,6 +931,11 @@ member's state, by making a request to
     "membership": "ban"
   }
 
+Listing rooms
+~~~~~~~~~~~~~
+
+{{list_public_rooms_http_api}}
+
 Profiles
 --------
 
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index 86eacb14..d7258e98 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -12,7 +12,7 @@
 {{":Requires auth: Yes." if endpoint.requires_auth else "" }}
 
 Request format:
-
+{% if (endpoint.req_param_by_loc | length) %}
 =========================================== ================= ===========================================
    Parameter                                 Value                        Description
 =========================================== ================= ===========================================
@@ -24,6 +24,9 @@ Request format:
 {% endfor -%}
 {% endfor -%}
 =========================================== ================= ===========================================
+{% else %}
+`No parameters`
+{% endif %}
 
 {% if endpoint.res_tables|length > 0 -%}
 Response format:

From dcf54e11b10b07fd30cd18be127d9cde9b198e8a Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 11 Nov 2015 11:53:31 +0000
Subject: [PATCH 434/467] Specify /publicRooms world_readable and guest_access

Depends on https://github.com/matrix-org/matrix-doc/pull/154
---
 api/client-server/v1/list_public_rooms.yaml | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/list_public_rooms.yaml b/api/client-server/v1/list_public_rooms.yaml
index 2f189d2e..23b1c3d0 100644
--- a/api/client-server/v1/list_public_rooms.yaml
+++ b/api/client-server/v1/list_public_rooms.yaml
@@ -57,6 +57,16 @@ paths:
                       type: string
                       description: |-
                         The topic of the room, if any. May be null.
+                    world_readable:
+                      type: boolean
+                      description: |-
+                        Whether the room may be viewed by guest users without joining.
+                    guest_can_join:
+                      type: boolean
+                      description: |-
+                        Whether guest users may join the room and participate in it.
+                        If they can, they will be subject to ordinary power level
+                        rules like any other user.
               start:
                 type: string
                 description: |-
@@ -71,10 +81,12 @@ paths:
                 "chunk": [
                   {
                     "aliases": ["#murrays:cheese.bar"],
+                    "guest_can_join": false,
                     "name": "CHEESE",
                     "num_joined_members": 37,
                     "room_id": "!ol19s:bleecker.street",
-                    "topic": "Tasty tasty cheese"
+                    "topic": "Tasty tasty cheese",
+                    "world_readable": true
                   }
                 ],
                 "start": "p190q",

From 740cc66a7cccc366ca34eac1ebf65fd7c9af90b3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 11 Nov 2015 14:01:45 +0000
Subject: [PATCH 435/467] speculator: Fetch before deciding head is fresh

---
 scripts/speculator/main.go | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index e1898d7e..e467018e 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -178,6 +178,10 @@ func (s *server) serveSpec(w http.ResponseWriter, req *http.Request) {
 	var sha string
 
 	if strings.ToLower(req.URL.Path) == "/spec/head" {
+		if err := gitFetch(s.matrixDocCloneURL); err != nil {
+			writeError(w, 500, err)
+			return
+		}
 		originHead, err := s.getSHAOf("origin/master")
 		if err != nil {
 			writeError(w, 500, err)

From 0f0359d9c1049c60dbea365344b40cadc1a50c16 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 11 Nov 2015 15:13:34 +0000
Subject: [PATCH 436/467] speculator: Nessle up some more if statements

---
 scripts/speculator/main.go | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go
index e467018e..97e67c8c 100644
--- a/scripts/speculator/main.go
+++ b/scripts/speculator/main.go
@@ -74,8 +74,7 @@ func gitClone(url string, shared bool) (string, error) {
 		cmd.Args = append(cmd.Args, "--shared")
 	}
 
-	err := cmd.Run()
-	if err != nil {
+	if err := cmd.Run(); err != nil {
 		return "", fmt.Errorf("error cloning repo: %v", err)
 	}
 	return directory, nil
@@ -92,8 +91,7 @@ func gitFetch(path string) error {
 func runGitCommand(path string, args []string) error {
 	cmd := exec.Command("git", args...)
 	cmd.Dir = path
-	err := cmd.Run()
-	if err != nil {
+	if err := cmd.Run(); err != nil {
 		return fmt.Errorf("error running %q: %v", strings.Join(cmd.Args, " "), err)
 	}
 	return nil
@@ -126,8 +124,7 @@ func generate(dir string) error {
 	cmd.Dir = path.Join(dir, "scripts")
 	var b bytes.Buffer
 	cmd.Stderr = &b
-	err := cmd.Run()
-	if err != nil {
+	if err := cmd.Run(); err != nil {
 		return fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())
 	}
 	return nil
@@ -167,8 +164,7 @@ func (s *server) getSHAOf(ref string) (string, error) {
 	cmd.Dir = path.Join(s.matrixDocCloneURL)
 	var b bytes.Buffer
 	cmd.Stdout = &b
-	err := cmd.Run()
-	if err != nil {
+	if err := cmd.Run(); err != nil {
 		return "", fmt.Errorf("error generating spec: %v\nOutput from gendoc:\n%v", err, b.String())
 	}
 	return strings.TrimSpace(b.String()), nil

From 061105c9dc25d52a01ba260afc9c90bd0158931b Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 11 Nov 2015 17:18:58 +0000
Subject: [PATCH 437/467] Guest users are allowed room initialSync

---
 api/client-server/v1/rooms.yaml        | 2 +-
 specification/modules/guest_access.rst | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/api/client-server/v1/rooms.yaml b/api/client-server/v1/rooms.yaml
index 9300d7d1..90b51290 100644
--- a/api/client-server/v1/rooms.yaml
+++ b/api/client-server/v1/rooms.yaml
@@ -371,7 +371,7 @@ paths:
                 description: |-
                   Whether this room is visible to the ``/publicRooms`` API
                   or not."
-            required: ["room_id", "membership"]
+            required: ["room_id"]
         403:
           description: >
             You aren't a member of the room and weren't previously a
diff --git a/specification/modules/guest_access.rst b/specification/modules/guest_access.rst
index ac373af9..ae8d156e 100644
--- a/specification/modules/guest_access.rst
+++ b/specification/modules/guest_access.rst
@@ -32,6 +32,7 @@ retrieving events:
 * `GET /rooms/:room_id/state <#get-matrix-client-api-v1-rooms-roomid-state>`_
 * `GET /rooms/:room_id/state/:event_type/:state_key <#get-matrix-client-api-v1-rooms-roomid-state-eventtype-statekey>`_
 * `GET /rooms/:room_id/messages <#get-matrix-client-api-v1-rooms-roomid-messages>`_
+* `GET /rooms/:room_id/initialSync <#get-matrix-client-api-v1-rooms-roomid-initialsync>`_
 
 There is also a special version of the
 `GET /events <#get-matrix-client-api-v1-events>`_ endpoint:

From 4d3175fc8b5c01a41afb802c2e4bc0e191b822db Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 12 Nov 2015 10:45:57 +0000
Subject: [PATCH 438/467] Note that invite_room_state is optional

---
 event-schemas/schema/v1/m.room.member | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/event-schemas/schema/v1/m.room.member b/event-schemas/schema/v1/m.room.member
index 81057049..25b30109 100644
--- a/event-schemas/schema/v1/m.room.member
+++ b/event-schemas/schema/v1/m.room.member
@@ -1,7 +1,7 @@
 {
     "type": "object",
     "title": "The current membership state of a user in the room.",
-    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event also includes an ``invite_room_state`` key **outside the** ``content`` **key**. This contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
+    "description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event may also include an ``invite_room_state`` key **outside the** ``content`` **key**. If present, this contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
     "allOf": [{
         "$ref": "core-event-schema/state_event.json"
     }],

From 29d9c8eec6a0a7da49b78c4885b168662fc04f99 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Thu, 12 Nov 2015 12:05:02 +0000
Subject: [PATCH 439/467] Guests are allowed to set displaynames

---
 specification/modules/guest_access.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/specification/modules/guest_access.rst b/specification/modules/guest_access.rst
index ac373af9..d7359f23 100644
--- a/specification/modules/guest_access.rst
+++ b/specification/modules/guest_access.rst
@@ -51,6 +51,11 @@ sending events:
 
 Guest clients *do* need to join rooms in order to send events to them.
 
+The following API endpoints are allowed to be accessed by guest accounts for
+their own account maintenance:
+
+* `PUT /profile/:user_id/displayname <#put-matrix-client-api-v1-profile-userid-displayname>`_
+
 Server behaviour
 ----------------
 Servers are required to only return events to guest accounts for rooms where

From d7c69fae438410e1b91030e60466f48f28e27360 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 13 Nov 2015 16:33:34 +0000
Subject: [PATCH 440/467] Fix typo 'process'

---
 specification/server_server_api.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index d26cc26e..1c3e761d 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -555,8 +555,8 @@ sign, and emit a new ``m.room.member`` state event adding the user into that
 room. When the homeserver does not yet know about the room it cannot do this
 directly. Instead, it must take a longer multi-stage handshaking process by
 which it first selects a remote homeserver which is already participating in
-that room, and uses it to assist in the joining procss. This is the remote join
-handshake.
+that room, and uses it to assist in the joining process. This is the remote
+join handshake.
 
 This handshake involves the homeserver of the new member wishing to join
 (referred to here as the "joining" server), the directory server hosting the

From aac45295ee9dc615cfc7290cc928c011ca0d4e79 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 13 Nov 2015 16:34:22 +0000
Subject: [PATCH 441/467] Remark that the directory server step could be
 skipped in an invite case

---
 specification/server_server_api.rst | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 1c3e761d..e158ae23 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -596,9 +596,12 @@ homeservers, though most in practice will use just two.
                  |
   <---------- join response
 
-The first part of the handshake involves using the directory server to request
-the room ID and join candidates. This is covered in more detail on the
-directory server documentation, below.
+The first part of the handshake usually involves using the directory server to
+request the room ID and join candidates. This is covered in more detail on the
+directory server documentation, below. In the case of a new user joining a
+room as a result of a received invite, the joining user's homeserver could
+optimise this step away by picking the origin server of that invite message as
+the join candidate.
 
 Once the joining server has the room ID and the join candidates, it then needs
 to obtain enough of the current state of the room to fill in the required

From db5a90edcd7fcc33385d2145b2ad1ed93313182f Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 13 Nov 2015 16:36:26 +0000
Subject: [PATCH 442/467] Avoid using the words 'current state' when talking
 about the result of the /make_join request

---
 specification/server_server_api.rst | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index e158ae23..6dfbd86f 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -566,9 +566,8 @@ room members are already present (referred to as the "resident" server).
 In summary, the remote join handshake consists of the joining server querying
 the directory server for information about the room alias; receiving a room ID
 and a list of join candidates. The joining server then requests information
-about the current state of the room from one of the residents. It uses this
-information to construct a ``m.room.member`` event which it finally sends to
-a resident server.
+about the room from one of the residents. It uses this information to construct
+a ``m.room.member`` event which it finally sends to a resident server.
 
 Conceptually these are three different roles of homeserver. In practice the
 directory server is likely to be resident in the room, and so may be selected
@@ -604,11 +603,11 @@ optimise this step away by picking the origin server of that invite message as
 the join candidate.
 
 Once the joining server has the room ID and the join candidates, it then needs
-to obtain enough of the current state of the room to fill in the required
-fields of the ``m.room.member`` event. It obtains this by selecting a resident
-from the candidate list, and requesting the ``make_join`` endpoint using a
-``GET`` request, specifying the room ID and the user ID of the new member who
-is attempting to join.
+to obtain enough information about the room to fill in the required fields of
+the ``m.room.member`` event. It obtains this by selecting a resident from the
+candidate list, and requesting the ``make_join`` endpoint using a ``GET``
+request, specifying the room ID and the user ID of the new member who is
+attempting to join.
 
 The resident server replies to this request with a JSON-encoded object having a
 single key called ``event``; within this is an object whose fields contain some

From 885dd1e86cc8d6ee818ab32ab246eeec9a1f5ede Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 13 Nov 2015 16:39:31 +0000
Subject: [PATCH 443/467] Explain the 'prev_events' join protoevent key

---
 specification/server_server_api.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 6dfbd86f..ebf55eb4 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -629,7 +629,8 @@ the assisting resident. The required fields are:
                               resident
 ``origin``           String   The name of the assisting resident homeserver
 ``origin_server_ts`` Integer  A timestamp added by the resident homeserver
-``prev_events``      List     (TODO(paul): ? - I notice these can be blank)
+``prev_events``      List     An event-reference list containing the immediate
+                              predecessor events
 ``prev_state``       List     (TODO(paul): ? - I notice these can be blank)
 ``room_id``          String   The room ID of the room
 ``sender``           String   The user ID of the joining member

From 988d77347651ce2504ab1e52196166358dffe7f7 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Fri, 13 Nov 2015 17:49:22 +0000
Subject: [PATCH 444/467] /make_join protoevent no longer needs the pointless
 'prev_state' key (SYN-517)

---
 specification/server_server_api.rst | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index ebf55eb4..9cc10141 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -631,7 +631,6 @@ the assisting resident. The required fields are:
 ``origin_server_ts`` Integer  A timestamp added by the resident homeserver
 ``prev_events``      List     An event-reference list containing the immediate
                               predecessor events
-``prev_state``       List     (TODO(paul): ? - I notice these can be blank)
 ``room_id``          String   The room ID of the room
 ``sender``           String   The user ID of the joining member
 ``state_key``        String   The user ID of the joining member
@@ -688,8 +687,6 @@ integer 200, and whose second element contains the following keys:
   - (paul) I don't really understand why the full auth_chain events are given
     here. What purpose does it serve expanding them out in full, when surely
     they'll appear in the state anyway?
-  - (paul) the state seems to be entirely ignored by synapse, so I'm not really
-    sure what ought to be there.
 
 Backfilling
 -----------

From cc8ef691fb20331d135a39993e843d42588e0761 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Sun, 15 Nov 2015 16:49:22 +0000
Subject: [PATCH 445/467] Rewrite the table templates

Allow columns to stretch if they end up with wide content.

Apart from the hassle of having to manually update the calculations, having the
columns wide enough to hold the widest thing they might ever have leads to
tables with lots of whitespace in the results.
---
 templating/build.py                           |  22 ++++
 .../templates/common-event-fields.tmpl        |  15 +--
 .../matrix_templates/templates/events.tmpl    |  15 +--
 .../matrix_templates/templates/http-api.tmpl  |  27 +----
 .../matrix_templates/templates/msgtypes.tmpl  |  15 +--
 .../matrix_templates/templates/tables.tmpl    | 104 ++++++++++++++++++
 6 files changed, 139 insertions(+), 59 deletions(-)
 create mode 100644 templating/matrix_templates/templates/tables.tmpl

diff --git a/templating/build.py b/templating/build.py
index a35d8a08..77fecf91 100755
--- a/templating/build.py
+++ b/templating/build.py
@@ -93,6 +93,27 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
 
         return '\n\n'.join(output_lines)
 
+    def fieldwidths(input, keys, defaults=[], default_width=15):
+        """
+        A template filter to help in the generation of tables.
+
+        Given a list of rows, returns a list giving the maximum length of the
+        values in each column.
+
+        :param list[dict[str, str]] input: a list of rows. Each row should be a
+           dict with the keys given in ``keys``.
+        :param list[str] keys: the keys corresponding to the table columns
+        :param list[int] defaults: for each column, the default column width.
+        :param int default_width: if ``defaults`` is shorter than ``keys``, this
+           will be used as a fallback
+        """
+        def colwidth(key, default):
+            return reduce(max, (len(row[key]) for row in input),
+                          default if default is not None else default_width)
+
+        results = map(colwidth, keys, defaults)
+        return results
+
     # make Jinja aware of the templates and filters
     env = Environment(
         loader=FileSystemLoader(in_mod.exports["templates"]),
@@ -102,6 +123,7 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
     env.filters["indent"] = indent
     env.filters["indent_block"] = indent_block
     env.filters["wrap"] = wrap
+    env.filters["fieldwidths"] = fieldwidths
 
     # load up and parse the lowest single units possible: we don't know or care
     # which spec section will use it, we just need it there in memory for when
diff --git a/templating/matrix_templates/templates/common-event-fields.tmpl b/templating/matrix_templates/templates/common-event-fields.tmpl
index 3f16be3d..8d8c8f0c 100644
--- a/templating/matrix_templates/templates/common-event-fields.tmpl
+++ b/templating/matrix_templates/templates/common-event-fields.tmpl
@@ -1,17 +1,8 @@
+{% import 'tables.tmpl' as tables -%}
+
 {{common_event.title}} Fields
 {{(7 + common_event.title | length) * title_kind}}
 
 {{common_event.desc | wrap(80)}}
 
-================== ================= ===========================================
-   Key                  Type                 Description
-================== ================= ===========================================
-{% for row in common_event.rows -%}
-{# -#}
-{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
-{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
-{#      It also needs to then wrap inside the desc col (43 ch width) -#}
-{# -#}
-{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc | indent(18 - (row.type|length)) |wrap(43) |indent_block(37)}}
-{% endfor -%}
-================== ================= ===========================================
+{{ tables.paramtable(common_event.rows, ["Key", "Type", "Description"]) }}
diff --git a/templating/matrix_templates/templates/events.tmpl b/templating/matrix_templates/templates/events.tmpl
index fb876440..324a8f3e 100644
--- a/templating/matrix_templates/templates/events.tmpl
+++ b/templating/matrix_templates/templates/events.tmpl
@@ -1,3 +1,5 @@
+{% import 'tables.tmpl' as tables -%}
+
 ``{{event.type}}``
 {{(4 + event.type | length) * title_kind}}
 *{{event.typeof}}*
@@ -7,18 +9,7 @@
 {% for table in event.content_fields -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-======================= ================= ===========================================
-   {{table.title or "Content"}} Key                Type                 Description
-======================= ================= ===========================================
-{% for row in table.rows -%}
-{# -#}
-{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
-{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
-{#      It also needs to then wrap inside the desc col (43 ch width) -#}
-{# -#}
-{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(42)}}
-{% endfor -%}
-======================= ================= ===========================================
+{{ tables.paramtable(table.rows, [(table.title or "Content") ~ " Key", "Type", "Description"]) }}
 
 {% endfor %}
 Example:
diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index d7258e98..1253e66e 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -1,3 +1,5 @@
+{% import 'tables.tmpl' as tables -%}
+
 ``{{endpoint.method}} {{endpoint.path}}``
 {{(5 + (endpoint.path | length) + (endpoint.method | length)) * title_kind}}
 {% if "alias_for_path" in endpoint -%}
@@ -13,17 +15,7 @@
 
 Request format:
 {% if (endpoint.req_param_by_loc | length) %}
-=========================================== ================= ===========================================
-   Parameter                                 Value                        Description
-=========================================== ================= ===========================================
-{% for loc in endpoint.req_param_by_loc -%}
-*{{loc}} parameters*
---------------------------------------------------------------------------------------------------------- 
-{% for param in endpoint.req_param_by_loc[loc] -%}
-{{param.key}}{{param.type|indent(44-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(62)}}
-{% endfor -%}
-{% endfor -%}
-=========================================== ================= ===========================================
+{{ tables.split_paramtable(endpoint.req_param_by_loc) }}
 {% else %}
 `No parameters`
 {% endif %}
@@ -34,18 +26,7 @@ Response format:
 {% for table in endpoint.res_tables -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-======================= ========================= ==========================================
-      Param                        Type                           Description
-======================= ========================= ==========================================
-{% for row in table.rows -%}
-{# -#}
-{# Row type needs to prepend spaces to line up with the type column (20 ch) -#}
-{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
-{#      It also needs to then wrap inside the desc col (42 ch width) -#}
-{# -#}
-{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(26 - (row.type|length))) |indent_block(50)}}
-{% endfor -%}
-======================= ========================= ==========================================
+{{ tables.paramtable(table.rows) }}
 
 {% endfor %}
 {% endif -%}
diff --git a/templating/matrix_templates/templates/msgtypes.tmpl b/templating/matrix_templates/templates/msgtypes.tmpl
index 18d3492b..87cf4a19 100644
--- a/templating/matrix_templates/templates/msgtypes.tmpl
+++ b/templating/matrix_templates/templates/msgtypes.tmpl
@@ -1,21 +1,12 @@
+{% import 'tables.tmpl' as tables -%}
+
 ``{{event.msgtype}}``
 {{(4 + event.msgtype | length) * title_kind}}
 {{event.desc | wrap(80)}}
 {% for table in event.content_fields -%}
 {{"``"+table.title+"``" if table.title else "" }}
 
-================== ================= ===========================================
-   {{table.title or "Content"}} Key          Type                 Description
-================== ================= ===========================================
-{% for row in table.rows -%}
-{# -#}
-{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
-{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
-{#      It also needs to then wrap inside the desc col (43 ch width) -#}
-{# -#}
-{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(37)}}
-{% endfor -%}
-================== ================= ===========================================
+{{ tables.paramtable(table.rows, [(table.title or "Content") ~ " Key", "Type", "Description"]) }}
 
 {% endfor %}
 Example:
diff --git a/templating/matrix_templates/templates/tables.tmpl b/templating/matrix_templates/templates/tables.tmpl
new file mode 100644
index 00000000..aba6c0c4
--- /dev/null
+++ b/templating/matrix_templates/templates/tables.tmpl
@@ -0,0 +1,104 @@
+{#
+ # A set of macros for generating RST tables
+ #}
+
+
+{#
+ # write a table for a list of parameters.
+ #
+ # 'rows' is the list of parameters. Each row should have the keys
+ # 'key', 'type', and 'desc'.
+ #}
+{% macro paramtable(rows, titles=["Parameter", "Type", "Description"]) -%}
+{{ split_paramtable({None: rows}, titles) }}
+{% endmacro %}
+
+
+{#
+ # write a table for the request parameters, split by location.
+ # 'rows_by_loc' is a map from location to a list of parameters.
+ #
+ # As a special case, if a key of 'rows_by_loc' is 'None', no title row is
+ # written for that location. This is used by the standard 'paramtable' macro.
+ #}
+{% macro split_paramtable(rows_by_loc, 
+      titles=["Parameter", "Type", "Description"]) -%}
+
+{% set rowkeys = ['key', 'type', 'desc'] %}
+{% set titlerow = {'key': titles[0], 'type': titles[1], 'desc': titles[2]} %}
+
+{# We need the rows flattened into a single list. Abuse the 'sum' filter to
+ # join arrays instead of add numbers. -#}
+{% set flatrows = rows_by_loc.values()|sum(start=[]) -%}
+
+{# Figure out the widths of the columns. The last column is always 50 characters
+ # wide; the others default to 10, but stretch if there is wider text in the
+ # column. -#}
+{% set fieldwidths = (([titlerow] + flatrows) | 
+        fieldwidths(rowkeys[0:-1], [10, 10])) + [50] -%}
+
+{{ tableheader(fieldwidths) }}
+{{ tablerow(fieldwidths, titlerow, rowkeys) }}
+{{ tableheader(fieldwidths) }}
+{% for loc in rows_by_loc -%}
+
+{% if loc != None -%}
+{{ tablespan(fieldwidths, "*" ~ loc ~ " parameters*") }}
+{% endif -%}
+
+{% for row in rows_by_loc[loc] -%}
+{{ tablerow(fieldwidths, row, rowkeys) }}
+{% endfor -%}
+{% endfor -%}
+
+{{ tableheader(fieldwidths) }}
+{% endmacro %}
+
+
+
+{#
+ # Write a table header row, for the given column widths 
+ #}
+{% macro tableheader(widths) -%}
+{% for arg in widths -%}
+{{"="*arg}} {% endfor -%}
+{% endmacro %}
+
+
+
+{#
+ # Write a normal table row. Each of 'widths' and 'keys' should be sequences 
+ # of the same length; 'widths' defines the column widths, and 'keys' the
+ # attributes of 'row' to look up for values to put in the columns.
+ #}
+{% macro tablerow(widths, row, keys) -%}
+{% for key in keys -%}
+{% set value=row[key] -%}
+{% if not loop.last -%}
+   {# the first few columns need space after them -#}
+   {{ value }}{{" "*(1+widths[loop.index0]-value|length) -}}
+{% else -%}
+   {# the last column needs wrapping and indenting (by the sum of the widths of
+   the preceding columns, plus the number of preceding columns (for the
+   separators)) -#}
+   {{ value | wrap(widths[loop.index0]) | 
+         indent_block(widths[0:-1]|sum + loop.index0) -}}
+{% endif -%}
+{% endfor -%}
+{% endmacro %}
+
+
+
+
+{# 
+ # write a tablespan row. This is a single value which spans the entire table.
+ #}
+{% macro tablespan(widths, value) -%}
+{{value}}
+{# we write a trailing space to stop the separator being misinterpreted
+ # as a header line. -#}
+{{"-"*(widths|sum + widths|length -1)}} {% endmacro %}
+
+
+
+

From 3f0262081c7a900cd1e5bcd7ca9a2991facb6ef9 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Wed, 11 Nov 2015 19:22:38 +0000
Subject: [PATCH 446/467] Update sync API defn to reflect SPEC-254 changes

State now corresponds to the start of the timeline, not the end.
---
 api/client-server/v2_alpha/sync.yaml | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index a2d5a2b8..9e921610 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -119,7 +119,11 @@ paths:
                           title: State
                           type: object
                           description: |-
-                            The state updates for the room.
+                            Updates to the state, between the time indicated by
+                            the ``since`` parameter, and the start of the
+                            ``timeline`` (or all state up to the start of the
+                            ``timeline``, if ``since`` is not given, or
+                            ``full_state`` is true).
                           allOf:
                             - $ref: "definitions/room_event_batch.json"
                         timeline:
@@ -193,8 +197,7 @@ paths:
                           title: State
                           type: object
                           description: |-
-                            The state updates for the room up to the point when
-                            the user left.
+                            The state updates for the room up to the start of the timeline.
                           allOf:
                             - $ref: "definitions/room_event_batch.json"
                         timeline:
@@ -260,7 +263,6 @@ paths:
                       "state": {
                         "events": [
                           "$66697273743031:example.com",
-                          "$7365636s6r6432:example.com"
                         ]
                       },
                       "timeline": {

From 57995a815a83ec7496283eae5511ebd89ddeb747 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Sun, 15 Nov 2015 23:30:29 +0000
Subject: [PATCH 447/467] Fix a number of untruths in the documentation about
 /sync.

Attempts to make the /sync documentation better represent fact as it currently
stands - in particular document the structure of the returned events.
---
 api/client-server/v2_alpha/core-event-schema  |  1 -
 .../v2_alpha/definitions/event.json           | 53 +++++++++++++++++++
 .../v2_alpha/definitions/event_batch.json     |  4 +-
 .../v2_alpha/definitions/timeline_batch.json  |  4 +-
 api/client-server/v2_alpha/sync.yaml          | 16 +++---
 specification/events.rst                      |  5 +-
 6 files changed, 68 insertions(+), 15 deletions(-)
 delete mode 120000 api/client-server/v2_alpha/core-event-schema
 create mode 100644 api/client-server/v2_alpha/definitions/event.json

diff --git a/api/client-server/v2_alpha/core-event-schema b/api/client-server/v2_alpha/core-event-schema
deleted file mode 120000
index b020e6da..00000000
--- a/api/client-server/v2_alpha/core-event-schema
+++ /dev/null
@@ -1 +0,0 @@
-../../../event-schemas/schema/v1/core-event-schema
\ No newline at end of file
diff --git a/api/client-server/v2_alpha/definitions/event.json b/api/client-server/v2_alpha/definitions/event.json
new file mode 100644
index 00000000..6a01c688
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/event.json
@@ -0,0 +1,53 @@
+{
+  "type": "object",
+  "title": "Event",
+  "properties": {
+    "content": {
+      "type": "object",
+      "title": "EventContent",
+      "description": "The content of this event. The fields in this object will vary depending on the type of event."
+    },
+    "origin_server_ts": {
+      "type": "integer",
+      "format": "int64",
+      "description": "Timestamp in milliseconds on originating homeserver when this event was sent."
+    },
+    "sender": {
+      "type": "string",
+      "description": "The MXID of the user who sent this event."
+    },
+    "state_key": {
+      "type": "string",
+      "description": "Optional. This key will only be present for state events. A unique key which defines the overwriting semantics for this piece of room state."
+    },
+    "type": {
+      "type": "string",
+      "description": "The type of event."
+    },
+    "unsigned": {
+      "type": "object",
+      "title": "Unsigned",
+      "description": "Information about this event which was not sent by the originating homeserver",
+      "properties": {
+        "age": {
+          "type": "integer",
+          "format": "int64",
+          "description": "Time in milliseconds since the event was sent."
+        },
+        "prev_content": {
+          "title": "EventContent",
+          "type": "object",
+          "description": "Optional. The previous ``content`` for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
+        },
+        "replaces_state": {
+          "type": "string",
+          "description": "Optional. The event_id of the previous event for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
+        },
+        "txn_id": {
+          "type": "string",
+          "description": "Optional. The transaction ID set when this message was sent. This key will only be present for message events sent by the device calling this API."
+        }
+      }
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/definitions/event_batch.json b/api/client-server/v2_alpha/definitions/event_batch.json
index 395aed13..7f489423 100644
--- a/api/client-server/v2_alpha/definitions/event_batch.json
+++ b/api/client-server/v2_alpha/definitions/event_batch.json
@@ -5,8 +5,8 @@
       "type": "array",
       "description": "List of events",
       "items": {
-        "title": "Event",
-        "type": "object"
+        "type": "object",
+        "allOf": [{"$ref": "event.json" }]
       }
     }
   }
diff --git a/api/client-server/v2_alpha/definitions/timeline_batch.json b/api/client-server/v2_alpha/definitions/timeline_batch.json
index ddf8d341..15ca1ead 100644
--- a/api/client-server/v2_alpha/definitions/timeline_batch.json
+++ b/api/client-server/v2_alpha/definitions/timeline_batch.json
@@ -4,11 +4,11 @@
   "properties": {
     "limited": {
       "type": "boolean",
-      "description": "Whether there are more events on the server"
+      "description": "True if the number of events returned was limited by the ``limit`` on the filter"
     },
     "prev_batch": {
       "type": "string",
-      "description": "If the batch was limited then this is a token that can be supplied to the server to retrieve more events"
+      "description": "If the batch was limited then this is a token that can be supplied to the server to retrieve earlier events"
     }
   }
 }
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 9e921610..06763a08 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -96,8 +96,10 @@ paths:
                   Updates to rooms.
                 properties:
                   joined:
-                    title: Joined
+                    title: Joined Rooms
                     type: object
+                    description: |-
+                      The rooms that the user has joined.
                     additionalProperties:
                       title: Joined Room
                       type: object
@@ -114,7 +116,7 @@ paths:
                             description: An event object.
                             type: object
                             allOf:
-                              - $ref: "core-event-schema/event.json"
+                              - $ref: "definitions/event.json"
                         state:
                           title: State
                           type: object
@@ -144,7 +146,7 @@ paths:
                           allOf:
                             - $ref: "definitions/event_batch.json"
                   invited:
-                    title: Invited
+                    title: Invited Rooms
                     type: object
                     description: |-
                       The rooms that the user has been invited to.
@@ -171,11 +173,10 @@ paths:
                           allOf:
                             - $ref: "definitions/event_batch.json"
                   archived:
-                    title: Archived
+                    title: Archived rooms
                     type: object
                     description: |-
-                      The rooms that the user has left or been banned from. The
-                      entries in the room_map will lack an ``ephemeral`` key.
+                      The rooms that the user has left or been banned from.
                     additionalProperties:
                       title: Archived Room
                       type: object
@@ -192,7 +193,7 @@ paths:
                             description: An event object.
                             type: object
                             allOf:
-                              - $ref: "core-event-schema/event.json"
+                              - $ref: "definitions/event.json"
                         state:
                           title: State
                           type: object
@@ -276,7 +277,6 @@ paths:
                       "ephemeral": {
                         "events": [
                           {
-                            "room_id": "!726s6s6q:example.com",
                             "type": "m.typing",
                             "content": {"user_ids": ["@alice:example.com"]}
                           }
diff --git a/specification/events.rst b/specification/events.rst
index a1aece1c..7bf08012 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -30,12 +30,13 @@ formatted for federation by:
   ``auth_events``, ``prev_events``, ``hashes``, ``signatures``, ``depth``,
   ``origin``, ``prev_state``.
 * Adding an ``age`` to the ``unsigned`` object which gives the time in
-  milliseconds that has ellapsed since the event was sent.
+  milliseconds that has elapsed since the event was sent.
 * Adding a ``prev_content`` to the ``unsigned`` object if the event is
   a ``state event`` which gives previous content of that state key.
 * Adding a ``redacted_because`` to the ``unsigned`` object if the event was
   redacted which gives the event that redacted it.
-* Adding a ``transaction_id`` if the event was sent by the client requesting it.
+* Adding a ``txn_id`` to the ``unsigned`` object if the event was sent by the
+  client requesting it.
 
 Events in responses for APIs with the /v1 prefix are generated from an event
 formatted for the /v2 prefix by:

From b41d771c1586a06e387222030f923ed5fc95ecf0 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Sun, 15 Nov 2015 23:58:04 +0000
Subject: [PATCH 448/467] Fix typos in /sync example response

---
 api/client-server/v2_alpha/sync.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 06763a08..2290b070 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -253,7 +253,7 @@ paths:
                         "$74686972643033:example.com": {
                           "sender": "@alice:example.com",
                           "type": "m.room.message",
-                          "unsigned": {"age": "124524", "txn_id": "1234"},
+                          "unsigned": {"age": 124524, "txn_id": "1234"},
                           "content": {
                             "body": "I am a fish",
                             "msgtype": "m.text"
@@ -263,7 +263,7 @@ paths:
                       },
                       "state": {
                         "events": [
-                          "$66697273743031:example.com",
+                          "$66697273743031:example.com"
                         ]
                       },
                       "timeline": {

From 29bd4d45ee9155bd80e61ce50ab34745be1fd8a4 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Sun, 15 Nov 2015 23:40:41 +0000
Subject: [PATCH 449/467] Flatten the response to /sync

Now that we don't expect there to be duplication betwen the 'timeline' and
'state' results, there's no point in having the state map. (That does mean the
events themselves need event_id fields though).

Also:

- move the contents of the 'unsigned' dictionary into the events themselves
- replace the state list with two layers of dictionary keyed on type and
  state_key
- rename the children of the 'rooms' dict from "joined/invited/archived" to
  "join/invite/leave" to match the membership states
---
 .../v2_alpha/definitions/event.json           |  45 +++---
 .../definitions/room_event_batch.json         |  12 --
 .../v2_alpha/definitions/timeline_batch.json  |   2 +-
 api/client-server/v2_alpha/sync.yaml          | 134 +++++++-----------
 specification/events.rst                      |  17 +--
 5 files changed, 84 insertions(+), 126 deletions(-)
 delete mode 100644 api/client-server/v2_alpha/definitions/room_event_batch.json

diff --git a/api/client-server/v2_alpha/definitions/event.json b/api/client-server/v2_alpha/definitions/event.json
index 6a01c688..3a15357a 100644
--- a/api/client-server/v2_alpha/definitions/event.json
+++ b/api/client-server/v2_alpha/definitions/event.json
@@ -2,16 +2,34 @@
   "type": "object",
   "title": "Event",
   "properties": {
+    "age": {
+      "type": "integer",
+      "format": "int64",
+      "description": "Time in milliseconds since the event was sent."
+    },
     "content": {
       "type": "object",
       "title": "EventContent",
       "description": "The content of this event. The fields in this object will vary depending on the type of event."
     },
+    "event_id": {
+      "type": "string",
+      "description": "Globally unique identifier for this event."
+    },
     "origin_server_ts": {
       "type": "integer",
       "format": "int64",
       "description": "Timestamp in milliseconds on originating homeserver when this event was sent."
     },
+    "prev_content": {
+      "title": "EventContent",
+      "type": "object",
+      "description": "Optional. The previous ``content`` for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
+    },
+    "prev_sender": {
+      "type": "string",
+      "description": "Optional. The ``sender`` of the previous event for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there was no previous event for this state, this key will be missing."
+    },
     "sender": {
       "type": "string",
       "description": "The MXID of the user who sent this event."
@@ -24,30 +42,9 @@
       "type": "string",
       "description": "The type of event."
     },
-    "unsigned": {
-      "type": "object",
-      "title": "Unsigned",
-      "description": "Information about this event which was not sent by the originating homeserver",
-      "properties": {
-        "age": {
-          "type": "integer",
-          "format": "int64",
-          "description": "Time in milliseconds since the event was sent."
-        },
-        "prev_content": {
-          "title": "EventContent",
-          "type": "object",
-          "description": "Optional. The previous ``content`` for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
-        },
-        "replaces_state": {
-          "type": "string",
-          "description": "Optional. The event_id of the previous event for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
-        },
-        "txn_id": {
-          "type": "string",
-          "description": "Optional. The transaction ID set when this message was sent. This key will only be present for message events sent by the device calling this API."
-        }
-      }
+    "txn_id": {
+      "type": "string",
+      "description": "Optional. The transaction ID set when this message was sent. This key will only be present for message events sent by the device calling this API."
     }
   }
 }
diff --git a/api/client-server/v2_alpha/definitions/room_event_batch.json b/api/client-server/v2_alpha/definitions/room_event_batch.json
deleted file mode 100644
index fcf148f3..00000000
--- a/api/client-server/v2_alpha/definitions/room_event_batch.json
+++ /dev/null
@@ -1,12 +0,0 @@
-{
-  "type": "object",
-  "properties": {
-    "events": {
-      "type": "array",
-      "description": "List of event ids",
-      "items": {
-        "type": "string"
-      }
-    }
-  }
-}
diff --git a/api/client-server/v2_alpha/definitions/timeline_batch.json b/api/client-server/v2_alpha/definitions/timeline_batch.json
index 15ca1ead..f27a5746 100644
--- a/api/client-server/v2_alpha/definitions/timeline_batch.json
+++ b/api/client-server/v2_alpha/definitions/timeline_batch.json
@@ -1,6 +1,6 @@
 {
   "type": "object",
-  "allOf": [{"$ref":"definitions/room_event_batch.json"}],
+  "allOf": [{"$ref":"definitions/event_batch.json"}],
   "properties": {
     "limited": {
       "type": "boolean",
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 2290b070..c63f3432 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -95,7 +95,7 @@ paths:
                 description: |-
                   Updates to rooms.
                 properties:
-                  joined:
+                  join:
                     title: Joined Rooms
                     type: object
                     description: |-
@@ -104,19 +104,6 @@ paths:
                       title: Joined Room
                       type: object
                       properties:
-                        event_map:
-                          title: EventMap
-                          type: object
-                          description: |-
-                            A map from event ID to events for this room. The
-                            events are referenced from the ``timeline`` and
-                            ``state`` keys for this room.
-                          additionalProperties:
-                            title: Event
-                            description: An event object.
-                            type: object
-                            allOf:
-                              - $ref: "definitions/event.json"
                         state:
                           title: State
                           type: object
@@ -127,7 +114,7 @@ paths:
                             ``timeline``, if ``since`` is not given, or
                             ``full_state`` is true).
                           allOf:
-                            - $ref: "definitions/room_event_batch.json"
+                            - $ref: "definitions/state_batch.json"
                         timeline:
                           title: Timeline
                           type: object
@@ -145,7 +132,7 @@ paths:
                             e.g. typing.
                           allOf:
                             - $ref: "definitions/event_batch.json"
-                  invited:
+                  invite:
                     title: Invited Rooms
                     type: object
                     description: |-
@@ -171,36 +158,23 @@ paths:
                             delta against the archived ``state`` not the
                             ``invite_state``.
                           allOf:
-                            - $ref: "definitions/event_batch.json"
-                  archived:
-                    title: Archived rooms
+                            - $ref: "definitions/state_batch.json"
+                  leave:
+                    title: Left rooms
                     type: object
                     description: |-
                       The rooms that the user has left or been banned from.
                     additionalProperties:
-                      title: Archived Room
+                      title: Left Room
                       type: object
                       properties:
-                        event_map:
-                          title: EventMap
-                          type: object
-                          description: |-
-                            A map from event ID to events for this room. The
-                            events are referenced from the ``timeline`` and
-                            ``state`` keys for this room.
-                          additionalProperties:
-                            title: Event
-                            description: An event object.
-                            type: object
-                            allOf:
-                              - $ref: "definitions/event.json"
                         state:
                           title: State
                           type: object
                           description: |-
                             The state updates for the room up to the start of the timeline.
                           allOf:
-                            - $ref: "definitions/room_event_batch.json"
+                            - $ref: "definitions/state_batch.json"
                         timeline:
                           title: Timeline
                           type: object
@@ -230,46 +204,43 @@ paths:
                   ]
                 },
                 "rooms": {
-                  "joined": {
+                  "join": {
                     "!726s6s6q:example.com": {
-                      "event_map": {
-                        "$66697273743031:example.com": {
-                          "sender": "@alice:example.com",
-                          "type": "m.room.member",
-                          "state_key": "@alice:example.com",
-                          "content": {"membership": "join"},
-                          "origin_server_ts": 1417731086795
-                        },
-                        "$7365636s6r6432:example.com": {
-                          "sender": "@bob:example.com",
-                          "type": "m.room.member",
-                          "state_key": "@bob:example.com",
-                          "content": {"membership": "join"},
-                          "unsigned": {
-                            "prev_content": {"membership": "invite"}
-                          },
-                          "origin_server_ts": 1417731086795
-                        },
-                        "$74686972643033:example.com": {
-                          "sender": "@alice:example.com",
-                          "type": "m.room.message",
-                          "unsigned": {"age": 124524, "txn_id": "1234"},
-                          "content": {
-                            "body": "I am a fish",
-                            "msgtype": "m.text"
-                          },
-                          "origin_server_ts": 1417731086797
-                        }
-                      },
                       "state": {
-                        "events": [
-                          "$66697273743031:example.com"
-                        ]
+                        "m.room.member": {
+                          "@alice:example.com": {
+                            "sender": "@alice:example.com",
+                            "type": "m.room.member",
+                            "state_key": "@alice:example.com",
+                            "content": {"membership": "join"},
+                            "origin_server_ts": 1417731086795,
+                            "event_id": "$66697273743031:example.com"
+                          }
+                        }
                       },
                       "timeline": {
                         "events": [
-                          "$7365636s6r6432:example.com",
-                          "$74686972643033:example.com"
+                          {
+                            "sender": "@bob:example.com",
+                            "type": "m.room.member",
+                            "state_key": "@bob:example.com",
+                            "content": {"membership": "join"},
+                            "prev_content": {"membership": "invite"},
+                            "origin_server_ts": 1417731086795,
+                            "event_id": "$7365636s6r6432:example.com":
+                          },
+                          {
+                            "sender": "@alice:example.com",
+                            "type": "m.room.message",
+                            "age": 124524,
+                            "txn_id": "1234",
+                            "content": {
+                              "body": "I am a fish",
+                              "msgtype": "m.text"
+                            },
+                            "origin_server_ts": 1417731086797,
+                            "event_id": "$74686972643033:example.com"
+                          }
                         ],
                         "limited": true,
                         "prev_batch": "t34-23535_0_0"
@@ -277,6 +248,7 @@ paths:
                       "ephemeral": {
                         "events": [
                           {
+                            "room_id": "!726s6s6q:example.com",
                             "type": "m.typing",
                             "content": {"user_ids": ["@alice:example.com"]}
                           }
@@ -284,26 +256,30 @@ paths:
                       }
                     }
                   },
-                  "invited": {
+                  "invite": {
                     "!696r7674:example.com": {
                       "invite_state": {
-                        "events": [
-                          {
+                        "m.room.name": {
+                          "": {
                             "sender": "@alice:example.com",
                             "type": "m.room.name",
                             "state_key": "",
-                            "content": {"name": "My Room Name"}
-                          },
-                          {
+                            "content": {"name": "My Room Name"},
+                            "event_id": "$asdkgjrsfg2314375:example.com",
+
+                          }
+                        },
+                        "m.room.member": {
+                          "@bob:example.com": {
                             "sender": "@alice:example.com",
                             "type": "m.room.member",
                             "state_key": "@bob:example.com",
-                            "content": {"membership": "invite"}
-                          }
-                        ]
+                            "content": {"membership": "invite"},
+                            "event_id": "$257kasjdg315324akhg:example.com",
+                        }
                       }
                     }
                   },
-                  "archived": {}
+                  "leave": {}
                 }
               }
diff --git a/specification/events.rst b/specification/events.rst
index 7bf08012..3d069be8 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -28,22 +28,19 @@ formatted for federation by:
 
 * Removing the following keys:
   ``auth_events``, ``prev_events``, ``hashes``, ``signatures``, ``depth``,
-  ``origin``, ``prev_state``.
-* Adding an ``age`` to the ``unsigned`` object which gives the time in
+  ``origin``, ``prev_state``, ``unsigned``.
+* Adding an ``age`` to the event object which gives the time in
   milliseconds that has elapsed since the event was sent.
-* Adding a ``prev_content`` to the ``unsigned`` object if the event is
-  a ``state event`` which gives previous content of that state key.
-* Adding a ``redacted_because`` to the ``unsigned`` object if the event was
+* Adding ``prev_content`` and ``prev_sender`` to the event object if the event
+  is a ``state event``, which give the previous content and previous sender of
+  that state key
+* Adding a ``redacted_because`` to event object if the event was
   redacted which gives the event that redacted it.
-* Adding a ``txn_id`` to the ``unsigned`` object if the event was sent by the
-  client requesting it.
+* Adding a ``txn_id`` if the event was sent by the client requesting it.
 
 Events in responses for APIs with the /v1 prefix are generated from an event
 formatted for the /v2 prefix by:
 
-* Moving the folling keys from the ``unsigned`` object to the top level event
-  object: ``age``, ``redacted_because``, ``replaces_state``, ``prev_content``.
-* Removing the ``unsigned`` object.
 * Rename the ``sender`` key to ``user_id``.
 * If the event was an ``m.room.member`` with ``membership`` set to ``invite``
   then adding a ``invite_room_state`` key to the top level event object.

From e1b12a753ecd96227effef3612f70d9adf48c0a9 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Mon, 16 Nov 2015 00:17:22 +0000
Subject: [PATCH 450/467] Fix typos and missing file

---
 .../v2_alpha/definitions/state_batch.json            | 12 ++++++++++++
 api/client-server/v2_alpha/sync.yaml                 |  8 ++++----
 2 files changed, 16 insertions(+), 4 deletions(-)
 create mode 100644 api/client-server/v2_alpha/definitions/state_batch.json

diff --git a/api/client-server/v2_alpha/definitions/state_batch.json b/api/client-server/v2_alpha/definitions/state_batch.json
new file mode 100644
index 00000000..45728f40
--- /dev/null
+++ b/api/client-server/v2_alpha/definitions/state_batch.json
@@ -0,0 +1,12 @@
+{
+  "type": "object",
+  "additionalProperties": {
+    "type": "object",
+    "x-pattern": "$EVENT_TYPE",
+    "additionalProperties": {
+      "type": "object",
+      "x-pattern": "$STATE_KEY",
+      "allOf": [{"$ref": "event.json" }]
+    }
+  }
+}
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index c63f3432..198f4833 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -227,7 +227,7 @@ paths:
                             "content": {"membership": "join"},
                             "prev_content": {"membership": "invite"},
                             "origin_server_ts": 1417731086795,
-                            "event_id": "$7365636s6r6432:example.com":
+                            "event_id": "$7365636s6r6432:example.com"
                           },
                           {
                             "sender": "@alice:example.com",
@@ -265,8 +265,7 @@ paths:
                             "type": "m.room.name",
                             "state_key": "",
                             "content": {"name": "My Room Name"},
-                            "event_id": "$asdkgjrsfg2314375:example.com",
-
+                            "event_id": "$asdkgjrsfg2314375:example.com"
                           }
                         },
                         "m.room.member": {
@@ -275,7 +274,8 @@ paths:
                             "type": "m.room.member",
                             "state_key": "@bob:example.com",
                             "content": {"membership": "invite"},
-                            "event_id": "$257kasjdg315324akhg:example.com",
+                            "event_id": "$257kasjdg315324akhg:example.com"
+                          }
                         }
                       }
                     }

From 96be7ff24140ec80937db48da300c6f4e05473e9 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Mon, 16 Nov 2015 15:00:31 +0000
Subject: [PATCH 451/467] Support more nesting

---
 templating/matrix_templates/units.py | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/templating/matrix_templates/units.py b/templating/matrix_templates/units.py
index adb7f427..e5a7c319 100644
--- a/templating/matrix_templates/units.py
+++ b/templating/matrix_templates/units.py
@@ -122,10 +122,12 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
                         "x-pattern", "string"
                     )
                     value_type = "{%s: %s}" % (key, nested_object[0]["title"])
+                    value_id = "%s: %s" % (key, nested_object[0]["title"])
                     if not nested_object[0].get("no-table"):
                         tables += nested_object
                 else:
-                    value_type = "{string: %s}" % prop_val
+                    value_type = "{string: %s}" % (prop_val,)
+                    value_id = "string: %s" % (prop_val,)
             else:
                 nested_object = get_json_schema_object_fields(
                     props[key_name],
@@ -133,6 +135,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
                     include_parents=include_parents,
                 )
                 value_type = "{%s}" % nested_object[0]["title"]
+                value_id = "%s" % (nested_object[0]["title"],)
 
                 if not nested_object[0].get("no-table"):
                     tables += nested_object
@@ -145,12 +148,14 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
                     include_parents=include_parents,
                 )
                 value_type = "[%s]" % nested_object[0]["title"]
+                value_id = "%s" % (nested_object[0]["title"],)
                 tables += nested_object
             else:
                 value_type = props[key_name]["items"]["type"]
                 if isinstance(value_type, list):
                     value_type = " or ".join(value_type)
                 value_type = "[%s]" % value_type
+                value_id = "%s" % (value_type,)
                 array_enums = props[key_name]["items"].get("enum")
                 if array_enums:
                     if len(array_enums) > 1:
@@ -164,6 +169,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
                         )
         else:
             value_type = props[key_name]["type"]
+            value_id = props[key_name]["type"]
             if props[key_name].get("enum"):
                 if len(props[key_name].get("enum")) > 1:
                     value_type = "enum"
@@ -184,6 +190,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
         fields["rows"].append({
             "key": key_name,
             "type": value_type,
+            "id": value_id,
             "required": required,
             "desc": desc,
             "req_str": "**Required.** " if required else ""
@@ -313,10 +320,16 @@ class MatrixUnits(Units):
 
                             if req_tables > 1:
                                 for table in req_tables[1:]:
-                                    nested_key_name = [
-                                        s["key"] for s in req_tables[0]["rows"] if
-                                        s["type"] == ("{%s}" % (table["title"],))
-                                    ][0]
+                                    nested_key_name = {
+                                        "key": s["key"]
+                                        for rtable in req_tables
+                                        for s in rtable["rows"]
+                                        if s["id"] == table["title"]
+                                    }.get("key", None)
+
+                                    if nested_key_name is None:
+                                        raise Exception("Failed to find table for %r" % (table["title"],))
+
                                     for row in table["rows"]:
                                         row["key"] = "%s.%s" % (nested_key_name, row["key"])
 

From 8648f86032bea6c7576fa92a4bd8ea8e0fdd9793 Mon Sep 17 00:00:00 2001
From: Kegan Dougal <kegan@matrix.org>
Date: Mon, 16 Nov 2015 15:08:37 +0000
Subject: [PATCH 452/467] Moar spaces

---
 templating/matrix_templates/templates/http-api.tmpl | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/templating/matrix_templates/templates/http-api.tmpl b/templating/matrix_templates/templates/http-api.tmpl
index d7258e98..4a9491f3 100644
--- a/templating/matrix_templates/templates/http-api.tmpl
+++ b/templating/matrix_templates/templates/http-api.tmpl
@@ -13,17 +13,17 @@
 
 Request format:
 {% if (endpoint.req_param_by_loc | length) %}
-=========================================== ================= ===========================================
-   Parameter                                 Value                        Description
-=========================================== ================= ===========================================
+=============================================================== ================= ===========================================
+                       Parameter                                 Value                        Description
+=============================================================== ================= ===========================================
 {% for loc in endpoint.req_param_by_loc -%}
 *{{loc}} parameters*
---------------------------------------------------------------------------------------------------------- 
+----------------------------------------------------------------------------------------------------------------------------- 
 {% for param in endpoint.req_param_by_loc[loc] -%}
-{{param.key}}{{param.type|indent(44-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(62)}}
+{{param.key}}{{param.type|indent(64-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(82)}}
 {% endfor -%}
 {% endfor -%}
-=========================================== ================= ===========================================
+=============================================================== ================= ===========================================
 {% else %}
 `No parameters`
 {% endif %}

From 36af793f05992d0bef5ef325c303a753f16f24e0 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 16 Nov 2015 15:30:22 +0000
Subject: [PATCH 453/467] s/full object/full event/

---
 specification/server_server_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 9cc10141..127a3572 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -612,7 +612,7 @@ attempting to join.
 The resident server replies to this request with a JSON-encoded object having a
 single key called ``event``; within this is an object whose fields contain some
 of the information that the joining server will need. Despite its name, this
-object is not a full object; notably it does not need to be hashed or signed by
+object is not a full event; notably it does not need to be hashed or signed by
 the assisting resident. The required fields are:
 
 ==================== ======== ============

From 923f05e5541f354c2003acf77dbe4e2c3d6f90ec Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 16 Nov 2015 15:34:11 +0000
Subject: [PATCH 454/467] More consistency around 'resident homeserver'

---
 specification/server_server_api.rst | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 127a3572..4c315edc 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -613,7 +613,7 @@ The resident server replies to this request with a JSON-encoded object having a
 single key called ``event``; within this is an object whose fields contain some
 of the information that the joining server will need. Despite its name, this
 object is not a full event; notably it does not need to be hashed or signed by
-the assisting resident. The required fields are:
+the resident homeserver. The required fields are:
 
 ==================== ======== ============
  Key                  Type     Description
@@ -625,9 +625,9 @@ the assisting resident. The required fields are:
 ``content``          Object   The event content
 ``depth``            Integer  (this field must be present but is ignored; it
                               may be 0)
-``event_id``         String   A new event ID specified by the assisting
-                              resident
-``origin``           String   The name of the assisting resident homeserver
+``event_id``         String   A new event ID specified by the resident
+                              homeserver
+``origin``           String   The name of the resident homeserver
 ``origin_server_ts`` Integer  A timestamp added by the resident homeserver
 ``prev_events``      List     An event-reference list containing the immediate
                               predecessor events
@@ -665,12 +665,12 @@ algorithm to it, resulting in the addition of the ``hashes`` and ``signatures``
 fields.
 
 To complete the join handshake, the joining server must now submit this new
-event to an assisting resident, by using the ``send_join`` endpoint. This is
+event to an resident homeserver, by using the ``send_join`` endpoint. This is
 invoked using the room ID and the event ID of the new member event.
 
-The assisting resident then accepts this event into the room's event graph, and
-responds to the joining server with the full set of state for the newly-joined
-room. This is returned as a two-element list, whose first element is the
+The resident homeserver then accepts this event into the room's event graph,
+and responds to the joining server with the full set of state for the newly-
+joined room. This is returned as a two-element list, whose first element is the
 integer 200, and whose second element contains the following keys:
 
 ============== ===== ============
@@ -919,6 +919,6 @@ Querying directory information::
 
 The list of join candidates is a list of server names that are likely to hold
 the given room; these are servers that the requesting server may wish to use as
-assisting resident servers as part of the remote join handshake. This list may
-or may not include the server answering the query.
+resident servers as part of the remote join handshake. This list may or may not
+include the server answering the query.
 

From 122c082fcf0ef14cb93e09d1b8d2b7469866c786 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 16 Nov 2015 16:29:43 +0000
Subject: [PATCH 455/467] Comment about origin servers of invites having
 subsequently left the room

---
 specification/server_server_api.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 4c315edc..20026890 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -600,7 +600,9 @@ request the room ID and join candidates. This is covered in more detail on the
 directory server documentation, below. In the case of a new user joining a
 room as a result of a received invite, the joining user's homeserver could
 optimise this step away by picking the origin server of that invite message as
-the join candidate.
+the join candidate. However, the joining server should be aware that the origin
+server of the invite might since have left the room, so should be prepared to
+fall back on the regular join flow if this optimisation fails.
 
 Once the joining server has the room ID and the join candidates, it then needs
 to obtain enough information about the room to fill in the required fields of

From 6cbfba70113e302985c4f93af181141e7b0c84f1 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 16 Nov 2015 16:33:26 +0000
Subject: [PATCH 456/467] 'auth_events' is a List, not a String

---
 specification/server_server_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 20026890..6c66b3d9 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -621,7 +621,7 @@ the resident homeserver. The required fields are:
  Key                  Type     Description
 ==================== ======== ============
 ``type``             String   The value ``m.room.member``
-``auth_events``      String   An event-reference list containing the
+``auth_events``      List     An event-reference list containing the
                               authorization events that would allow this member
                               to join
 ``content``          Object   The event content

From 83168813939ea1dc4c5a778781cb12e1f8eb9d87 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Mon, 16 Nov 2015 16:50:54 +0000
Subject: [PATCH 457/467] txn_id field in events is called transaction_id

---
 api/client-server/v2_alpha/definitions/event.json | 2 +-
 api/client-server/v2_alpha/sync.yaml              | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/api/client-server/v2_alpha/definitions/event.json b/api/client-server/v2_alpha/definitions/event.json
index 6a01c688..5a8f52f6 100644
--- a/api/client-server/v2_alpha/definitions/event.json
+++ b/api/client-server/v2_alpha/definitions/event.json
@@ -43,7 +43,7 @@
           "type": "string",
           "description": "Optional. The event_id of the previous event for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
         },
-        "txn_id": {
+        "transaction_id": {
           "type": "string",
           "description": "Optional. The transaction ID set when this message was sent. This key will only be present for message events sent by the device calling this API."
         }
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 2290b070..d23929df 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -253,7 +253,7 @@ paths:
                         "$74686972643033:example.com": {
                           "sender": "@alice:example.com",
                           "type": "m.room.message",
-                          "unsigned": {"age": 124524, "txn_id": "1234"},
+                          "unsigned": {"age": 124524, "transaction_id": "1234"},
                           "content": {
                             "body": "I am a fish",
                             "msgtype": "m.text"

From 233e8486bc41e4ad87fc8e06a57bbcd4b8579974 Mon Sep 17 00:00:00 2001
From: "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk>
Date: Mon, 16 Nov 2015 16:50:58 +0000
Subject: [PATCH 458/467] Wording fix - objects contain keys, not list elements
 directly

---
 specification/server_server_api.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst
index 6c66b3d9..cc9426e2 100644
--- a/specification/server_server_api.rst
+++ b/specification/server_server_api.rst
@@ -673,7 +673,8 @@ invoked using the room ID and the event ID of the new member event.
 The resident homeserver then accepts this event into the room's event graph,
 and responds to the joining server with the full set of state for the newly-
 joined room. This is returned as a two-element list, whose first element is the
-integer 200, and whose second element contains the following keys:
+integer 200, and whose second element is an object which contains the
+following keys:
 
 ============== ===== ============
  Key            Type  Description

From 0db055b4ea22f4d29064728eaaaf2c37a2f9d926 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Mon, 16 Nov 2015 16:54:57 +0000
Subject: [PATCH 459/467] Fix another reference to 'txn_id'

---
 specification/events.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification/events.rst b/specification/events.rst
index 7bf08012..9d0be55f 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -35,8 +35,8 @@ formatted for federation by:
   a ``state event`` which gives previous content of that state key.
 * Adding a ``redacted_because`` to the ``unsigned`` object if the event was
   redacted which gives the event that redacted it.
-* Adding a ``txn_id`` to the ``unsigned`` object if the event was sent by the
-  client requesting it.
+* Adding a ``transaction_id`` to the ``unsigned`` object if the event was sent
+  by the client requesting it.
 
 Events in responses for APIs with the /v1 prefix are generated from an event
 formatted for the /v2 prefix by:

From 03a0377c76861eb1c9af503b718b31508dcdd7dc Mon Sep 17 00:00:00 2001
From: Mark Haines <mark.haines@matrix.org>
Date: Mon, 16 Nov 2015 20:07:42 +0000
Subject: [PATCH 460/467] Plans for end-to-end in matrix

---
 drafts/markjh_end_to_end.rst | 142 +++++++++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)
 create mode 100644 drafts/markjh_end_to_end.rst

diff --git a/drafts/markjh_end_to_end.rst b/drafts/markjh_end_to_end.rst
new file mode 100644
index 00000000..1699f79b
--- /dev/null
+++ b/drafts/markjh_end_to_end.rst
@@ -0,0 +1,142 @@
+Goals of Key-Distribution in Matrix
+===================================
+
+* No Central Authority: Users should not need to trust a central authority
+  when determining the authenticity of keys.
+
+* Easy to Add New Devices: It should be easy for a user to start using a
+  new device.
+
+* Possible to discover MITM: It should be possible for a user to determine if
+  they are being MITM.
+
+* Lost Devices: It should be possible for a user to recover if they lose all
+  their devices.
+
+* No Copying Keys: Keys should be per device and shouldn't leave the device
+  they were created on.
+
+A Possible Mechanism for Key Distribution
+=========================================
+
+Basic API for setting up keys on a server:
+
+https://github.com/matrix-org/matrix-doc/pull/24
+
+Client shouldn't trust the keys unless they have been verified, e.g by
+comparing fingerprints.
+
+If a user adds a new device it should some yet to be specified protocol
+communicate with an old device and obtain a cross-signature from the old
+device for its public key.
+
+The new device can then present the cross-signed key to all the devices
+that the user is in conversations with. Those devices should then include
+the new device into those conversations.
+
+If the user cannot cross-sign the new key, e.g. because their old device
+is lost or stolen. Then they will need to reauthenticate their conversations
+out of band, e.g by comparing fingerprints.
+
+
+Goals of End-to-end encryption in Matrix
+========================================
+
+* Access to Chat History: Users should be able to see the history of a
+  conversation on a new device. User should be able to control who can
+  see their chat history and how much of the chat history they can see.
+
+* Forward Secrecy of Discarded Chat History: Users should be able to discard
+  history from their device, once they have discarded the history it should be
+  impossible for an adversary to recover that history.
+
+* Forward Secrecy of Future Messages: Users should be able to recover from
+  disclosure of the chat history on their device.
+
+* Deniablity of Chat History: It should not be possible to prove to a third
+  party that a given user sent a message.
+
+* Authenticity of Chat History: It should be possible to prove amoungst
+  the members of a chat that a message sent by a user was authored by that
+  user.
+
+
+Bonus Goals:
+
+* Traffic Analysis: It would be nice if the protocol was resilient to traffic
+  or metadata analysis. However it's not something we want to persue if it
+  harms the usability of the protocol. It might be cool if there was a
+  way for the user to could specify the trade off between performance and
+  resilience to traffic analysis that they wanted.
+
+
+A Possible Design for Group Chat using Olm
+==========================================
+
+Protecting the secrecy of history
+---------------------------------
+
+Each message sent by a client has a 32-bit counter. This counter increments
+by one for each message sent by the client. This counter is used to advance a
+ratchet. The ratchet is split into a vector four 256-bit values,
+:math:`R_{n,j}` for :math:`j \in {0,1,2,3}`. The ratchet can be advanced as
+follows:
+
+.. math::
+    \begin{align}
+    R_{2^24n,0} &= H_1\left(R_{2^24(i-1),0}\right) \\
+    R_{2^24n,1} &= H_2\left(R_{2^24(i-1),0}\right) \\
+    R_{2^16n,1} &= H_1\left(R_{2^16(i-1),1}\right) \\
+    R_{2^16n,2} &= H_2\left(R_{2^16(i-1),1}\right) \\
+    R_{2^8i,2}  &= H_1\left(R_{2^8(i-1),2}\right) \\
+    R_{2^8i,3}  &= H_2\left(R_{2^8(i-1),2}\right) \\
+    R_{i,3}     &= H_1\left(R_{(i-1),3}\right)
+    \end{align}
+
+Where :math:`H_1` and :math:`H_2` are different hash functions. For example
+:math:`H_1` could be :math:`HMAC\left(X,\text{"\textbackslash x01"}\right)` and
+:math:`H_2` could be :math:`HMAC\left(X,\text{"\textbackslash x02"}\right)`.
+
+So every :math:`2^24` iterations :math:`R_{n,1}` is reseeded from :math:`R_{n,0}`.
+Every :math:`2^16` iterations :math:`R_{n,2}` is reseeded from :math:`R_{n,1}`.
+Every :math:`2^8` iterations :math:`R_{n,3}` is reseeded from :math:`R_{n,2}`.
+
+This scheme allows the ratchet to be advanced an arbitrary amount forwards
+while needing only 1024 hash computations.
+
+This the value of the ratchet is hashed to generate the keys used to encrypt
+each mesage.
+
+A client can decrypt chat history onwards from the earliest value of the
+ratchet it is aware of. But cannot decrypt history from before that point
+without reversing the hash function.
+
+This allows a client to share its ability to decrypt chat history with another
+from a point in the conversation onwards by giving a copy of the ratchet at
+that point in the conversation.
+
+A client can discard history by advancing a ratchet to beyond the last message
+they want to discard and then forgetting all previous values of the ratchet.
+
+Proving and denying the authenticity of history
+-----------------------------------------------
+
+Client sign the messages they send using a Ed25519 key generated per
+conversation. That key, along with the ratchet key, is distributed
+to other clients using 1:1 olm ratchets. Those 1:1 ratchets are started using
+Triple Diffie-Hellman which provides authenticity of the messages to the
+participants and deniability of the messages to third parties. Therefore
+any keys shared over those keys inherit the same levels of deniability and
+authenticity.
+
+Protecting the secrecy of future messages
+-----------------------------------------
+
+A client would need to generate new keys if it wanted to prevent access to
+messages beyond a given point in the conversation. It must generate new keys
+whenever someone leaves the room. It should generate new keys periodically
+anyway.
+
+The frequency of key generation in a large room may need to be restricted to
+keep the frequency of messages broadcast over the individual 1:1 channels
+low.

From 8201eaa042875feb69389f14262356fd3cfa73ed Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 17 Nov 2015 15:31:10 +0000
Subject: [PATCH 461/467] Swaggerify /rooms/:room_id/leave

---
 .../v1/{membership.yaml => inviting.yaml}     | 51 +-------------
 api/client-server/v1/joining.yaml             | 68 +++++++++++++++++++
 api/client-server/v1/leaving.yaml             | 57 ++++++++++++++++
 specification/client_server_api.rst           | 32 ++-------
 4 files changed, 132 insertions(+), 76 deletions(-)
 rename api/client-server/v1/{membership.yaml => inviting.yaml} (61%)
 create mode 100644 api/client-server/v1/joining.yaml
 create mode 100644 api/client-server/v1/leaving.yaml

diff --git a/api/client-server/v1/membership.yaml b/api/client-server/v1/inviting.yaml
similarity index 61%
rename from api/client-server/v1/membership.yaml
rename to api/client-server/v1/inviting.yaml
index c089e154..0a028710 100644
--- a/api/client-server/v1/membership.yaml
+++ b/api/client-server/v1/inviting.yaml
@@ -1,6 +1,6 @@
 swagger: '2.0'
 info:
-  title: "Matrix Client-Server v1 Room Membership API"
+  title: "Matrix Client-Server v1 Room Joining API"
   version: "1.0.0"
 host: localhost:8008
 schemes:
@@ -18,55 +18,6 @@ securityDefinitions:
     name: access_token
     in: query
 paths:
-  "/rooms/{roomId}/join":
-    post:
-      summary: Start the requesting user participating in a particular room.
-      description: |-
-        This API starts a user participating in a particular room, if that user
-        is allowed to participate in that room. After this call, the client is
-        allowed to see all current state events in the room, and all subsequent
-        events associated with the room until the user leaves the room.
-
-        After a user has joined a room, the room will appear as an entry in the
-        response of the |initialSync| API.
-      security:
-        - accessToken: []
-      parameters:
-        - in: path
-          type: string
-          name: roomId
-          description: The room identifier or room alias to join.
-          required: true
-          x-example: "#monkeys:matrix.org"
-      responses:
-        200:
-          description: |-
-            The room has been joined.
-
-            The joined room ID must be returned in the ``room_id`` field.
-          examples:
-            application/json: |-
-              {"room_id": "!d41d8cd:matrix.org"}
-          schema:
-            type: object
-        403:
-          description: |-
-            You do not have permission to join the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejection are:
-
-            - The room is invite-only and the user was not invited.
-            - The user has been banned from the room.
-          examples:
-            application/json: |-
-              {"errcode": "M_FORBIDDEN", "error": "You are not invited to this room."}
-        429:
-          description: This request was rate-limited.
-          schema:
-            "$ref": "definitions/error.yaml"
-      x-alias:
-        canonical-link: "post-matrix-client-api-v1-rooms-roomid-join"
-        aliases:
-          - /join/{roomId}
-
   # With an extra " " to disambiguate from the 3pid invite endpoint
   # The extra space makes it sort first for what I'm sure is a good reason.
   "/rooms/{roomId}/invite ":
diff --git a/api/client-server/v1/joining.yaml b/api/client-server/v1/joining.yaml
new file mode 100644
index 00000000..20fdbd65
--- /dev/null
+++ b/api/client-server/v1/joining.yaml
@@ -0,0 +1,68 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Room Inviting API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/join":
+    post:
+      summary: Start the requesting user participating in a particular room.
+      description: |-
+        This API starts a user participating in a particular room, if that user
+        is allowed to participate in that room. After this call, the client is
+        allowed to see all current state events in the room, and all subsequent
+        events associated with the room until the user leaves the room.
+
+        After a user has joined a room, the room will appear as an entry in the
+        response of the |initialSync| API.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier or room alias to join.
+          required: true
+          x-example: "#monkeys:matrix.org"
+      responses:
+        200:
+          description: |-
+            The room has been joined.
+
+            The joined room ID must be returned in the ``room_id`` field.
+          examples:
+            application/json: |-
+              {"room_id": "!d41d8cd:matrix.org"}
+          schema:
+            type: object
+        403:
+          description: |-
+            You do not have permission to join the room. A meaningful ``errcode`` and description error text will be returned. Example reasons for rejection are:
+
+            - The room is invite-only and the user was not invited.
+            - The user has been banned from the room.
+          examples:
+            application/json: |-
+              {"errcode": "M_FORBIDDEN", "error": "You are not invited to this room."}
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
+      x-alias:
+        canonical-link: "post-matrix-client-api-v1-rooms-roomid-join"
+        aliases:
+          - /join/{roomId}
diff --git a/api/client-server/v1/leaving.yaml b/api/client-server/v1/leaving.yaml
new file mode 100644
index 00000000..376b2060
--- /dev/null
+++ b/api/client-server/v1/leaving.yaml
@@ -0,0 +1,57 @@
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server v1 Room Leaving API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/api/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  accessToken:
+    type: apiKey
+    description: The user_id or application service access_token
+    name: access_token
+    in: query
+paths:
+  "/rooms/{roomId}/leave":
+    post:
+      summary: Stop the requesting user participating in a particular room.
+      description: |-
+        This API stops a user participating in a particular room.
+
+        If the user was already in the room, they will no longer be able to see
+        new events in the room. If the room requires an invite to join, they
+        will need to be re-invited before they can re-join.
+
+        If the user was invited to the room, but had not joined, this call
+        serves to reject the invite.
+
+        The user will still be allowed to retrieve history from the room which
+        they were previously allowed to see.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier to leave.
+          required: true
+          x-example: "!nkl290a:matrix.org"
+      responses:
+        200:
+          description: |-
+            The room has been left.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"
diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst
index 05b6ff3c..ba5578bc 100644
--- a/specification/client_server_api.rst
+++ b/specification/client_server_api.rst
@@ -870,42 +870,22 @@ following values:
 ``invite``
   This room can only be joined if you were invited.
 
-{{membership_http_api}}
+{{inviting_http_api}}
+
+{{joining_http_api}}
 
 Leaving rooms
 ~~~~~~~~~~~~~
-.. TODO-spec - HS deleting rooms they are no longer a part of. Not implemented.
-  - This is actually Very Tricky. If all clients a HS is serving leave a room,
-  the HS will no longer get any new events for that room, because the servers
-  who get the events are determined on the *membership list*. There should
-  probably be a way for a HS to lurk on a room even if there are 0 of their
-  members in the room.
-  - Grace period before deletion?
-  - Under what conditions should a room NOT be purged?
-
-
 A user can leave a room to stop receiving events for that room. A user must
 have been invited to or have joined the room before they are eligible to leave
 the room. Leaving a room to which the user has been invited rejects the invite.
+Once a user leaves a room, it will no longer appear on the |initialSync|_ API.
 
 Whether or not they actually joined the room, if the room is
 an "invite-only" room they will need to be re-invited before they can re-join
-the room.  To leave a room, a request should be made to
-|/rooms/<room_id>/leave|_ with::
-
-  {}
-
-Alternatively, the membership state for this user in this room can be modified
-directly by sending the following request to
-``/rooms/<room id>/state/m.room.member/<url encoded user id>``::
-
-  {
-    "membership": "leave"
-  }
+the room.
 
-See the `Room events`_ section for more information on ``m.room.member``. Once a
-user has left a room, that room will no longer appear on the |initialSync|_ API.
-If all members in a room leave, that room becomes eligible for deletion.
+{{leaving_http_api}}
 
 Banning users in a room
 ~~~~~~~~~~~~~~~~~~~~~~~

From 6763317e64645945224b7c68ac385ed055fe80c3 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Tue, 17 Nov 2015 10:33:46 -0500
Subject: [PATCH 462/467] Specify /rooms/:room_id/forget

---
 api/client-server/v1/leaving.yaml | 35 +++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/api/client-server/v1/leaving.yaml b/api/client-server/v1/leaving.yaml
index 376b2060..e81d812b 100644
--- a/api/client-server/v1/leaving.yaml
+++ b/api/client-server/v1/leaving.yaml
@@ -55,3 +55,38 @@ paths:
           description: This request was rate-limited.
           schema:
             "$ref": "definitions/error.yaml"
+  "/rooms/{roomId}/forget":
+    post:
+      summary: Stop the requesting user remembering about a particular room.
+      description: |-
+        This API stops a user remembering about a particular room.
+
+        In general, history is a first class citizen in Matrix. After this API
+        is called, however, a user will no longer be able to retrieve history
+        for this room. If all users on a homeserver forget a room, the room is
+        eligible for deletion from that homeserver.
+
+        If the user is currently joined to the room, they will implicitly leave
+        the room as part of this API call.
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room identifier to forget.
+          required: true
+          x-example: "!au1ba7o:matrix.org"
+      responses:
+        200:
+          description: |-
+            The room has been forgotten.
+          examples:
+            application/json: |-
+              {}
+          schema:
+            type: object
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/error.yaml"

From 6653362f315d1379d1dda248dfd299c8c3c8530d Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Wed, 18 Nov 2015 15:15:21 +0000
Subject: [PATCH 463/467] Unflatten 'unsigned'

It turns out that flattening 'unsigned' comes with too many downsides. Let's
stick with the status quo.
---
 .../v2_alpha/definitions/event.json           | 45 ++++++++++---------
 specification/events.rst                      | 19 ++++----
 2 files changed, 35 insertions(+), 29 deletions(-)

diff --git a/api/client-server/v2_alpha/definitions/event.json b/api/client-server/v2_alpha/definitions/event.json
index 3a15357a..5a8f52f6 100644
--- a/api/client-server/v2_alpha/definitions/event.json
+++ b/api/client-server/v2_alpha/definitions/event.json
@@ -2,34 +2,16 @@
   "type": "object",
   "title": "Event",
   "properties": {
-    "age": {
-      "type": "integer",
-      "format": "int64",
-      "description": "Time in milliseconds since the event was sent."
-    },
     "content": {
       "type": "object",
       "title": "EventContent",
       "description": "The content of this event. The fields in this object will vary depending on the type of event."
     },
-    "event_id": {
-      "type": "string",
-      "description": "Globally unique identifier for this event."
-    },
     "origin_server_ts": {
       "type": "integer",
       "format": "int64",
       "description": "Timestamp in milliseconds on originating homeserver when this event was sent."
     },
-    "prev_content": {
-      "title": "EventContent",
-      "type": "object",
-      "description": "Optional. The previous ``content`` for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
-    },
-    "prev_sender": {
-      "type": "string",
-      "description": "Optional. The ``sender`` of the previous event for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there was no previous event for this state, this key will be missing."
-    },
     "sender": {
       "type": "string",
       "description": "The MXID of the user who sent this event."
@@ -42,9 +24,30 @@
       "type": "string",
       "description": "The type of event."
     },
-    "txn_id": {
-      "type": "string",
-      "description": "Optional. The transaction ID set when this message was sent. This key will only be present for message events sent by the device calling this API."
+    "unsigned": {
+      "type": "object",
+      "title": "Unsigned",
+      "description": "Information about this event which was not sent by the originating homeserver",
+      "properties": {
+        "age": {
+          "type": "integer",
+          "format": "int64",
+          "description": "Time in milliseconds since the event was sent."
+        },
+        "prev_content": {
+          "title": "EventContent",
+          "type": "object",
+          "description": "Optional. The previous ``content`` for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
+        },
+        "replaces_state": {
+          "type": "string",
+          "description": "Optional. The event_id of the previous event for this state. This will be present only for state events appearing in the ``timeline``. If this is not a state event, or there is no previous content, this key will be missing."
+        },
+        "transaction_id": {
+          "type": "string",
+          "description": "Optional. The transaction ID set when this message was sent. This key will only be present for message events sent by the device calling this API."
+        }
+      }
     }
   }
 }
diff --git a/specification/events.rst b/specification/events.rst
index eb48ca17..5a003115 100644
--- a/specification/events.rst
+++ b/specification/events.rst
@@ -28,20 +28,23 @@ formatted for federation by:
 
 * Removing the following keys:
   ``auth_events``, ``prev_events``, ``hashes``, ``signatures``, ``depth``,
-  ``origin``, ``prev_state``, ``unsigned``.
-* Adding an ``age`` to the event object which gives the time in
+  ``origin``, ``prev_state``.
+* Adding an ``age`` to the ``unsigned`` object which gives the time in
   milliseconds that has elapsed since the event was sent.
-* Adding ``prev_content`` and ``prev_sender`` to the event object if the event
-  is a ``state event``, which give the previous content and previous sender of
-  that state key
-* Adding a ``redacted_because`` to event object if the event was
+* Adding ``prev_content`` and ``prev_sender`` to the ``unsigned`` object if the
+  event is a ``state event``, which give the previous content and previous
+  sender of that state key
+* Adding a ``redacted_because`` to the ``unsigned`` object if the event was
   redacted which gives the event that redacted it.
-* Adding a ``txn_id`` to the event object if the event was sent by the client
-  requesting it.
+* Adding a ``transaction_id`` to the ``unsigned`` object if the event was sent
+  by the client requesting it.
 
 Events in responses for APIs with the /v1 prefix are generated from an event
 formatted for the /v2 prefix by:
 
+* Moving the folling keys from the ``unsigned`` object to the top level event
+  object: ``age``, ``redacted_because``, ``replaces_state``, ``prev_content``.
+* Removing the ``unsigned`` object.
 * Rename the ``sender`` key to ``user_id``.
 * If the event was an ``m.room.member`` with ``membership`` set to ``invite``
   then adding a ``invite_room_state`` key to the top level event object.

From d7d59d78e1dca06407dea20ad875f64569844ad3 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Wed, 18 Nov 2015 16:17:29 +0000
Subject: [PATCH 464/467] /sync: Put state dict back to being a list

Turning the state into a dict-of-dicts caused more pain than it solved. Put it
back to a list.
---
 .../v2_alpha/definitions/state_batch.json     | 12 -------
 api/client-server/v2_alpha/sync.yaml          | 31 ++++++++-----------
 2 files changed, 13 insertions(+), 30 deletions(-)
 delete mode 100644 api/client-server/v2_alpha/definitions/state_batch.json

diff --git a/api/client-server/v2_alpha/definitions/state_batch.json b/api/client-server/v2_alpha/definitions/state_batch.json
deleted file mode 100644
index 45728f40..00000000
--- a/api/client-server/v2_alpha/definitions/state_batch.json
+++ /dev/null
@@ -1,12 +0,0 @@
-{
-  "type": "object",
-  "additionalProperties": {
-    "type": "object",
-    "x-pattern": "$EVENT_TYPE",
-    "additionalProperties": {
-      "type": "object",
-      "x-pattern": "$STATE_KEY",
-      "allOf": [{"$ref": "event.json" }]
-    }
-  }
-}
diff --git a/api/client-server/v2_alpha/sync.yaml b/api/client-server/v2_alpha/sync.yaml
index 198f4833..0c9d6186 100644
--- a/api/client-server/v2_alpha/sync.yaml
+++ b/api/client-server/v2_alpha/sync.yaml
@@ -114,7 +114,7 @@ paths:
                             ``timeline``, if ``since`` is not given, or
                             ``full_state`` is true).
                           allOf:
-                            - $ref: "definitions/state_batch.json"
+                            - $ref: "definitions/event_batch.json"
                         timeline:
                           title: Timeline
                           type: object
@@ -158,7 +158,7 @@ paths:
                             delta against the archived ``state`` not the
                             ``invite_state``.
                           allOf:
-                            - $ref: "definitions/state_batch.json"
+                            - $ref: "definitions/event_batch.json"
                   leave:
                     title: Left rooms
                     type: object
@@ -174,7 +174,7 @@ paths:
                           description: |-
                             The state updates for the room up to the start of the timeline.
                           allOf:
-                            - $ref: "definitions/state_batch.json"
+                            - $ref: "definitions/event_batch.json"
                         timeline:
                           title: Timeline
                           type: object
@@ -207,8 +207,8 @@ paths:
                   "join": {
                     "!726s6s6q:example.com": {
                       "state": {
-                        "m.room.member": {
-                          "@alice:example.com": {
+                        "events": [
+                          {
                             "sender": "@alice:example.com",
                             "type": "m.room.member",
                             "state_key": "@alice:example.com",
@@ -216,7 +216,7 @@ paths:
                             "origin_server_ts": 1417731086795,
                             "event_id": "$66697273743031:example.com"
                           }
-                        }
+                        ]
                       },
                       "timeline": {
                         "events": [
@@ -248,7 +248,6 @@ paths:
                       "ephemeral": {
                         "events": [
                           {
-                            "room_id": "!726s6s6q:example.com",
                             "type": "m.typing",
                             "content": {"user_ids": ["@alice:example.com"]}
                           }
@@ -259,24 +258,20 @@ paths:
                   "invite": {
                     "!696r7674:example.com": {
                       "invite_state": {
-                        "m.room.name": {
-                          "": {
+                        "events": [
+                          {
                             "sender": "@alice:example.com",
                             "type": "m.room.name",
                             "state_key": "",
-                            "content": {"name": "My Room Name"},
-                            "event_id": "$asdkgjrsfg2314375:example.com"
-                          }
-                        },
-                        "m.room.member": {
-                          "@bob:example.com": {
+                            "content": {"name": "My Room Name"}
+                          },
+                          {
                             "sender": "@alice:example.com",
                             "type": "m.room.member",
                             "state_key": "@bob:example.com",
-                            "content": {"membership": "invite"},
-                            "event_id": "$257kasjdg315324akhg:example.com"
+                            "content": {"membership": "invite"}
                           }
-                        }
+                        ]
                       }
                     }
                   },

From 5aeaa42a5039bd0435f66be0a9787cb3efc24985 Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 18 Nov 2015 18:41:25 -0500
Subject: [PATCH 465/467] Changelog for 0.3.0

---
 CHANGELOG.rst | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 6e3198ef..4916c771 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -6,6 +6,35 @@
 .. in Jenkins. Comments like this are ignored by both RST and the templating
 .. system. Add the newest release notes beneath this comment.
 
+Specification changes in v0.3.0 (2015-11-18)
+============================================
+
+Many sections have been filled in, or added to reflect long-standing status
+quos. Several API endpoints have also been converted to Swagger. Several
+presentation improvements (e.g. syntax highlighting) have also been added.
+
+Additions:
+ - Extensive narrative on instant messaging
+ - Documentation of event sending
+ - Documentation of state event retrieval
+ - Redaction algorithm
+ - Algorithm for calculating display names for rooms and users
+ - New endpoint: ``/publicRooms``
+ - New event: ``m.room.avatar``
+ - invite_room_state on ``m.room.member`` events
+ - New section: Third party invitations
+ - Ability to reject invitations
+ - New API: Search
+ - Guest access to rooms
+ - Descriptions of server-server API endpoints for joining rooms
+ - Sample expected crypto output
+ - Several security warnings
+
+ Modifications:
+  - Fixed errors in parameters of /login
+  - Significant clarification and improvements to description of pagination
+  - Changes to v2_alpha/sync API (Note: this API is still not stable)
+
 Specification changes in v0.2.0 (2015-10-02)
 ============================================
 
@@ -38,4 +67,4 @@ Specification changes in v0.1.0 (2015-06-01)
 ============================================
 - First numbered release.
 - Restructure the format of Event information. Add more information.
-- Restructure the format of the Client-Server HTTP APIs.
\ No newline at end of file
+- Restructure the format of the Client-Server HTTP APIs.

From 1a1a7d87dcd1f8ada30592cae7b4da86b8b2468f Mon Sep 17 00:00:00 2001
From: Daniel Wagner-Hall <daniel@matrix.org>
Date: Wed, 18 Nov 2015 18:53:20 -0500
Subject: [PATCH 466/467] Revert "Changelog for 0.3.0"

This reverts commit 5aeaa42a5039bd0435f66be0a9787cb3efc24985.
---
 CHANGELOG.rst | 31 +------------------------------
 1 file changed, 1 insertion(+), 30 deletions(-)

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 4916c771..6e3198ef 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -6,35 +6,6 @@
 .. in Jenkins. Comments like this are ignored by both RST and the templating
 .. system. Add the newest release notes beneath this comment.
 
-Specification changes in v0.3.0 (2015-11-18)
-============================================
-
-Many sections have been filled in, or added to reflect long-standing status
-quos. Several API endpoints have also been converted to Swagger. Several
-presentation improvements (e.g. syntax highlighting) have also been added.
-
-Additions:
- - Extensive narrative on instant messaging
- - Documentation of event sending
- - Documentation of state event retrieval
- - Redaction algorithm
- - Algorithm for calculating display names for rooms and users
- - New endpoint: ``/publicRooms``
- - New event: ``m.room.avatar``
- - invite_room_state on ``m.room.member`` events
- - New section: Third party invitations
- - Ability to reject invitations
- - New API: Search
- - Guest access to rooms
- - Descriptions of server-server API endpoints for joining rooms
- - Sample expected crypto output
- - Several security warnings
-
- Modifications:
-  - Fixed errors in parameters of /login
-  - Significant clarification and improvements to description of pagination
-  - Changes to v2_alpha/sync API (Note: this API is still not stable)
-
 Specification changes in v0.2.0 (2015-10-02)
 ============================================
 
@@ -67,4 +38,4 @@ Specification changes in v0.1.0 (2015-06-01)
 ============================================
 - First numbered release.
 - Restructure the format of Event information. Add more information.
-- Restructure the format of the Client-Server HTTP APIs.
+- Restructure the format of the Client-Server HTTP APIs.
\ No newline at end of file

From 181d3f976d4f236c34ba0e86f58f729690d82e34 Mon Sep 17 00:00:00 2001
From: Richard van der Hoff <richard@matrix.org>
Date: Mon, 16 Nov 2015 13:01:20 +0000
Subject: [PATCH 467/467] Initial proposal for websockets support.

---
 drafts/websockets.rst | 285 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 285 insertions(+)
 create mode 100644 drafts/websockets.rst

diff --git a/drafts/websockets.rst b/drafts/websockets.rst
new file mode 100644
index 00000000..bd0ff081
--- /dev/null
+++ b/drafts/websockets.rst
@@ -0,0 +1,285 @@
+WebSockets API
+==============
+
+Introduction
+------------
+This document is a proposal for a WebSockets-based client-server API. It is not
+intended to replace the REST API, but rather to complement it and provide an
+alternative interface for certain operations.
+
+The primary goal is to offer a more efficient interface than the REST API: by
+using a bidirectional protocol such as WebSockets we can avoid the overheads
+involved in long-polling (SSL negotiation, HTTP headers, etc). In doing so we
+will reduce the latency between server and client by allowing the server to
+send events as soon as they arrive, rather than having to wait for a poll from
+the client.
+
+Handshake
+---------
+1. Instead of calling ``/sync``, the client makes a websocket request to
+   ``/_matrix/client/rN/stream``, passing the query parameters ``access_token``
+   and ``since``, and optionally ``filter`` - all of which have the same
+   meaning as for ``/sync``.
+
+   * The client sets the ``Sec-WebSocket-Protocol`` to ``m.json``. (Servers may
+     offer alternative encodings; at present only the JSON encoding is
+     specified but in future we will specify alternative encodings.)
+
+#. The server returns the websocket handshake; the socket is then connected.
+
+If the server does not return a valid websocket handshake, this indicates that
+the server or an intermediate proxy does not support WebSockets. In this case,
+the client should fall back to polling the ``/sync`` REST endpoint.
+
+Example
+~~~~~~~
+
+Client request:
+
+.. code:: http
+
+    GET /_matrix/client/v2_alpha/stream?access_token=123456&since=s72594_4483_1934 HTTP/1.1
+    Host: matrix.org
+    Upgrade: websocket
+    Connection: Upgrade
+    Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
+    Sec-WebSocket-Protocol: m.json
+    Sec-WebSocket-Version: 13
+    Origin: https://matrix.org
+
+Server response:
+
+.. code:: http
+
+    HTTP/1.1 101 Switching Protocols
+    Upgrade: websocket
+    Connection: Upgrade
+    Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
+    Sec-WebSocket-Protocol: m.json
+
+
+Update Notifications
+--------------------
+Once the socket is connected, the server begins streaming updates over the
+websocket. The server sends Update notifications about new messages or state
+changes. To make it easy for clients to parse, Update notifications have the
+same structure as the response to ``/sync``: an object with the following
+members:
+
+============= ========== ===================================================
+Key           Type       Description
+============= ========== ===================================================
+next_batch    string     The batch token to supply in the ``since`` param of
+                         the next /sync request. This is not required for
+                         streaming of events over the WebSocket, but is
+                         provided so that clients can reconnect if the
+                         socket is disconnected.
+presence      Presence   The updates to the presence status of other users.
+rooms         Rooms      Updates to rooms.
+============= ========== ===================================================
+
+Example
+~~~~~~~
+Message from the server:
+
+.. code:: json
+
+    {
+        "next_batch": "s72595_4483_1934",
+        "presence": {
+            "events": []
+        },
+        "rooms": {
+            "join": {},
+            "invite": {},
+            "leave": {}
+        }
+    }
+
+
+Client-initiated operations
+---------------------------
+
+The client can perform certain operations by sending a websocket message to
+the server. Such a "Request" message should be a JSON-encoded object with
+the following members:
+
+============= ========== ===================================================
+Key           Type       Description
+============= ========== ===================================================
+id            string     A unique identifier for this request
+method        string     Specifies the name of the operation to be
+                         performed; see below for available operations
+param         object     The parameters for the requested operation.
+============= ========== ===================================================
+
+The server responds to a client Request with a Response message. This is a
+JSON-encoded object with the following members:
+
+============= ========== ===================================================
+Key           Type       Description
+============= ========== ===================================================
+id            string     The same as the value in the corresponding Request
+                         object. The presence of the ``id`` field
+                         distinguishes a Response message from an Update 
+                         notification.
+result        object     On success, the results of the request.
+error         object     On error, an object giving the resons for the
+                         error. This has the same structure as the "standard
+                         error response" for the Matrix API: an object with
+                         the fields ``errcode`` and ``error``.
+============= ========== ===================================================
+
+Request methods
+~~~~~~~~~~~~~~~
+It is not intended that all operations which are available via the REST API
+will be available via the WebSockets API, but a few simple, common operations
+will be exposed. The initial operations will be as follows.
+
+``ping``
+^^^^^^^^
+This is a no-op which clients may use to keep their connection alive.
+
+The request ``params`` and the response ``result`` should be empty.
+
+``send``
+^^^^^^^^
+Send a message event to a room. The parameters are as follows:
+
+============= ========== ===================================================
+Parameter     Type       Description
+============= ========== ===================================================
+room_id       string     **Required.** The room to send the event to
+event_type    string     **Required.** The type of event to send.
+content       object     **Required.** The content of the event.
+============= ========== ===================================================
+
+The result is as follows:
+
+============= ========== ===================================================
+Key           Type       Description
+============= ========== ===================================================
+event_id      string     A unique identifier for the event.
+============= ========== ===================================================
+
+The ``id`` from the Request message is used as the transaction ID by the
+server.
+
+``state``
+^^^^^^^^^
+Update the state on a room.
+
+============= ========== ===================================================
+Parameter     Type       Description
+============= ========== ===================================================
+room_id       string     **Required.** The room to set the state in
+event_type    string     **Required.** The type of event to send.
+state_key     string     **Required.** The state_key for the state to send.
+content       object     **Required.** The content of the event.
+============= ========== ===================================================
+
+The result is as follows:
+
+============= ========== ===================================================
+Key           Type       Description
+============= ========== ===================================================
+event_id      string     A unique identifier for the event.
+============= ========== ===================================================
+
+
+Example
+~~~~~~~
+Client request:
+
+.. code:: json
+
+    {
+        "id": "12345",
+        "method": "send",
+        "params": {
+            "room_id": "!d41d8cd:matrix.org",
+            "event_type": "m.room.message",
+            "content": {
+                "msgtype": "m.text",
+                "body": "hello"
+            }
+        }
+    }
+
+Server response:
+
+.. code:: json
+
+    {
+        "id": "12345",
+        "result": {
+            "event_id": "$66697273743031:matrix.org"
+        }
+    }
+
+Alternative server response, in case of error:
+
+.. code:: json
+
+    {
+        "id": "12345",
+        "error": {
+           "errcode": "M_MISSING_PARAM",
+           "error": "Missing parameter: event_type"
+        }
+    }
+
+
+Rationale
+---------
+Alternatives to WebSockets include HTTP/2, CoAP, and simply rolling our own
+protocol over raw TCP sockets. However, the need to implement browser-based
+clients essentially reduces our choice to WebSockets. HTTP/2 streams will
+probably provide an interesting alternative in the future, but current browsers
+do not appear to give javascript applications low-level access to the protocol.
+
+Concerning the continued use of the JSON encoding: we prefer to focus on the
+transition to WebSockets initially. Replacing JSON with a compact
+representation such as CBOR, MessagePack, or even just compressed JSON will be
+a likely extension for the future. The support for negotiation of subprotocols
+within WebSockets should make this a simple transition once time permits.
+
+The number of methods available for client requests is deliberately limited, as
+each method requires code to be written to map it onto the equivalent REST
+implementation. Some REST methods - for instance, user registration and login -
+would be pointless to expose via WebSockets. It is likely, however, that we
+will increate the number of methods available via the WebSockets API as it
+becomes clear which would be most useful.
+
+Open questions
+--------------
+
+Throttling
+~~~~~~~~~~
+At least in v2 sync, clients are inherently self-throttling - if they do not
+poll quickly enough, events will be dropped from the next result. This proposal
+raises the possibility that events will be produced more quickly than they can
+be sent to the client; backlogs will build up on the server and/or in the
+intermediate network, which will not only lead to high latency on events being
+delivered, but will lead to responses to client requests also being delayed.
+
+We may need to implement some sort of throttling mechanism by which the server
+can start to drop events. The difficulty is in knowing when to start dropping
+events. A few ideas:
+
+* Use websocket pings to measure the RTT; if it starts to increase, start
+  dropping events. But this requires knowledge of the base RTT, and a useful
+  model of what constitutes an excessive increase.
+
+* Have the client acknowledge each batch of events, and use a window to ensure
+  the number of outstanding batches is limited. This is annoying as it requires
+  the client to have to acknowledge batches - and it's not clear what the right
+  window size is: we want a big window for long fat networks (think of mobile
+  clients), but a small one for one with lower latency.
+
+* Start dropping events if the server's TCP buffer starts filling up. This has
+  the advantage of delegating the congestion-detection to TCP (which already
+  has a number of algorithms to deal with it, to greater or lesser
+  effectiveness), but relies on homeservers being hosted on OSes which use
+  sensible TCP congestion-avoidance algorithms, and more critically, an ability
+  to read the fill level of the TCP send buffer.