Merge branch 'master' into rav/rework_objects

Conflicts:
	templating/matrix_templates/units.py
pull/977/head
Richard van der Hoff 9 years ago
commit 34ac544290

@ -0,0 +1,142 @@
Goals of Key-Distribution in Matrix
===================================
* No Central Authority: Users should not need to trust a central authority
when determining the authenticity of keys.
* Easy to Add New Devices: It should be easy for a user to start using a
new device.
* Possible to discover MITM: It should be possible for a user to determine if
they are being MITM.
* Lost Devices: It should be possible for a user to recover if they lose all
their devices.
* No Copying Keys: Keys should be per device and shouldn't leave the device
they were created on.
A Possible Mechanism for Key Distribution
=========================================
Basic API for setting up keys on a server:
https://github.com/matrix-org/matrix-doc/pull/24
Client shouldn't trust the keys unless they have been verified, e.g by
comparing fingerprints.
If a user adds a new device it should some yet to be specified protocol
communicate with an old device and obtain a cross-signature from the old
device for its public key.
The new device can then present the cross-signed key to all the devices
that the user is in conversations with. Those devices should then include
the new device into those conversations.
If the user cannot cross-sign the new key, e.g. because their old device
is lost or stolen. Then they will need to reauthenticate their conversations
out of band, e.g by comparing fingerprints.
Goals of End-to-end encryption in Matrix
========================================
* Access to Chat History: Users should be able to see the history of a
conversation on a new device. User should be able to control who can
see their chat history and how much of the chat history they can see.
* Forward Secrecy of Discarded Chat History: Users should be able to discard
history from their device, once they have discarded the history it should be
impossible for an adversary to recover that history.
* Forward Secrecy of Future Messages: Users should be able to recover from
disclosure of the chat history on their device.
* Deniablity of Chat History: It should not be possible to prove to a third
party that a given user sent a message.
* Authenticity of Chat History: It should be possible to prove amoungst
the members of a chat that a message sent by a user was authored by that
user.
Bonus Goals:
* Traffic Analysis: It would be nice if the protocol was resilient to traffic
or metadata analysis. However it's not something we want to persue if it
harms the usability of the protocol. It might be cool if there was a
way for the user to could specify the trade off between performance and
resilience to traffic analysis that they wanted.
A Possible Design for Group Chat using Olm
==========================================
Protecting the secrecy of history
---------------------------------
Each message sent by a client has a 32-bit counter. This counter increments
by one for each message sent by the client. This counter is used to advance a
ratchet. The ratchet is split into a vector four 256-bit values,
:math:`R_{n,j}` for :math:`j \in {0,1,2,3}`. The ratchet can be advanced as
follows:
.. math::
\begin{align}
R_{2^24n,0} &= H_1\left(R_{2^24(i-1),0}\right) \\
R_{2^24n,1} &= H_2\left(R_{2^24(i-1),0}\right) \\
R_{2^16n,1} &= H_1\left(R_{2^16(i-1),1}\right) \\
R_{2^16n,2} &= H_2\left(R_{2^16(i-1),1}\right) \\
R_{2^8i,2} &= H_1\left(R_{2^8(i-1),2}\right) \\
R_{2^8i,3} &= H_2\left(R_{2^8(i-1),2}\right) \\
R_{i,3} &= H_1\left(R_{(i-1),3}\right)
\end{align}
Where :math:`H_1` and :math:`H_2` are different hash functions. For example
:math:`H_1` could be :math:`HMAC\left(X,\text{"\textbackslash x01"}\right)` and
:math:`H_2` could be :math:`HMAC\left(X,\text{"\textbackslash x02"}\right)`.
So every :math:`2^24` iterations :math:`R_{n,1}` is reseeded from :math:`R_{n,0}`.
Every :math:`2^16` iterations :math:`R_{n,2}` is reseeded from :math:`R_{n,1}`.
Every :math:`2^8` iterations :math:`R_{n,3}` is reseeded from :math:`R_{n,2}`.
This scheme allows the ratchet to be advanced an arbitrary amount forwards
while needing only 1024 hash computations.
This the value of the ratchet is hashed to generate the keys used to encrypt
each mesage.
A client can decrypt chat history onwards from the earliest value of the
ratchet it is aware of. But cannot decrypt history from before that point
without reversing the hash function.
This allows a client to share its ability to decrypt chat history with another
from a point in the conversation onwards by giving a copy of the ratchet at
that point in the conversation.
A client can discard history by advancing a ratchet to beyond the last message
they want to discard and then forgetting all previous values of the ratchet.
Proving and denying the authenticity of history
-----------------------------------------------
Client sign the messages they send using a Ed25519 key generated per
conversation. That key, along with the ratchet key, is distributed
to other clients using 1:1 olm ratchets. Those 1:1 ratchets are started using
Triple Diffie-Hellman which provides authenticity of the messages to the
participants and deniability of the messages to third parties. Therefore
any keys shared over those keys inherit the same levels of deniability and
authenticity.
Protecting the secrecy of future messages
-----------------------------------------
A client would need to generate new keys if it wanted to prevent access to
messages beyond a given point in the conversation. It must generate new keys
whenever someone leaves the room. It should generate new keys periodically
anyway.
The frequency of key generation in a large room may need to be restricted to
keep the frequency of messages broadcast over the individual 1:1 channels
low.

@ -1,7 +1,7 @@
{
"type": "object",
"title": "The current membership state of a user in the room.",
"description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event also includes an ``invite_room_state`` key **outside the** ``content`` **key**. This contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
"description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event may also include an ``invite_room_state`` key **outside the** ``content`` **key**. If present, this contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
"allOf": [{
"$ref": "core-event-schema/state_event.json"
}],

@ -94,6 +94,27 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
return '\n\n'.join(output_lines)
def fieldwidths(input, keys, defaults=[], default_width=15):
"""
A template filter to help in the generation of tables.
Given a list of rows, returns a list giving the maximum length of the
values in each column.
:param list[dict[str, str]] input: a list of rows. Each row should be a
dict with the keys given in ``keys``.
:param list[str] keys: the keys corresponding to the table columns
:param list[int] defaults: for each column, the default column width.
:param int default_width: if ``defaults`` is shorter than ``keys``, this
will be used as a fallback
"""
def colwidth(key, default):
return reduce(max, (len(row[key]) for row in input),
default if default is not None else default_width)
results = map(colwidth, keys, defaults)
return results
# make Jinja aware of the templates and filters
env = Environment(
loader=FileSystemLoader(in_mod.exports["templates"]),
@ -103,6 +124,7 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
env.filters["indent"] = indent
env.filters["indent_block"] = indent_block
env.filters["wrap"] = wrap
env.filters["fieldwidths"] = fieldwidths
# load up and parse the lowest single units possible: we don't know or care
# which spec section will use it, we just need it there in memory for when

@ -1,17 +1,8 @@
{% import 'tables.tmpl' as tables -%}
{{common_event.title}} Fields
{{(7 + common_event.title | length) * title_kind}}
{{common_event.desc | wrap(80)}}
================== ================= ===========================================
Key Type Description
================== ================= ===========================================
{% for row in common_event.rows -%}
{# -#}
{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
{# It also needs to then wrap inside the desc col (43 ch width) -#}
{# -#}
{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc | indent(18 - (row.type|length)) |wrap(43) |indent_block(37)}}
{% endfor -%}
================== ================= ===========================================
{{ tables.paramtable(common_event.rows, ["Key", "Type", "Description"]) }}

@ -1,3 +1,5 @@
{% import 'tables.tmpl' as tables -%}
``{{event.type}}``
{{(4 + event.type | length) * title_kind}}
*{{event.typeof}}*
@ -7,18 +9,7 @@
{% for table in event.content_fields -%}
{{"``"+table.title+"``" if table.title else "" }}
======================= ================= ===========================================
{{table.title or "Content"}} Key Type Description
======================= ================= ===========================================
{% for row in table.rows -%}
{# -#}
{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
{# It also needs to then wrap inside the desc col (43 ch width) -#}
{# -#}
{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(42)}}
{% endfor -%}
======================= ================= ===========================================
{{ tables.paramtable(table.rows, [(table.title or "Content") ~ " Key", "Type", "Description"]) }}
{% endfor %}
Example:

@ -1,3 +1,5 @@
{% import 'tables.tmpl' as tables -%}
``{{endpoint.method}} {{endpoint.path}}``
{{(5 + (endpoint.path | length) + (endpoint.method | length)) * title_kind}}
{% if "alias_for_path" in endpoint -%}
@ -13,17 +15,7 @@
Request format:
{% if (endpoint.req_param_by_loc | length) %}
=========================================== ================= ===========================================
Parameter Value Description
=========================================== ================= ===========================================
{% for loc in endpoint.req_param_by_loc -%}
*{{loc}} parameters*
---------------------------------------------------------------------------------------------------------
{% for param in endpoint.req_param_by_loc[loc] -%}
{{param.key}}{{param.type|indent(44-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(62)}}
{% endfor -%}
{% endfor -%}
=========================================== ================= ===========================================
{{ tables.split_paramtable(endpoint.req_param_by_loc) }}
{% else %}
`No parameters`
{% endif %}
@ -34,18 +26,7 @@ Response format:
{% for table in endpoint.res_tables -%}
{{"``"+table.title+"``" if table.title else "" }}
======================= ========================= ==========================================
Param Type Description
======================= ========================= ==========================================
{% for row in table.rows -%}
{# -#}
{# Row type needs to prepend spaces to line up with the type column (20 ch) -#}
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
{# It also needs to then wrap inside the desc col (42 ch width) -#}
{# -#}
{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(26 - (row.type|length))) |indent_block(50)}}
{% endfor -%}
======================= ========================= ==========================================
{{ tables.paramtable(table.rows) }}
{% endfor %}
{% endif -%}

@ -1,21 +1,12 @@
{% import 'tables.tmpl' as tables -%}
``{{event.msgtype}}``
{{(4 + event.msgtype | length) * title_kind}}
{{event.desc | wrap(80)}}
{% for table in event.content_fields -%}
{{"``"+table.title+"``" if table.title else "" }}
================== ================= ===========================================
{{table.title or "Content"}} Key Type Description
================== ================= ===========================================
{% for row in table.rows -%}
{# -#}
{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
{# It also needs to then wrap inside the desc col (43 ch width) -#}
{# -#}
{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(37)}}
{% endfor -%}
================== ================= ===========================================
{{ tables.paramtable(table.rows, [(table.title or "Content") ~ " Key", "Type", "Description"]) }}
{% endfor %}
Example:

@ -0,0 +1,104 @@
{#
# A set of macros for generating RST tables
#}
{#
# write a table for a list of parameters.
#
# 'rows' is the list of parameters. Each row should have the keys
# 'key', 'type', and 'desc'.
#}
{% macro paramtable(rows, titles=["Parameter", "Type", "Description"]) -%}
{{ split_paramtable({None: rows}, titles) }}
{% endmacro %}
{#
# write a table for the request parameters, split by location.
# 'rows_by_loc' is a map from location to a list of parameters.
#
# As a special case, if a key of 'rows_by_loc' is 'None', no title row is
# written for that location. This is used by the standard 'paramtable' macro.
#}
{% macro split_paramtable(rows_by_loc,
titles=["Parameter", "Type", "Description"]) -%}
{% set rowkeys = ['key', 'type', 'desc'] %}
{% set titlerow = {'key': titles[0], 'type': titles[1], 'desc': titles[2]} %}
{# We need the rows flattened into a single list. Abuse the 'sum' filter to
# join arrays instead of add numbers. -#}
{% set flatrows = rows_by_loc.values()|sum(start=[]) -%}
{# Figure out the widths of the columns. The last column is always 50 characters
# wide; the others default to 10, but stretch if there is wider text in the
# column. -#}
{% set fieldwidths = (([titlerow] + flatrows) |
fieldwidths(rowkeys[0:-1], [10, 10])) + [50] -%}
{{ tableheader(fieldwidths) }}
{{ tablerow(fieldwidths, titlerow, rowkeys) }}
{{ tableheader(fieldwidths) }}
{% for loc in rows_by_loc -%}
{% if loc != None -%}
{{ tablespan(fieldwidths, "*" ~ loc ~ " parameters*") }}
{% endif -%}
{% for row in rows_by_loc[loc] -%}
{{ tablerow(fieldwidths, row, rowkeys) }}
{% endfor -%}
{% endfor -%}
{{ tableheader(fieldwidths) }}
{% endmacro %}
{#
# Write a table header row, for the given column widths
#}
{% macro tableheader(widths) -%}
{% for arg in widths -%}
{{"="*arg}} {% endfor -%}
{% endmacro %}
{#
# Write a normal table row. Each of 'widths' and 'keys' should be sequences
# of the same length; 'widths' defines the column widths, and 'keys' the
# attributes of 'row' to look up for values to put in the columns.
#}
{% macro tablerow(widths, row, keys) -%}
{% for key in keys -%}
{% set value=row[key] -%}
{% if not loop.last -%}
{# the first few columns need space after them -#}
{{ value }}{{" "*(1+widths[loop.index0]-value|length) -}}
{% else -%}
{# the last column needs wrapping and indenting (by the sum of the widths of
the preceding columns, plus the number of preceding columns (for the
separators)) -#}
{{ value | wrap(widths[loop.index0]) |
indent_block(widths[0:-1]|sum + loop.index0) -}}
{% endif -%}
{% endfor -%}
{% endmacro %}
{#
# write a tablespan row. This is a single value which spans the entire table.
#}
{% macro tablespan(widths, value) -%}
{{value}}
{# we write a trailing space to stop the separator being misinterpreted
# as a header line. -#}
{{"-"*(widths|sum + widths|length -1)}} {% endmacro %}

@ -171,6 +171,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
include_parents=include_parents,
)
value_type = nested_objects[0]["title"]
value_id = value_type
tables += [x for x in nested_objects if not x.get("no-table")]
elif prop_type == "array":
@ -181,12 +182,14 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
enforce_title=True,
include_parents=include_parents,
)
value_type = "[%s]" % nested_objects[0]["title"]
value_id = nested_objects[0]["title"]
value_type = "[%s]" % value_id
tables += nested_objects
else:
value_type = props[key_name]["items"]["type"]
if isinstance(value_type, list):
value_type = " or ".join(value_type)
value_id = value_type
value_type = "[%s]" % value_type
array_enums = props[key_name]["items"].get("enum")
if array_enums:
@ -201,6 +204,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
)
else:
value_type = prop_type
value_id = prop_type
if props[key_name].get("enum"):
if len(props[key_name].get("enum")) > 1:
value_type = "enum"
@ -221,6 +225,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
fields["rows"].append({
"key": key_name,
"type": value_type,
"id": value_id,
"required": required,
"desc": desc,
"req_str": "**Required.** " if required else ""
@ -365,10 +370,16 @@ class MatrixUnits(Units):
if req_tables > 1:
for table in req_tables[1:]:
nested_key_name = [
s["key"] for s in req_tables[0]["rows"] if
s["type"] == ("%s" % (table["title"],))
][0]
nested_key_name = {
"key": s["key"]
for rtable in req_tables
for s in rtable["rows"]
if s["id"] == table["title"]
}.get("key", None)
if nested_key_name is None:
raise Exception("Failed to find table for %r" % (table["title"],))
for row in table["rows"]:
row["key"] = "%s.%s" % (nested_key_name, row["key"])

Loading…
Cancel
Save