Merge pull request #718 from matrix-org/rav/move_signing_json

Move 'Signing JSON' to appendices
pull/722/head
Richard van der Hoff 8 years ago committed by GitHub
commit 22ae6528c7

@ -15,287 +15,5 @@
Appendices
==========
Security Threat Model
----------------------
Denial of Service
~~~~~~~~~~~~~~~~~
The attacker could attempt to prevent delivery of messages to or from the
victim in order to:
* Disrupt service or marketing campaign of a commercial competitor.
* Censor a discussion or censor a participant in a discussion.
* Perform general vandalism.
Threat: Resource Exhaustion
+++++++++++++++++++++++++++
An attacker could cause the victims server to exhaust a particular resource
(e.g. open TCP connections, CPU, memory, disk storage)
Threat: Unrecoverable Consistency Violations
++++++++++++++++++++++++++++++++++++++++++++
An attacker could send messages which created an unrecoverable "split-brain"
state in the cluster such that the victim's servers could no longer derive a
consistent view of the chatroom state.
Threat: Bad History
+++++++++++++++++++
An attacker could convince the victim to accept invalid messages which the
victim would then include in their view of the chatroom history. Other servers
in the chatroom would reject the invalid messages and potentially reject the
victims messages as well since they depended on the invalid messages.
.. TODO-spec
Track trustworthiness of HS or users based on if they try to pretend they
haven't seen recent events, and fake a splitbrain... --M
Threat: Block Network Traffic
+++++++++++++++++++++++++++++
An attacker could try to firewall traffic between the victim's server and some
or all of the other servers in the chatroom.
Threat: High Volume of Messages
+++++++++++++++++++++++++++++++
An attacker could send large volumes of messages to a chatroom with the victim
making the chatroom unusable.
Threat: Banning users without necessary authorisation
+++++++++++++++++++++++++++++++++++++++++++++++++++++
An attacker could attempt to ban a user from a chatroom with the necessary
authorisation.
Spoofing
~~~~~~~~
An attacker could try to send a message claiming to be from the victim without
the victim having sent the message in order to:
* Impersonate the victim while performing illicit activity.
* Obtain privileges of the victim.
Threat: Altering Message Contents
+++++++++++++++++++++++++++++++++
An attacker could try to alter the contents of an existing message from the
victim.
Threat: Fake Message "origin" Field
+++++++++++++++++++++++++++++++++++
An attacker could try to send a new message purporting to be from the victim
with a phony "origin" field.
Spamming
~~~~~~~~
The attacker could try to send a high volume of solicited or unsolicited
messages to the victim in order to:
* Find victims for scams.
* Market unwanted products.
Threat: Unsolicited Messages
++++++++++++++++++++++++++++
An attacker could try to send messages to victims who do not wish to receive
them.
Threat: Abusive Messages
++++++++++++++++++++++++
An attacker could send abusive or threatening messages to the victim
Spying
~~~~~~
The attacker could try to access message contents or metadata for messages sent
by the victim or to the victim that were not intended to reach the attacker in
order to:
* Gain sensitive personal or commercial information.
* Impersonate the victim using credentials contained in the messages.
(e.g. password reset messages)
* Discover who the victim was talking to and when.
Threat: Disclosure during Transmission
++++++++++++++++++++++++++++++++++++++
An attacker could try to expose the message contents or metadata during
transmission between the servers.
Threat: Disclosure to Servers Outside Chatroom
++++++++++++++++++++++++++++++++++++++++++++++
An attacker could try to convince servers within a chatroom to send messages to
a server it controls that was not authorised to be within the chatroom.
Threat: Disclosure to Servers Within Chatroom
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An attacker could take control of a server within a chatroom to expose message
contents or metadata for messages in that room.
Cryptographic Test Vectors
--------------------------
To assist in the development of compatible implementations, the following test
values may be useful for verifying the cryptographic event signing code.
Signing Key
~~~~~~~~~~~
The following test vectors all use the 32-byte value given by the following
Base64-encoded string as the seed for generating the ``ed25519`` signing key:
.. code::
SIGNING_KEY_SEED = decode_base64(
"YJDBA9Xnr2sVqXD9Vj7XVUnmFZcZrlw8Md7kMW+3XA1"
)
In each case, the server name and key ID are as follows:
.. code::
SERVER_NAME = "domain"
KEY_ID = "ed25519:1"
JSON Signing
~~~~~~~~~~~~
Given an empty JSON object:
.. code:: json
{}
The JSON signing algorithm should emit the following signed data:
.. code:: json
{
"signatures": {
"domain": {
"ed25519:1": "K8280/U9SSy9IVtjBuVeLr+HpOB4BQFWbg+UZaADMtTdGYI7Geitb76LTrr5QV/7Xg4ahLwYGYZzuHGZKM5ZAQ"
}
}
}
Given the following JSON object with data values in it:
.. code:: json
{
"one": 1,
"two": "Two"
}
The JSON signing algorithm should emit the following signed JSON:
.. code:: json
{
"one": 1,
"signatures": {
"domain": {
"ed25519:1": "KqmLSbO39/Bzb0QIYE82zqLwsA+PDzYIpIRA2sRQ4sL53+sN6/fpNSoqE7BP7vBZhG6kYdD13EIMJpvhJI+6Bw"
}
},
"two": "Two"
}
Event Signing
~~~~~~~~~~~~~
Given the following minimally-sized event:
.. code:: json
{
"event_id": "$0:domain",
"origin": "domain",
"origin_server_ts": 1000000,
"signatures": {},
"type": "X",
"unsigned": {
"age_ts": 1000000
}
}
The event signing algorithm should emit the following signed event:
.. code:: json
{
"event_id": "$0:domain",
"hashes": {
"sha256": "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI"
},
"origin": "domain",
"origin_server_ts": 1000000,
"signatures": {
"domain": {
"ed25519:1": "2Wptgo4CwmLo/Y8B8qinxApKaCkBG2fjTWB7AbP5Uy+aIbygsSdLOFzvdDjww8zUVKCmI02eP9xtyJxc/cLiBA"
}
},
"type": "X",
"unsigned": {
"age_ts": 1000000
}
}
Given the following event containing redactable content:
.. code:: json
{
"content": {
"body": "Here is the message content",
},
"event_id": "$0:domain",
"origin": "domain",
"origin_server_ts": 1000000,
"type": "m.room.message",
"room_id": "!r:domain",
"sender": "@u:domain",
"signatures": {},
"unsigned": {
"age_ts": 1000000
}
}
The event signing algorithm should emit the following signed event:
.. code:: json
{
"content": {
"body": "Here is the message content",
},
"event_id": "$0:domain",
"hashes": {
"sha256": "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"
},
"origin": "domain",
"origin_server_ts": 1000000,
"type": "m.room.message",
"room_id": "!r:domain",
"sender": "@u:domain",
"signatures": {
"domain": {
"ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA"
}
},
"unsigned": {
"age_ts": 1000000
}
}
.. contents:: Table of Contents
.. sectnum::

@ -0,0 +1,167 @@
.. Copyright 2016 OpenMarket Ltd
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
Signing JSON
------------
Various points in the Matrix specification require JSON objects to be
cryptographically signed. This requires us to encode the JSON as a binary
string. Unfortunately the same JSON can be encoded in different ways by
changing how much white space is used or by changing the order of keys within
objects.
Signing an object therefore requires it to be encoded as a sequence of bytes
using `Canonical JSON`_, computing the signature for that sequence and then
adding the signature to the original JSON object.
Canonical JSON
~~~~~~~~~~~~~~
We define the canonical JSON encoding for a value to be the shortest UTF-8 JSON
encoding with dictionary keys lexicographically sorted by unicode codepoint.
Numbers in the JSON must be integers in the range ``[-(2**53)+1, (2**53)-1]``.
We pick UTF-8 as the encoding as it should be available to all platforms and
JSON received from the network is likely to be already encoded using UTF-8.
We sort the keys to give a consistent ordering. We force integers to be in the
range where they can be accurately represented using IEEE double precision
floating point numbers since a number of JSON libraries represent all numbers
using this representation.
.. code:: python
import json
def canonical_json(value):
return json.dumps(
value,
# Encode code-points outside of ASCII as UTF-8 rather than \u escapes
ensure_ascii=False,
# Remove unnecessary white space.
separators=(',',':'),
# Sort the keys of dictionaries.
sort_keys=True,
# Encode the resulting unicode as UTF-8 bytes.
).encode("UTF-8")
Grammar
+++++++
Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
insignificant whitespace, fractions, exponents and redundant character escapes
.. code::
value = false / null / true / object / array / number / string
false = %x66.61.6c.73.65
null = %x6e.75.6c.6c
true = %x74.72.75.65
object = %x7B [ member *( %x2C member ) ] %7D
member = string %x3A value
array = %x5B [ value *( %x2C value ) ] %5B
number = [ %x2D ] int
int = %x30 / ( %x31-39 *digit )
digit = %x30-39
string = %x22 *char %x22
char = unescaped / %x5C escaped
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
escaped = %x22 ; " quotation mark U+0022
/ %x5C ; \ reverse solidus U+005C
/ %x62 ; b backspace U+0008
/ %x66 ; f form feed U+000C
/ %x6E ; n line feed U+000A
/ %x72 ; r carriage return U+000D
/ %x74 ; t tab U+0009
/ %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X
/ %x75.30.30.31 (%x30-39 / %x61-66) ; u001X
Signing Details
~~~~~~~~~~~~~~~
JSON is signed by encoding the JSON object without ``signatures`` or keys grouped
as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the
signature algorithm and the signature is encoded using base64 with the padding
stripped. The resulting base64 signature is added to an object under the
*signing key identifier* which is added to the ``signatures`` object under the
name of the entity signing it which is added back to the original JSON object
along with the ``unsigned`` object.
The *signing key identifier* is the concatenation of the *signing algorithm*
and a *key identifier*. The *signing algorithm* identifies the algorithm used
to sign the JSON. The currently supported value for *signing algorithm* is
``ed25519`` as implemented by NACL (http://nacl.cr.yp.to/). The *key identifier*
is used to distinguish between different signing keys used by the same entity.
The ``unsigned`` object and the ``signatures`` object are not covered by the
signature. Therefore intermediate entities can add unsigned data such as
timestamps and additional signatures.
.. code:: json
{
"name": "example.org",
"signing_keys": {
"ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ"
},
"unsigned": {
"age_ts": 922834800000
},
"signatures": {
"example.org": {
"ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw"
}
}
}
.. code:: python
def sign_json(json_object, signing_key, signing_name):
signatures = json_object.pop("signatures", {})
unsigned = json_object.pop("unsigned", None)
signed = signing_key.sign(encode_canonical_json(json_object))
signature_base64 = encode_base64(signed.signature)
key_id = "%s:%s" % (signing_key.alg, signing_key.version)
signatures.setdefault(signing_name, {})[key_id] = signature_base64
json_object["signatures"] = signatures
if unsigned is not None:
json_object["unsigned"] = unsigned
return json_object
Checking for a Signature
~~~~~~~~~~~~~~~~~~~~~~~~
To check if an entity has signed a JSON object an implementation does the
following:
1. Checks if the ``signatures`` member of the object contains an entry with
the name of the entity. If the entry is missing then the check fails.
2. Removes any *signing key identifiers* from the entry with algorithms it
doesn't understand. If there are no *signing key identifiers* left then the
check fails.
3. Looks up *verification keys* for the remaining *signing key identifiers*
either from a local cache or by consulting a trusted key server. If it
cannot find a *verification key* then the check fails.
4. Decodes the base64 encoded signature bytes. If base64 decoding fails then
the check fails.
5. Removes the ``signatures`` and ``unsigned`` members of the object.
6. Encodes the remainder of the JSON object using the `Canonical JSON`_
encoding.
7. Checks the signature bytes against the encoded object using the
*verification key*. If this fails then the check fails. Otherwise the check
succeeds.

@ -0,0 +1,171 @@
.. Copyright 2015 OpenMarket Ltd
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
Cryptographic Test Vectors
--------------------------
To assist in the development of compatible implementations, the following test
values may be useful for verifying the cryptographic event signing code.
Signing Key
~~~~~~~~~~~
The following test vectors all use the 32-byte value given by the following
Base64-encoded string as the seed for generating the ``ed25519`` signing key:
.. code::
SIGNING_KEY_SEED = decode_base64(
"YJDBA9Xnr2sVqXD9Vj7XVUnmFZcZrlw8Md7kMW+3XA1"
)
In each case, the server name and key ID are as follows:
.. code::
SERVER_NAME = "domain"
KEY_ID = "ed25519:1"
JSON Signing
~~~~~~~~~~~~
Given an empty JSON object:
.. code:: json
{}
The JSON signing algorithm should emit the following signed data:
.. code:: json
{
"signatures": {
"domain": {
"ed25519:1": "K8280/U9SSy9IVtjBuVeLr+HpOB4BQFWbg+UZaADMtTdGYI7Geitb76LTrr5QV/7Xg4ahLwYGYZzuHGZKM5ZAQ"
}
}
}
Given the following JSON object with data values in it:
.. code:: json
{
"one": 1,
"two": "Two"
}
The JSON signing algorithm should emit the following signed JSON:
.. code:: json
{
"one": 1,
"signatures": {
"domain": {
"ed25519:1": "KqmLSbO39/Bzb0QIYE82zqLwsA+PDzYIpIRA2sRQ4sL53+sN6/fpNSoqE7BP7vBZhG6kYdD13EIMJpvhJI+6Bw"
}
},
"two": "Two"
}
Event Signing
~~~~~~~~~~~~~
Given the following minimally-sized event:
.. code:: json
{
"event_id": "$0:domain",
"origin": "domain",
"origin_server_ts": 1000000,
"signatures": {},
"type": "X",
"unsigned": {
"age_ts": 1000000
}
}
The event signing algorithm should emit the following signed event:
.. code:: json
{
"event_id": "$0:domain",
"hashes": {
"sha256": "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI"
},
"origin": "domain",
"origin_server_ts": 1000000,
"signatures": {
"domain": {
"ed25519:1": "2Wptgo4CwmLo/Y8B8qinxApKaCkBG2fjTWB7AbP5Uy+aIbygsSdLOFzvdDjww8zUVKCmI02eP9xtyJxc/cLiBA"
}
},
"type": "X",
"unsigned": {
"age_ts": 1000000
}
}
Given the following event containing redactable content:
.. code:: json
{
"content": {
"body": "Here is the message content",
},
"event_id": "$0:domain",
"origin": "domain",
"origin_server_ts": 1000000,
"type": "m.room.message",
"room_id": "!r:domain",
"sender": "@u:domain",
"signatures": {},
"unsigned": {
"age_ts": 1000000
}
}
The event signing algorithm should emit the following signed event:
.. code:: json
{
"content": {
"body": "Here is the message content",
},
"event_id": "$0:domain",
"hashes": {
"sha256": "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"
},
"origin": "domain",
"origin_server_ts": 1000000,
"type": "m.room.message",
"room_id": "!r:domain",
"sender": "@u:domain",
"signatures": {
"domain": {
"ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA"
}
},
"unsigned": {
"age_ts": 1000000
}
}

@ -0,0 +1,140 @@
.. Copyright 2015 OpenMarket Ltd
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
Security Threat Model
----------------------
Denial of Service
~~~~~~~~~~~~~~~~~
The attacker could attempt to prevent delivery of messages to or from the
victim in order to:
* Disrupt service or marketing campaign of a commercial competitor.
* Censor a discussion or censor a participant in a discussion.
* Perform general vandalism.
Threat: Resource Exhaustion
+++++++++++++++++++++++++++
An attacker could cause the victims server to exhaust a particular resource
(e.g. open TCP connections, CPU, memory, disk storage)
Threat: Unrecoverable Consistency Violations
++++++++++++++++++++++++++++++++++++++++++++
An attacker could send messages which created an unrecoverable "split-brain"
state in the cluster such that the victim's servers could no longer derive a
consistent view of the chatroom state.
Threat: Bad History
+++++++++++++++++++
An attacker could convince the victim to accept invalid messages which the
victim would then include in their view of the chatroom history. Other servers
in the chatroom would reject the invalid messages and potentially reject the
victims messages as well since they depended on the invalid messages.
.. TODO-spec
Track trustworthiness of HS or users based on if they try to pretend they
haven't seen recent events, and fake a splitbrain... --M
Threat: Block Network Traffic
+++++++++++++++++++++++++++++
An attacker could try to firewall traffic between the victim's server and some
or all of the other servers in the chatroom.
Threat: High Volume of Messages
+++++++++++++++++++++++++++++++
An attacker could send large volumes of messages to a chatroom with the victim
making the chatroom unusable.
Threat: Banning users without necessary authorisation
+++++++++++++++++++++++++++++++++++++++++++++++++++++
An attacker could attempt to ban a user from a chatroom with the necessary
authorisation.
Spoofing
~~~~~~~~
An attacker could try to send a message claiming to be from the victim without
the victim having sent the message in order to:
* Impersonate the victim while performing illicit activity.
* Obtain privileges of the victim.
Threat: Altering Message Contents
+++++++++++++++++++++++++++++++++
An attacker could try to alter the contents of an existing message from the
victim.
Threat: Fake Message "origin" Field
+++++++++++++++++++++++++++++++++++
An attacker could try to send a new message purporting to be from the victim
with a phony "origin" field.
Spamming
~~~~~~~~
The attacker could try to send a high volume of solicited or unsolicited
messages to the victim in order to:
* Find victims for scams.
* Market unwanted products.
Threat: Unsolicited Messages
++++++++++++++++++++++++++++
An attacker could try to send messages to victims who do not wish to receive
them.
Threat: Abusive Messages
++++++++++++++++++++++++
An attacker could send abusive or threatening messages to the victim
Spying
~~~~~~
The attacker could try to access message contents or metadata for messages sent
by the victim or to the victim that were not intended to reach the attacker in
order to:
* Gain sensitive personal or commercial information.
* Impersonate the victim using credentials contained in the messages.
(e.g. password reset messages)
* Discover who the victim was talking to and when.
Threat: Disclosure during Transmission
++++++++++++++++++++++++++++++++++++++
An attacker could try to expose the message contents or metadata during
transmission between the servers.
Threat: Disclosure to Servers Outside Chatroom
++++++++++++++++++++++++++++++++++++++++++++++
An attacker could try to convince servers within a chatroom to send messages to
a server it controls that was not authorised to be within the chatroom.
Threat: Disclosure to Servers Within Chatroom
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An attacker could take control of a server within a chatroom to expose message
contents or metadata for messages in that room.

@ -1,306 +0,0 @@
.. Copyright 2016 OpenMarket Ltd
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
Signing Events
--------------
Canonical JSON
~~~~~~~~~~~~~~
Matrix events are represented using JSON objects. If we want to sign JSON
events we need to encode the JSON as a binary string. Unfortunately the same
JSON can be encoded in different ways by changing how much white space is used
or by changing the order of keys within objects. Therefore we have to define an
encoding which can be reproduced byte for byte by any JSON library.
We define the canonical JSON encoding for a value to be the shortest UTF-8 JSON
encoding with dictionary keys lexicographically sorted by unicode codepoint.
Numbers in the JSON must be integers in the range [-(2**53)+1, (2**53)-1].
We pick UTF-8 as the encoding as it should be available to all platforms and
JSON received from the network is likely to be already encoded using UTF-8.
We sort the keys to give a consistent ordering. We force integers to be in the
range where they can be accurately represented using IEEE double precision
floating point numbers since a number of JSON libraries represent all numbers
using this representation.
.. code:: python
import json
def canonical_json(value):
return json.dumps(
value,
# Encode code-points outside of ASCII as UTF-8 rather than \u escapes
ensure_ascii=False,
# Remove unnecessary white space.
separators=(',',':'),
# Sort the keys of dictionaries.
sort_keys=True,
# Encode the resulting unicode as UTF-8 bytes.
).encode("UTF-8")
Grammar
+++++++
Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
insignificant whitespace, fractions, exponents and redundant character escapes
.. code::
value = false / null / true / object / array / number / string
false = %x66.61.6c.73.65
null = %x6e.75.6c.6c
true = %x74.72.75.65
object = %x7B [ member *( %x2C member ) ] %7D
member = string %x3A value
array = %x5B [ value *( %x2C value ) ] %5B
number = [ %x2D ] int
int = %x30 / ( %x31-39 *digit )
digit = %x30-39
string = %x22 *char %x22
char = unescaped / %x5C escaped
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
escaped = %x22 ; " quotation mark U+0022
/ %x5C ; \ reverse solidus U+005C
/ %x62 ; b backspace U+0008
/ %x66 ; f form feed U+000C
/ %x6E ; n line feed U+000A
/ %x72 ; r carriage return U+000D
/ %x74 ; t tab U+0009
/ %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X
/ %x75.30.30.31 (%x30-39 / %x61-66) ; u001X
Signing JSON
~~~~~~~~~~~~
We can now sign a JSON object by encoding it as a sequence of bytes, computing
the signature for that sequence and then adding the signature to the original
JSON object.
Signing Details
+++++++++++++++
JSON is signed by encoding the JSON object without ``signatures`` or keys grouped
as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the
signature algorithm and the signature is encoded using base64 with the padding
stripped. The resulting base64 signature is added to an object under the
*signing key identifier* which is added to the ``signatures`` object under the
name of the server signing it which is added back to the original JSON object
along with the ``unsigned`` object.
The *signing key identifier* is the concatenation of the *signing algorithm*
and a *key version*. The *signing algorithm* identifies the algorithm used to
sign the JSON. The currently support value for *signing algorithm* is
``ed25519`` as implemented by NACL (http://nacl.cr.yp.to/). The *key version*
is used to distinguish between different signing keys used by the same entity.
The ``unsigned`` object and the ``signatures`` object are not covered by the
signature. Therefore intermediate servers can add unsigned data such as timestamps
and additional signatures.
.. code:: json
{
"name": "example.org",
"signing_keys": {
"ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ"
},
"unsigned": {
"age_ts": 922834800000
},
"signatures": {
"example.org": {
"ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw"
}
}
}
.. code:: python
def sign_json(json_object, signing_key, signing_name):
signatures = json_object.pop("signatures", {})
unsigned = json_object.pop("unsigned", None)
signed = signing_key.sign(encode_canonical_json(json_object))
signature_base64 = encode_base64(signed.signature)
key_id = "%s:%s" % (signing_key.alg, signing_key.version)
signatures.setdefault(signing_name, {})[key_id] = signature_base64
json_object["signatures"] = signatures
if unsigned is not None:
json_object["unsigned"] = unsigned
return json_object
Checking for a Signature
++++++++++++++++++++++++
To check if an entity has signed a JSON object a server does the following
1. Checks if the ``signatures`` object contains an entry with the name of the
entity. If the entry is missing then the check fails.
2. Removes any *signing key identifiers* from the entry with algorithms it
doesn't understand. If there are no *signing key identifiers* left then the
check fails.
3. Looks up *verification keys* for the remaining *signing key identifiers*
either from a local cache or by consulting a trusted key server. If it
cannot find a *verification key* then the check fails.
4. Decodes the base64 encoded signature bytes. If base64 decoding fails then
the check fails.
5. Checks the signature bytes using the *verification key*. If this fails then
the check fails. Otherwise the check succeeds.
Signing Events
~~~~~~~~~~~~~~
Signing events is a more complicated process since servers can choose to redact
non-essential parts of an event. Before signing the event it is encoded as
Canonical JSON and hashed using SHA-256. The resulting hash is then stored
in the event JSON in a ``hash`` object under a ``sha256`` key.
.. code:: python
def hash_event(event_json_object):
# Keys under "unsigned" can be modified by other servers.
# They are useful for conveying information like the age of an
# event that will change in transit.
# Since they can be modifed we need to exclude them from the hash.
unsigned = event_json_object.pop("unsigned", None)
# Signatures will depend on the current value of the "hashes" key.
# We cannot add new hashes without invalidating existing signatures.
signatures = event_json_object.pop("signatures", None)
# The "hashes" key might contain multiple algorithms if we decide to
# migrate away from SHA-2. We don't want to include an existing hash
# output in our hash so we exclude the "hashes" dict from the hash.
hashes = event_json_object.pop("hashes", {})
# Encode the JSON using a canonical encoding so that we get the same
# bytes on every server for the same JSON object.
event_json_bytes = encode_canonical_json(event_json_bytes)
# Add the base64 encoded bytes of the hash to the "hashes" dict.
hashes["sha256"] = encode_base64(sha256(event_json_bytes).digest())
# Add the "hashes" dict back the event JSON under a "hashes" key.
event_json_object["hashes"] = hashes
if unsigned is not None:
event_json_object["unsigned"] = unsigned
return event_json_object
The event is then stripped of all non-essential keys both at the top level and
within the ``content`` object. Any top-level keys not in the following list
MUST be removed:
.. code::
auth_events
depth
event_id
hashes
membership
origin
origin_server_ts
prev_events
prev_state
room_id
sender
signatures
state_key
type
A new ``content`` object is constructed for the resulting event that contains
only the essential keys of the original ``content`` object. If the original
event lacked a ``content`` object at all, a new empty JSON object is created
for it.
The keys that are considered essential for the ``content`` object depend on the
the ``type`` of the event. These are:
.. code::
type is "m.room.aliases":
aliases
type is "m.room.create":
creator
type is "m.room.history_visibility":
history_visibility
type is "m.room.join_rules":
join_rule
type is "m.room.member":
membership
type is "m.room.power_levels":
ban
events
events_default
kick
redact
state_default
users
users_default
The resulting stripped object with the new ``content`` object and the original
``hashes`` key is then signed using the JSON signing algorithm outlined below:
.. code:: python
def sign_event(event_json_object, name, key):
# Make sure the event has a "hashes" key.
if "hashes" not in event_json_object:
event_json_object = hash_event(event_json_object)
# Strip all the keys that would be removed if the event was redacted.
# The hashes are not stripped and cover all the keys in the event.
# This means that we can tell if any of the non-essential keys are
# modified or removed.
stripped_json_object = strip_non_essential_keys(event_json_object)
# Sign the stripped JSON object. The signature only covers the
# essential keys and the hashes. This means that we can check the
# signature even if the event is redacted.
signed_json_object = sign_json(stripped_json_object)
# Copy the signatures from the stripped event to the original event.
event_json_object["signatures"] = signed_json_oject["signatures"]
return event_json_object
Servers can then transmit the entire event or the event with the non-essential
keys removed. If the entire event is present, receiving servers can then check
the event by computing the SHA-256 of the event, excluding the ``hash`` object.
If the keys have been redacted, then the ``hash`` object is included when
calculating the SHA-256 instead.
New hash functions can be introduced by adding additional keys to the ``hash``
object. Since the ``hash`` object cannot be redacted a server shouldn't allow
too many hashes to be listed, otherwise a server might embed illict data within
the ``hash`` object. For similar reasons a server shouldn't allow hash values
that are too long.
.. TODO
[[TODO(markjh): We might want to specify a maximum number of keys for the
``hash`` and we might want to specify the maximum output size of a hash]]
[[TODO(markjh) We might want to allow the server to omit the output of well
known hash functions like SHA-256 when none of the keys have been redacted]]

@ -998,3 +998,149 @@ the following EDU::
messages: The messages to send. A map from user ID, to a map from device ID
to message body. The device ID may also be *, meaning all known devices
for the user.
Signing Events
--------------
Signing events is complicated by the fact that servers can choose to redact
non-essential parts of an event.
Before signing the event, the ``unsigned`` and ``signature`` members are
removed, it is encoded as `Canonical JSON`_, and then hashed using SHA-256. The
resulting hash is then stored in the event JSON in a ``hash`` object under a
``sha256`` key.
.. code:: python
def hash_event(event_json_object):
# Keys under "unsigned" can be modified by other servers.
# They are useful for conveying information like the age of an
# event that will change in transit.
# Since they can be modifed we need to exclude them from the hash.
unsigned = event_json_object.pop("unsigned", None)
# Signatures will depend on the current value of the "hashes" key.
# We cannot add new hashes without invalidating existing signatures.
signatures = event_json_object.pop("signatures", None)
# The "hashes" key might contain multiple algorithms if we decide to
# migrate away from SHA-2. We don't want to include an existing hash
# output in our hash so we exclude the "hashes" dict from the hash.
hashes = event_json_object.pop("hashes", {})
# Encode the JSON using a canonical encoding so that we get the same
# bytes on every server for the same JSON object.
event_json_bytes = encode_canonical_json(event_json_bytes)
# Add the base64 encoded bytes of the hash to the "hashes" dict.
hashes["sha256"] = encode_base64(sha256(event_json_bytes).digest())
# Add the "hashes" dict back the event JSON under a "hashes" key.
event_json_object["hashes"] = hashes
if unsigned is not None:
event_json_object["unsigned"] = unsigned
return event_json_object
The event is then stripped of all non-essential keys both at the top level and
within the ``content`` object. Any top-level keys not in the following list
MUST be removed:
.. code::
auth_events
depth
event_id
hashes
membership
origin
origin_server_ts
prev_events
prev_state
room_id
sender
signatures
state_key
type
A new ``content`` object is constructed for the resulting event that contains
only the essential keys of the original ``content`` object. If the original
event lacked a ``content`` object at all, a new empty JSON object is created
for it.
The keys that are considered essential for the ``content`` object depend on the
the ``type`` of the event. These are:
.. code::
type is "m.room.aliases":
aliases
type is "m.room.create":
creator
type is "m.room.history_visibility":
history_visibility
type is "m.room.join_rules":
join_rule
type is "m.room.member":
membership
type is "m.room.power_levels":
ban
events
events_default
kick
redact
state_default
users
users_default
The resulting stripped object with the new ``content`` object and the original
``hashes`` key is then signed using the JSON signing algorithm outlined below:
.. code:: python
def sign_event(event_json_object, name, key):
# Make sure the event has a "hashes" key.
if "hashes" not in event_json_object:
event_json_object = hash_event(event_json_object)
# Strip all the keys that would be removed if the event was redacted.
# The hashes are not stripped and cover all the keys in the event.
# This means that we can tell if any of the non-essential keys are
# modified or removed.
stripped_json_object = strip_non_essential_keys(event_json_object)
# Sign the stripped JSON object. The signature only covers the
# essential keys and the hashes. This means that we can check the
# signature even if the event is redacted.
signed_json_object = sign_json(stripped_json_object)
# Copy the signatures from the stripped event to the original event.
event_json_object["signatures"] = signed_json_oject["signatures"]
return event_json_object
Servers can then transmit the entire event or the event with the non-essential
keys removed. If the entire event is present, receiving servers can then check
the event by computing the SHA-256 of the event, excluding the ``hash`` object.
If the keys have been redacted, then the ``hash`` object is included when
calculating the SHA-256 instead.
New hash functions can be introduced by adding additional keys to the ``hash``
object. Since the ``hash`` object cannot be redacted a server shouldn't allow
too many hashes to be listed, otherwise a server might embed illict data within
the ``hash`` object. For similar reasons a server shouldn't allow hash values
that are too long.
.. TODO
[[TODO(markjh): We might want to specify a maximum number of keys for the
``hash`` and we might want to specify the maximum output size of a hash]]
[[TODO(markjh) We might want to allow the server to omit the output of well
known hash functions like SHA-256 when none of the keys have been redacted]]
.. _`Canonical JSON`: ../appendices.html#canonical-json

@ -20,7 +20,6 @@ targets:
server_server:
files:
- server_server_api.rst
- { 1: event_signing.rst }
version_label: "%SERVER_RELEASE_LABEL%"
identity_service:
files:
@ -33,6 +32,9 @@ targets:
appendices:
files:
- appendices.rst
- appendices/signing_json.rst
- appendices/threat_model.rst
- appendices/test_vectors.rst
groups: # reusable blobs of files when prefixed with 'group:'
modules:
- modules/instant_messaging.rst

Loading…
Cancel
Save