Merge pull request #718 from matrix-org/rav/move_signing_json
Move 'Signing JSON' to appendicespull/977/head
Richard van der Hoff
8 years ago
committed by
GitHub
commit
22ae6528c7
@ -0,0 +1,167 @@
|
|||||||
|
.. Copyright 2016 OpenMarket Ltd
|
||||||
|
..
|
||||||
|
.. Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
.. you may not use this file except in compliance with the License.
|
||||||
|
.. You may obtain a copy of the License at
|
||||||
|
..
|
||||||
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
..
|
||||||
|
.. Unless required by applicable law or agreed to in writing, software
|
||||||
|
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
.. See the License for the specific language governing permissions and
|
||||||
|
.. limitations under the License.
|
||||||
|
|
||||||
|
Signing JSON
|
||||||
|
------------
|
||||||
|
|
||||||
|
Various points in the Matrix specification require JSON objects to be
|
||||||
|
cryptographically signed. This requires us to encode the JSON as a binary
|
||||||
|
string. Unfortunately the same JSON can be encoded in different ways by
|
||||||
|
changing how much white space is used or by changing the order of keys within
|
||||||
|
objects.
|
||||||
|
|
||||||
|
Signing an object therefore requires it to be encoded as a sequence of bytes
|
||||||
|
using `Canonical JSON`_, computing the signature for that sequence and then
|
||||||
|
adding the signature to the original JSON object.
|
||||||
|
|
||||||
|
Canonical JSON
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
We define the canonical JSON encoding for a value to be the shortest UTF-8 JSON
|
||||||
|
encoding with dictionary keys lexicographically sorted by unicode codepoint.
|
||||||
|
Numbers in the JSON must be integers in the range ``[-(2**53)+1, (2**53)-1]``.
|
||||||
|
|
||||||
|
We pick UTF-8 as the encoding as it should be available to all platforms and
|
||||||
|
JSON received from the network is likely to be already encoded using UTF-8.
|
||||||
|
We sort the keys to give a consistent ordering. We force integers to be in the
|
||||||
|
range where they can be accurately represented using IEEE double precision
|
||||||
|
floating point numbers since a number of JSON libraries represent all numbers
|
||||||
|
using this representation.
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
def canonical_json(value):
|
||||||
|
return json.dumps(
|
||||||
|
value,
|
||||||
|
# Encode code-points outside of ASCII as UTF-8 rather than \u escapes
|
||||||
|
ensure_ascii=False,
|
||||||
|
# Remove unnecessary white space.
|
||||||
|
separators=(',',':'),
|
||||||
|
# Sort the keys of dictionaries.
|
||||||
|
sort_keys=True,
|
||||||
|
# Encode the resulting unicode as UTF-8 bytes.
|
||||||
|
).encode("UTF-8")
|
||||||
|
|
||||||
|
Grammar
|
||||||
|
+++++++
|
||||||
|
|
||||||
|
Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
|
||||||
|
insignificant whitespace, fractions, exponents and redundant character escapes
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
value = false / null / true / object / array / number / string
|
||||||
|
false = %x66.61.6c.73.65
|
||||||
|
null = %x6e.75.6c.6c
|
||||||
|
true = %x74.72.75.65
|
||||||
|
object = %x7B [ member *( %x2C member ) ] %7D
|
||||||
|
member = string %x3A value
|
||||||
|
array = %x5B [ value *( %x2C value ) ] %5B
|
||||||
|
number = [ %x2D ] int
|
||||||
|
int = %x30 / ( %x31-39 *digit )
|
||||||
|
digit = %x30-39
|
||||||
|
string = %x22 *char %x22
|
||||||
|
char = unescaped / %x5C escaped
|
||||||
|
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
|
||||||
|
escaped = %x22 ; " quotation mark U+0022
|
||||||
|
/ %x5C ; \ reverse solidus U+005C
|
||||||
|
/ %x62 ; b backspace U+0008
|
||||||
|
/ %x66 ; f form feed U+000C
|
||||||
|
/ %x6E ; n line feed U+000A
|
||||||
|
/ %x72 ; r carriage return U+000D
|
||||||
|
/ %x74 ; t tab U+0009
|
||||||
|
/ %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X
|
||||||
|
/ %x75.30.30.31 (%x30-39 / %x61-66) ; u001X
|
||||||
|
|
||||||
|
Signing Details
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
JSON is signed by encoding the JSON object without ``signatures`` or keys grouped
|
||||||
|
as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the
|
||||||
|
signature algorithm and the signature is encoded using base64 with the padding
|
||||||
|
stripped. The resulting base64 signature is added to an object under the
|
||||||
|
*signing key identifier* which is added to the ``signatures`` object under the
|
||||||
|
name of the entity signing it which is added back to the original JSON object
|
||||||
|
along with the ``unsigned`` object.
|
||||||
|
|
||||||
|
The *signing key identifier* is the concatenation of the *signing algorithm*
|
||||||
|
and a *key identifier*. The *signing algorithm* identifies the algorithm used
|
||||||
|
to sign the JSON. The currently supported value for *signing algorithm* is
|
||||||
|
``ed25519`` as implemented by NACL (http://nacl.cr.yp.to/). The *key identifier*
|
||||||
|
is used to distinguish between different signing keys used by the same entity.
|
||||||
|
|
||||||
|
The ``unsigned`` object and the ``signatures`` object are not covered by the
|
||||||
|
signature. Therefore intermediate entities can add unsigned data such as
|
||||||
|
timestamps and additional signatures.
|
||||||
|
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"name": "example.org",
|
||||||
|
"signing_keys": {
|
||||||
|
"ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ"
|
||||||
|
},
|
||||||
|
"unsigned": {
|
||||||
|
"age_ts": 922834800000
|
||||||
|
},
|
||||||
|
"signatures": {
|
||||||
|
"example.org": {
|
||||||
|
"ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
def sign_json(json_object, signing_key, signing_name):
|
||||||
|
signatures = json_object.pop("signatures", {})
|
||||||
|
unsigned = json_object.pop("unsigned", None)
|
||||||
|
|
||||||
|
signed = signing_key.sign(encode_canonical_json(json_object))
|
||||||
|
signature_base64 = encode_base64(signed.signature)
|
||||||
|
|
||||||
|
key_id = "%s:%s" % (signing_key.alg, signing_key.version)
|
||||||
|
signatures.setdefault(signing_name, {})[key_id] = signature_base64
|
||||||
|
|
||||||
|
json_object["signatures"] = signatures
|
||||||
|
if unsigned is not None:
|
||||||
|
json_object["unsigned"] = unsigned
|
||||||
|
|
||||||
|
return json_object
|
||||||
|
|
||||||
|
Checking for a Signature
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
To check if an entity has signed a JSON object an implementation does the
|
||||||
|
following:
|
||||||
|
|
||||||
|
1. Checks if the ``signatures`` member of the object contains an entry with
|
||||||
|
the name of the entity. If the entry is missing then the check fails.
|
||||||
|
2. Removes any *signing key identifiers* from the entry with algorithms it
|
||||||
|
doesn't understand. If there are no *signing key identifiers* left then the
|
||||||
|
check fails.
|
||||||
|
3. Looks up *verification keys* for the remaining *signing key identifiers*
|
||||||
|
either from a local cache or by consulting a trusted key server. If it
|
||||||
|
cannot find a *verification key* then the check fails.
|
||||||
|
4. Decodes the base64 encoded signature bytes. If base64 decoding fails then
|
||||||
|
the check fails.
|
||||||
|
5. Removes the ``signatures`` and ``unsigned`` members of the object.
|
||||||
|
6. Encodes the remainder of the JSON object using the `Canonical JSON`_
|
||||||
|
encoding.
|
||||||
|
7. Checks the signature bytes against the encoded object using the
|
||||||
|
*verification key*. If this fails then the check fails. Otherwise the check
|
||||||
|
succeeds.
|
||||||
@ -0,0 +1,171 @@
|
|||||||
|
.. Copyright 2015 OpenMarket Ltd
|
||||||
|
..
|
||||||
|
.. Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
.. you may not use this file except in compliance with the License.
|
||||||
|
.. You may obtain a copy of the License at
|
||||||
|
..
|
||||||
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
..
|
||||||
|
.. Unless required by applicable law or agreed to in writing, software
|
||||||
|
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
.. See the License for the specific language governing permissions and
|
||||||
|
.. limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
|
Cryptographic Test Vectors
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
To assist in the development of compatible implementations, the following test
|
||||||
|
values may be useful for verifying the cryptographic event signing code.
|
||||||
|
|
||||||
|
Signing Key
|
||||||
|
~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following test vectors all use the 32-byte value given by the following
|
||||||
|
Base64-encoded string as the seed for generating the ``ed25519`` signing key:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
SIGNING_KEY_SEED = decode_base64(
|
||||||
|
"YJDBA9Xnr2sVqXD9Vj7XVUnmFZcZrlw8Md7kMW+3XA1"
|
||||||
|
)
|
||||||
|
|
||||||
|
In each case, the server name and key ID are as follows:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
SERVER_NAME = "domain"
|
||||||
|
|
||||||
|
KEY_ID = "ed25519:1"
|
||||||
|
|
||||||
|
JSON Signing
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Given an empty JSON object:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{}
|
||||||
|
|
||||||
|
The JSON signing algorithm should emit the following signed data:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"signatures": {
|
||||||
|
"domain": {
|
||||||
|
"ed25519:1": "K8280/U9SSy9IVtjBuVeLr+HpOB4BQFWbg+UZaADMtTdGYI7Geitb76LTrr5QV/7Xg4ahLwYGYZzuHGZKM5ZAQ"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Given the following JSON object with data values in it:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"one": 1,
|
||||||
|
"two": "Two"
|
||||||
|
}
|
||||||
|
|
||||||
|
The JSON signing algorithm should emit the following signed JSON:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"one": 1,
|
||||||
|
"signatures": {
|
||||||
|
"domain": {
|
||||||
|
"ed25519:1": "KqmLSbO39/Bzb0QIYE82zqLwsA+PDzYIpIRA2sRQ4sL53+sN6/fpNSoqE7BP7vBZhG6kYdD13EIMJpvhJI+6Bw"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"two": "Two"
|
||||||
|
}
|
||||||
|
|
||||||
|
Event Signing
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Given the following minimally-sized event:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"event_id": "$0:domain",
|
||||||
|
"origin": "domain",
|
||||||
|
"origin_server_ts": 1000000,
|
||||||
|
"signatures": {},
|
||||||
|
"type": "X",
|
||||||
|
"unsigned": {
|
||||||
|
"age_ts": 1000000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
The event signing algorithm should emit the following signed event:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"event_id": "$0:domain",
|
||||||
|
"hashes": {
|
||||||
|
"sha256": "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI"
|
||||||
|
},
|
||||||
|
"origin": "domain",
|
||||||
|
"origin_server_ts": 1000000,
|
||||||
|
"signatures": {
|
||||||
|
"domain": {
|
||||||
|
"ed25519:1": "2Wptgo4CwmLo/Y8B8qinxApKaCkBG2fjTWB7AbP5Uy+aIbygsSdLOFzvdDjww8zUVKCmI02eP9xtyJxc/cLiBA"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"type": "X",
|
||||||
|
"unsigned": {
|
||||||
|
"age_ts": 1000000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Given the following event containing redactable content:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"content": {
|
||||||
|
"body": "Here is the message content",
|
||||||
|
},
|
||||||
|
"event_id": "$0:domain",
|
||||||
|
"origin": "domain",
|
||||||
|
"origin_server_ts": 1000000,
|
||||||
|
"type": "m.room.message",
|
||||||
|
"room_id": "!r:domain",
|
||||||
|
"sender": "@u:domain",
|
||||||
|
"signatures": {},
|
||||||
|
"unsigned": {
|
||||||
|
"age_ts": 1000000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
The event signing algorithm should emit the following signed event:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"content": {
|
||||||
|
"body": "Here is the message content",
|
||||||
|
},
|
||||||
|
"event_id": "$0:domain",
|
||||||
|
"hashes": {
|
||||||
|
"sha256": "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"
|
||||||
|
},
|
||||||
|
"origin": "domain",
|
||||||
|
"origin_server_ts": 1000000,
|
||||||
|
"type": "m.room.message",
|
||||||
|
"room_id": "!r:domain",
|
||||||
|
"sender": "@u:domain",
|
||||||
|
"signatures": {
|
||||||
|
"domain": {
|
||||||
|
"ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"unsigned": {
|
||||||
|
"age_ts": 1000000
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,140 @@
|
|||||||
|
.. Copyright 2015 OpenMarket Ltd
|
||||||
|
..
|
||||||
|
.. Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
.. you may not use this file except in compliance with the License.
|
||||||
|
.. You may obtain a copy of the License at
|
||||||
|
..
|
||||||
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
..
|
||||||
|
.. Unless required by applicable law or agreed to in writing, software
|
||||||
|
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
.. See the License for the specific language governing permissions and
|
||||||
|
.. limitations under the License.
|
||||||
|
|
||||||
|
Security Threat Model
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
Denial of Service
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The attacker could attempt to prevent delivery of messages to or from the
|
||||||
|
victim in order to:
|
||||||
|
|
||||||
|
* Disrupt service or marketing campaign of a commercial competitor.
|
||||||
|
* Censor a discussion or censor a participant in a discussion.
|
||||||
|
* Perform general vandalism.
|
||||||
|
|
||||||
|
Threat: Resource Exhaustion
|
||||||
|
+++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could cause the victims server to exhaust a particular resource
|
||||||
|
(e.g. open TCP connections, CPU, memory, disk storage)
|
||||||
|
|
||||||
|
Threat: Unrecoverable Consistency Violations
|
||||||
|
++++++++++++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could send messages which created an unrecoverable "split-brain"
|
||||||
|
state in the cluster such that the victim's servers could no longer derive a
|
||||||
|
consistent view of the chatroom state.
|
||||||
|
|
||||||
|
Threat: Bad History
|
||||||
|
+++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could convince the victim to accept invalid messages which the
|
||||||
|
victim would then include in their view of the chatroom history. Other servers
|
||||||
|
in the chatroom would reject the invalid messages and potentially reject the
|
||||||
|
victims messages as well since they depended on the invalid messages.
|
||||||
|
|
||||||
|
.. TODO-spec
|
||||||
|
Track trustworthiness of HS or users based on if they try to pretend they
|
||||||
|
haven't seen recent events, and fake a splitbrain... --M
|
||||||
|
|
||||||
|
Threat: Block Network Traffic
|
||||||
|
+++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could try to firewall traffic between the victim's server and some
|
||||||
|
or all of the other servers in the chatroom.
|
||||||
|
|
||||||
|
Threat: High Volume of Messages
|
||||||
|
+++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could send large volumes of messages to a chatroom with the victim
|
||||||
|
making the chatroom unusable.
|
||||||
|
|
||||||
|
Threat: Banning users without necessary authorisation
|
||||||
|
+++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could attempt to ban a user from a chatroom with the necessary
|
||||||
|
authorisation.
|
||||||
|
|
||||||
|
Spoofing
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
An attacker could try to send a message claiming to be from the victim without
|
||||||
|
the victim having sent the message in order to:
|
||||||
|
|
||||||
|
* Impersonate the victim while performing illicit activity.
|
||||||
|
* Obtain privileges of the victim.
|
||||||
|
|
||||||
|
Threat: Altering Message Contents
|
||||||
|
+++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could try to alter the contents of an existing message from the
|
||||||
|
victim.
|
||||||
|
|
||||||
|
Threat: Fake Message "origin" Field
|
||||||
|
+++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could try to send a new message purporting to be from the victim
|
||||||
|
with a phony "origin" field.
|
||||||
|
|
||||||
|
Spamming
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
The attacker could try to send a high volume of solicited or unsolicited
|
||||||
|
messages to the victim in order to:
|
||||||
|
|
||||||
|
* Find victims for scams.
|
||||||
|
* Market unwanted products.
|
||||||
|
|
||||||
|
Threat: Unsolicited Messages
|
||||||
|
++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could try to send messages to victims who do not wish to receive
|
||||||
|
them.
|
||||||
|
|
||||||
|
Threat: Abusive Messages
|
||||||
|
++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could send abusive or threatening messages to the victim
|
||||||
|
|
||||||
|
Spying
|
||||||
|
~~~~~~
|
||||||
|
|
||||||
|
The attacker could try to access message contents or metadata for messages sent
|
||||||
|
by the victim or to the victim that were not intended to reach the attacker in
|
||||||
|
order to:
|
||||||
|
|
||||||
|
* Gain sensitive personal or commercial information.
|
||||||
|
* Impersonate the victim using credentials contained in the messages.
|
||||||
|
(e.g. password reset messages)
|
||||||
|
* Discover who the victim was talking to and when.
|
||||||
|
|
||||||
|
Threat: Disclosure during Transmission
|
||||||
|
++++++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could try to expose the message contents or metadata during
|
||||||
|
transmission between the servers.
|
||||||
|
|
||||||
|
Threat: Disclosure to Servers Outside Chatroom
|
||||||
|
++++++++++++++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
An attacker could try to convince servers within a chatroom to send messages to
|
||||||
|
a server it controls that was not authorised to be within the chatroom.
|
||||||
|
|
||||||
|
Threat: Disclosure to Servers Within Chatroom
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
An attacker could take control of a server within a chatroom to expose message
|
||||||
|
contents or metadata for messages in that room.
|
||||||
@ -1,306 +0,0 @@
|
|||||||
.. Copyright 2016 OpenMarket Ltd
|
|
||||||
..
|
|
||||||
.. Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
.. you may not use this file except in compliance with the License.
|
|
||||||
.. You may obtain a copy of the License at
|
|
||||||
..
|
|
||||||
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
..
|
|
||||||
.. Unless required by applicable law or agreed to in writing, software
|
|
||||||
.. distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
.. See the License for the specific language governing permissions and
|
|
||||||
.. limitations under the License.
|
|
||||||
|
|
||||||
Signing Events
|
|
||||||
--------------
|
|
||||||
|
|
||||||
Canonical JSON
|
|
||||||
~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Matrix events are represented using JSON objects. If we want to sign JSON
|
|
||||||
events we need to encode the JSON as a binary string. Unfortunately the same
|
|
||||||
JSON can be encoded in different ways by changing how much white space is used
|
|
||||||
or by changing the order of keys within objects. Therefore we have to define an
|
|
||||||
encoding which can be reproduced byte for byte by any JSON library.
|
|
||||||
|
|
||||||
We define the canonical JSON encoding for a value to be the shortest UTF-8 JSON
|
|
||||||
encoding with dictionary keys lexicographically sorted by unicode codepoint.
|
|
||||||
Numbers in the JSON must be integers in the range [-(2**53)+1, (2**53)-1].
|
|
||||||
|
|
||||||
We pick UTF-8 as the encoding as it should be available to all platforms and
|
|
||||||
JSON received from the network is likely to be already encoded using UTF-8.
|
|
||||||
We sort the keys to give a consistent ordering. We force integers to be in the
|
|
||||||
range where they can be accurately represented using IEEE double precision
|
|
||||||
floating point numbers since a number of JSON libraries represent all numbers
|
|
||||||
using this representation.
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
import json
|
|
||||||
|
|
||||||
def canonical_json(value):
|
|
||||||
return json.dumps(
|
|
||||||
value,
|
|
||||||
# Encode code-points outside of ASCII as UTF-8 rather than \u escapes
|
|
||||||
ensure_ascii=False,
|
|
||||||
# Remove unnecessary white space.
|
|
||||||
separators=(',',':'),
|
|
||||||
# Sort the keys of dictionaries.
|
|
||||||
sort_keys=True,
|
|
||||||
# Encode the resulting unicode as UTF-8 bytes.
|
|
||||||
).encode("UTF-8")
|
|
||||||
|
|
||||||
Grammar
|
|
||||||
+++++++
|
|
||||||
|
|
||||||
Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
|
|
||||||
insignificant whitespace, fractions, exponents and redundant character escapes
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
value = false / null / true / object / array / number / string
|
|
||||||
false = %x66.61.6c.73.65
|
|
||||||
null = %x6e.75.6c.6c
|
|
||||||
true = %x74.72.75.65
|
|
||||||
object = %x7B [ member *( %x2C member ) ] %7D
|
|
||||||
member = string %x3A value
|
|
||||||
array = %x5B [ value *( %x2C value ) ] %5B
|
|
||||||
number = [ %x2D ] int
|
|
||||||
int = %x30 / ( %x31-39 *digit )
|
|
||||||
digit = %x30-39
|
|
||||||
string = %x22 *char %x22
|
|
||||||
char = unescaped / %x5C escaped
|
|
||||||
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
|
|
||||||
escaped = %x22 ; " quotation mark U+0022
|
|
||||||
/ %x5C ; \ reverse solidus U+005C
|
|
||||||
/ %x62 ; b backspace U+0008
|
|
||||||
/ %x66 ; f form feed U+000C
|
|
||||||
/ %x6E ; n line feed U+000A
|
|
||||||
/ %x72 ; r carriage return U+000D
|
|
||||||
/ %x74 ; t tab U+0009
|
|
||||||
/ %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X
|
|
||||||
/ %x75.30.30.31 (%x30-39 / %x61-66) ; u001X
|
|
||||||
|
|
||||||
Signing JSON
|
|
||||||
~~~~~~~~~~~~
|
|
||||||
|
|
||||||
We can now sign a JSON object by encoding it as a sequence of bytes, computing
|
|
||||||
the signature for that sequence and then adding the signature to the original
|
|
||||||
JSON object.
|
|
||||||
|
|
||||||
Signing Details
|
|
||||||
+++++++++++++++
|
|
||||||
|
|
||||||
JSON is signed by encoding the JSON object without ``signatures`` or keys grouped
|
|
||||||
as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the
|
|
||||||
signature algorithm and the signature is encoded using base64 with the padding
|
|
||||||
stripped. The resulting base64 signature is added to an object under the
|
|
||||||
*signing key identifier* which is added to the ``signatures`` object under the
|
|
||||||
name of the server signing it which is added back to the original JSON object
|
|
||||||
along with the ``unsigned`` object.
|
|
||||||
|
|
||||||
The *signing key identifier* is the concatenation of the *signing algorithm*
|
|
||||||
and a *key version*. The *signing algorithm* identifies the algorithm used to
|
|
||||||
sign the JSON. The currently support value for *signing algorithm* is
|
|
||||||
``ed25519`` as implemented by NACL (http://nacl.cr.yp.to/). The *key version*
|
|
||||||
is used to distinguish between different signing keys used by the same entity.
|
|
||||||
|
|
||||||
The ``unsigned`` object and the ``signatures`` object are not covered by the
|
|
||||||
signature. Therefore intermediate servers can add unsigned data such as timestamps
|
|
||||||
and additional signatures.
|
|
||||||
|
|
||||||
|
|
||||||
.. code:: json
|
|
||||||
|
|
||||||
{
|
|
||||||
"name": "example.org",
|
|
||||||
"signing_keys": {
|
|
||||||
"ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ"
|
|
||||||
},
|
|
||||||
"unsigned": {
|
|
||||||
"age_ts": 922834800000
|
|
||||||
},
|
|
||||||
"signatures": {
|
|
||||||
"example.org": {
|
|
||||||
"ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
def sign_json(json_object, signing_key, signing_name):
|
|
||||||
signatures = json_object.pop("signatures", {})
|
|
||||||
unsigned = json_object.pop("unsigned", None)
|
|
||||||
|
|
||||||
signed = signing_key.sign(encode_canonical_json(json_object))
|
|
||||||
signature_base64 = encode_base64(signed.signature)
|
|
||||||
|
|
||||||
key_id = "%s:%s" % (signing_key.alg, signing_key.version)
|
|
||||||
signatures.setdefault(signing_name, {})[key_id] = signature_base64
|
|
||||||
|
|
||||||
json_object["signatures"] = signatures
|
|
||||||
if unsigned is not None:
|
|
||||||
json_object["unsigned"] = unsigned
|
|
||||||
|
|
||||||
return json_object
|
|
||||||
|
|
||||||
Checking for a Signature
|
|
||||||
++++++++++++++++++++++++
|
|
||||||
|
|
||||||
To check if an entity has signed a JSON object a server does the following
|
|
||||||
|
|
||||||
1. Checks if the ``signatures`` object contains an entry with the name of the
|
|
||||||
entity. If the entry is missing then the check fails.
|
|
||||||
2. Removes any *signing key identifiers* from the entry with algorithms it
|
|
||||||
doesn't understand. If there are no *signing key identifiers* left then the
|
|
||||||
check fails.
|
|
||||||
3. Looks up *verification keys* for the remaining *signing key identifiers*
|
|
||||||
either from a local cache or by consulting a trusted key server. If it
|
|
||||||
cannot find a *verification key* then the check fails.
|
|
||||||
4. Decodes the base64 encoded signature bytes. If base64 decoding fails then
|
|
||||||
the check fails.
|
|
||||||
5. Checks the signature bytes using the *verification key*. If this fails then
|
|
||||||
the check fails. Otherwise the check succeeds.
|
|
||||||
|
|
||||||
Signing Events
|
|
||||||
~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Signing events is a more complicated process since servers can choose to redact
|
|
||||||
non-essential parts of an event. Before signing the event it is encoded as
|
|
||||||
Canonical JSON and hashed using SHA-256. The resulting hash is then stored
|
|
||||||
in the event JSON in a ``hash`` object under a ``sha256`` key.
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
def hash_event(event_json_object):
|
|
||||||
|
|
||||||
# Keys under "unsigned" can be modified by other servers.
|
|
||||||
# They are useful for conveying information like the age of an
|
|
||||||
# event that will change in transit.
|
|
||||||
# Since they can be modifed we need to exclude them from the hash.
|
|
||||||
unsigned = event_json_object.pop("unsigned", None)
|
|
||||||
|
|
||||||
# Signatures will depend on the current value of the "hashes" key.
|
|
||||||
# We cannot add new hashes without invalidating existing signatures.
|
|
||||||
signatures = event_json_object.pop("signatures", None)
|
|
||||||
|
|
||||||
# The "hashes" key might contain multiple algorithms if we decide to
|
|
||||||
# migrate away from SHA-2. We don't want to include an existing hash
|
|
||||||
# output in our hash so we exclude the "hashes" dict from the hash.
|
|
||||||
hashes = event_json_object.pop("hashes", {})
|
|
||||||
|
|
||||||
# Encode the JSON using a canonical encoding so that we get the same
|
|
||||||
# bytes on every server for the same JSON object.
|
|
||||||
event_json_bytes = encode_canonical_json(event_json_bytes)
|
|
||||||
|
|
||||||
# Add the base64 encoded bytes of the hash to the "hashes" dict.
|
|
||||||
hashes["sha256"] = encode_base64(sha256(event_json_bytes).digest())
|
|
||||||
|
|
||||||
# Add the "hashes" dict back the event JSON under a "hashes" key.
|
|
||||||
event_json_object["hashes"] = hashes
|
|
||||||
if unsigned is not None:
|
|
||||||
event_json_object["unsigned"] = unsigned
|
|
||||||
return event_json_object
|
|
||||||
|
|
||||||
The event is then stripped of all non-essential keys both at the top level and
|
|
||||||
within the ``content`` object. Any top-level keys not in the following list
|
|
||||||
MUST be removed:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
auth_events
|
|
||||||
depth
|
|
||||||
event_id
|
|
||||||
hashes
|
|
||||||
membership
|
|
||||||
origin
|
|
||||||
origin_server_ts
|
|
||||||
prev_events
|
|
||||||
prev_state
|
|
||||||
room_id
|
|
||||||
sender
|
|
||||||
signatures
|
|
||||||
state_key
|
|
||||||
type
|
|
||||||
|
|
||||||
A new ``content`` object is constructed for the resulting event that contains
|
|
||||||
only the essential keys of the original ``content`` object. If the original
|
|
||||||
event lacked a ``content`` object at all, a new empty JSON object is created
|
|
||||||
for it.
|
|
||||||
|
|
||||||
The keys that are considered essential for the ``content`` object depend on the
|
|
||||||
the ``type`` of the event. These are:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
type is "m.room.aliases":
|
|
||||||
aliases
|
|
||||||
|
|
||||||
type is "m.room.create":
|
|
||||||
creator
|
|
||||||
|
|
||||||
type is "m.room.history_visibility":
|
|
||||||
history_visibility
|
|
||||||
|
|
||||||
type is "m.room.join_rules":
|
|
||||||
join_rule
|
|
||||||
|
|
||||||
type is "m.room.member":
|
|
||||||
membership
|
|
||||||
|
|
||||||
type is "m.room.power_levels":
|
|
||||||
ban
|
|
||||||
events
|
|
||||||
events_default
|
|
||||||
kick
|
|
||||||
redact
|
|
||||||
state_default
|
|
||||||
users
|
|
||||||
users_default
|
|
||||||
|
|
||||||
The resulting stripped object with the new ``content`` object and the original
|
|
||||||
``hashes`` key is then signed using the JSON signing algorithm outlined below:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
def sign_event(event_json_object, name, key):
|
|
||||||
|
|
||||||
# Make sure the event has a "hashes" key.
|
|
||||||
if "hashes" not in event_json_object:
|
|
||||||
event_json_object = hash_event(event_json_object)
|
|
||||||
|
|
||||||
# Strip all the keys that would be removed if the event was redacted.
|
|
||||||
# The hashes are not stripped and cover all the keys in the event.
|
|
||||||
# This means that we can tell if any of the non-essential keys are
|
|
||||||
# modified or removed.
|
|
||||||
stripped_json_object = strip_non_essential_keys(event_json_object)
|
|
||||||
|
|
||||||
# Sign the stripped JSON object. The signature only covers the
|
|
||||||
# essential keys and the hashes. This means that we can check the
|
|
||||||
# signature even if the event is redacted.
|
|
||||||
signed_json_object = sign_json(stripped_json_object)
|
|
||||||
|
|
||||||
# Copy the signatures from the stripped event to the original event.
|
|
||||||
event_json_object["signatures"] = signed_json_oject["signatures"]
|
|
||||||
return event_json_object
|
|
||||||
|
|
||||||
Servers can then transmit the entire event or the event with the non-essential
|
|
||||||
keys removed. If the entire event is present, receiving servers can then check
|
|
||||||
the event by computing the SHA-256 of the event, excluding the ``hash`` object.
|
|
||||||
If the keys have been redacted, then the ``hash`` object is included when
|
|
||||||
calculating the SHA-256 instead.
|
|
||||||
|
|
||||||
New hash functions can be introduced by adding additional keys to the ``hash``
|
|
||||||
object. Since the ``hash`` object cannot be redacted a server shouldn't allow
|
|
||||||
too many hashes to be listed, otherwise a server might embed illict data within
|
|
||||||
the ``hash`` object. For similar reasons a server shouldn't allow hash values
|
|
||||||
that are too long.
|
|
||||||
|
|
||||||
.. TODO
|
|
||||||
[[TODO(markjh): We might want to specify a maximum number of keys for the
|
|
||||||
``hash`` and we might want to specify the maximum output size of a hash]]
|
|
||||||
[[TODO(markjh) We might want to allow the server to omit the output of well
|
|
||||||
known hash functions like SHA-256 when none of the keys have been redacted]]
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue