Signing Events ============== Canonical JSON -------------- Matrix events are represented using JSON objects. If we want to sign JSON events we need to encode the JSON as a binary string. Unfortunately the same JSON can be encoded in different ways by changing how much white space is used or by changing the order of keys within objects. Therefore we have to define an encoding which can be reproduced byte for byte by any JSON library. We define the canonical JSON encoding for a value to be the shortest UTF-8 JSON encoding with dictionary keys lexicographically sorted by unicode codepoint. Numbers in the JSON must be integers in the range [-(2**53)+1, (2**53)-1]. We pick UTF-8 as the encoding as it should be available to all platforms and JSON received from the network is likely to be already encoded using UTF-8. We sort the keys to give a consistent ordering. We force integers to be in the range where they can be accurately represented using IEEE double precision floating point numbers since a number of JSON libraries represent all numbers using this representation. .. code:: python import json def canonical_json(value): return json.dumps( value, # Encode code-points outside of ASCII as UTF-8 rather than \u escapes ensure_ascii=False, # Remove unecessary white space. separators=(',',':'), # Sort the keys of dictionaries. sort_keys=True, # Encode the resulting unicode as UTF-8 bytes. ).encode("UTF-8") Grammar ~~~~~~~ Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing insignificant whitespace, fractions, exponents and redundant character escapes .. code:: value = false / null / true / object / array / number / string false = %x66.61.6c.73.65 null = %x6e.75.6c.6c true = %x74.72.75.65 object = %x7B [ member *( %x2C member ) ] %7D member = string %x3A value array = %x5B [ value *( %x2C value ) ] %5B number = [ %x2D ] int int = %x30 / ( %x31-39 *digit ) digit = %x30-39 string = %x22 *char %x22 char = unescaped / %x5C escaped unescaped = %x20-21 / %x23-5B / %x5D-10FFFF escaped = %x22 ; " quotation mark U+0022 / %x5C ; \ reverse solidus U+005C / %x62 ; b backspace U+0008 / %x66 ; f form feed U+000C / %x6E ; n line feed U+000A / %x72 ; r carriage return U+000D / %x74 ; t tab U+0009 / %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X / %x75.30.30.31 (%x30-39 / %x61-66) ; u001X Signing JSON ------------ We can now sign a JSON object by encoding it as a sequence of bytes, computing the signature for that sequence and then adding the signature to the original JSON object. Signing Details ~~~~~~~~~~~~~~~ JSON is signed by encoding the JSON object without ``signatures`` or keys grouped as ``unsigned``, using the canonical encoding described above. The JSON bytes are then signed using the signature algorithm and the signature encoded using base64 with the padding stripped. The resulting base64 signature is added to an object under the *signing key identifier* which is added to the ``signatures`` object under the name of the server signing it which is added back to the original JSON object along with the ``unsigned`` object. The *signing key identifier* is the concatenation of the *signing algorithm* and a *key version*. The *signing algorithm* identifies the algorithm used to sign the JSON. The currently support value for *signing algorithm* is ``ed25519`` as implemented by NACL (http://nacl.cr.yp.to/). The *key version* is used to distinguish between different signing keys used by the same entity. The ``unsigned`` object and the ``signatures`` object are not covered by the signature. Therefore intermediate servers can add unsigned data such as timestamps and additional signatures. .. code:: json { "name": "example.org", "signing_keys": { "ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ" }, "unsigned": { "age_ts": 922834800000 }, "signatures": { "example.org": { "ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw" } } } .. code:: python def sign_json(json_object, signing_key, signing_name): signatures = json_object.pop("signatures", {}) unsigned = json_object.pop("unsigned", None) signed = signing_key.sign(encode_canonical_json(json_object)) signature_base64 = encode_base64(signed.signature) key_id = "%s:%s" % (signing_key.alg, signing_key.version) signatures.setdefault(sigature_name, {})[key_id] = signature_base64 json_object["signatures"] = signatures if unsigned is not None: json_object["unsigned"] = unsigned return json_object Checking for a Signature ~~~~~~~~~~~~~~~~~~~~~~~~ To check if an entity has signed a JSON object a server does the following 1. Checks if the ``signatures`` object contains an entry with the name of the entity. If the entry is missing then the check fails. 2. Removes any *signing key identifiers* from the entry with algorithms it doesn't understand. If there are no *signing key identifiers* left then the check fails. 3. Looks up *verification keys* for the remaining *signing key identifiers* either from a local cache or by consulting a trusted key server. If it cannot find a *verification key* then the check fails. 4. Decodes the base64 encoded signature bytes. If base64 decoding fails then the check fails. 5. Checks the signature bytes using the *verification key*. If this fails then the check fails. Otherwise the check succeeds. Signing Events -------------- Signing events is a more complicated process since servers can choose to redact non-essential parts of an event. Before signing the event it is encoded as Canonical JSON and hashed using SHA-256. The resulting hash is then stored in the event JSON in a ``hash`` object under a ``sha256`` key. .. code:: python def hash_event(event_json_object): # Keys under "unsigned" can be modified by other servers. # They are useful for conveying information like the age of an # event that will change in transit. # Since they can be modifed we need to exclude them from the hash. unsigned = event_json_object.pop("unsigned", None) # Signatures will depend on the current value of the "hashes" key. # We cannot add new hashes without invalidating existing signatures. signatures = event_json_object.pop("signatures", None) # The "hashes" key might contain multiple algorithms if we decide to # migrate away from SHA-2. We don't want to include an existing hash # output in our hash so we exclude the "hashes" dict from the hash. hashes = event_json_object.pop("hashes", {}) # Encode the JSON using a canonical encoding so that we get the same # bytes on every server for the same JSON object. event_json_bytes = encode_canonical_json(event_json_bytes) # Add the base64 encoded bytes of the hash to the "hashes" dict. hashes["sha256"] = encode_base64(sha256(event_json_bytes).digest()) # Add the "hashes" dict back the event JSON under a "hashes" key. event_json_object["hashes"] = hashes if unsigned is not None: event_json_object["unsigned"] = unsigned return event_json_object Then all non-essential keys are stripped from the event object, and the resulting object which included the ``hash`` key is signed using the JSON signing algorithm .. code:: python def sign_event(event_json_object, name, key): # Make sure the event has a "hashes" key. if "hashes" not in event_json_object: event_json_object = hash_event(event_json_object) # Strip all the keys that would be removed if the event was redacted. # The hashes are not stripped and cover all the keys in the event. # This means that we can tell if any of the non-essential keys are # modified or removed. stripped_json_object = strip_non_essential_keys(event_json_object) # Sign the stripped JSON object. The signature only covers the # essential keys and the hashes. This means that we can check the # signature even if the event is redacted. signed_json_object = sign_json(stripped_json_object) # Copy the signatures from the stripped event to the original event. event_json_object["signatures"] = signed_json_oject["signatures"] return event_json_object Servers can then transmit the entire event or the event with the non-essential keys removed. If the entire event is present, receiving servers can then check the event by computing the SHA-256 of the event, excluding the ``hash`` object. If the keys have been redacted, then the ``hash`` object is included when calculating the SHA-256 instead. New hash functions can be introduced by adding additional keys to the ``hash`` object. Since the ``hash`` object cannot be redacted a server shouldn't allow too many hashes to be listed, otherwise a server might embed illict data within the ``hash`` object. For similar reasons a server shouldn't allow hash values that are too long. [[TODO(markjh): We might want to specify a maximum number of keys for the ``hash`` and we might want to specify the maximum output size of a hash]] [[TODO(markjh) We might want to allow the server to omit the output of well known hash functions like SHA-256 when none of the keys have been redacted]]