From fb0d9db0e19290dc4a1ded3b3a443eda134f3da3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal <7190048+kegsay@users.noreply.github.com> Date: Tue, 23 Jan 2024 17:59:45 +0000 Subject: [PATCH] MSC4092: Enforce tests around sensitive parts of the specification --- proposals/4092-enforce-tests.md | 86 +++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 proposals/4092-enforce-tests.md diff --git a/proposals/4092-enforce-tests.md b/proposals/4092-enforce-tests.md new file mode 100644 index 00000000..1f75316a --- /dev/null +++ b/proposals/4092-enforce-tests.md @@ -0,0 +1,86 @@ +# MSC4092: Enforce tests around sensitive parts of the specification + +*This MSC proposes a change in procedure to how proposals and clarifications land in the specification. +It does not add any new endpoints or alter the Matrix protocol in any way.* + +The specification contains the rules that all clients and servers must follow in order to interoperate. +It is crucial that these rules are correct and consistent for the health of the Matrix ecosystem. +This MSC proposes enforcing extra requirements on proposals _and clarifications_ that modify these rules +in order to ensure that any changes remain consistent and correct. + +## Proposal + +Recently, it has become apparent that there are insufficient guard rails around changes to the specification, +such that [clarifications](https://github.com/matrix-org/matrix-spec/issues/1710) have been made to the +specification incorrectly. These changes can have security implications on the entire Matrix ecosystem. This +proposal proposes that any changes to the following areas of the specification MUST have end-to-end testing in +at least one homeserver implementation: + - Room Versions + - Server-Server API: + * Authentication + * PDUs + * Room State Resolution + * Signing Events + * Joining Rooms + +All of these areas are exclusive to server implementations. This means end-to-end testing MUST test with a +real homeserver implementation. + +In practice, the ecosystem currently has the following end-to-end testing frameworks: + - [Sytest](https://github.com/matrix-org/sytest) + - [Complement](https://github.com/matrix-org/complement) + +As such, **this proposal would force any proposal or clarification which touches the named areas of the specification +to have tests** in one of these projects. However, any end-to-end testing framework would suffice, at the +spec core team's discretion. + +## Potential issues + +This adds extra barriers for contributors, who now need to write code in order to get their proposal landed into +the specification. However, this has always been the case as the MSC proposal demands that there is a working +client and server implementation. In practice, these implementations _should have tests_ so the actual impact on +the MSC process is reduced. + +However, this has significant implications on clarifications which typically do not have the same level of rigour +applied to them as the proposal process. This would likely result in fewer clarifications being made to these areas, +but when they are made they are more likely to be correct. The reduction in the number of clarifications to these +areas may outweigh the extra safety guarantees this proposal is attempting to make. + + +## Alternatives + +The areas of the specification which need extra guard rails can be changed to reduce friction. However, +these areas were chosen because they are critical to the security of Matrix rooms. + +The guard rails being applied could be made more or less strict, allowing unit tests to suffice in place +of end-to-end tests. Alternatively, a robust justification could be sufficient, provided it is backed +with evidence. In cases where clarifications to the specification were incorrect, whilst a robust +justification _was proposed_, it was not backed by evidence to support the claims being made. + + +## Security considerations + +This MSC _should_ increase security around the proposals and clarifications process. +Psychologically, there is a risk that reviewers may become less vigilant if there are tests. Tests +are error-prone and may not be testing what is described in the clarification/proposal. + +## Affected clarifications + +As of this writing, the following clarifications would now be subject to the extra guard rails. This is +not an exhaustive list: + - https://github.com/matrix-org/matrix-spec/issues/1708 + - https://github.com/matrix-org/matrix-spec/issues/1642 + - https://github.com/matrix-org/matrix-spec/issues/1569 + - https://github.com/matrix-org/matrix-spec/issues/1515 + - https://github.com/matrix-org/matrix-spec/issues/1514 + - https://github.com/matrix-org/matrix-spec/issues/1482 + - https://github.com/matrix-org/matrix-spec/issues/1373 + - https://github.com/matrix-org/matrix-spec/issues/1247 + - https://github.com/matrix-org/matrix-spec/issues/1246 + - https://github.com/matrix-org/matrix-spec/issues/1244 + - https://github.com/matrix-org/matrix-spec/issues/1136 + - https://github.com/matrix-org/matrix-spec/issues/1098 + - https://github.com/matrix-org/matrix-spec/issues/1061 + - https://github.com/matrix-org/matrix-spec/issues/1048 + - https://github.com/matrix-org/matrix-spec/issues/1046 + - https://github.com/matrix-org/matrix-spec/issues/849 \ No newline at end of file