matrix-spec-proposals/proposals/4092-enforce-tests.md

MSC4092: Enforce tests around sensitive parts of the specification

This MSC proposes a change in procedure to how proposals and clarifications land in the specification. It does not add any new endpoints or alter the Matrix protocol in any way.

The specification contains the rules that all clients and servers must follow in order to interoperate. It is crucial that these rules are correct and consistent for the health of the Matrix ecosystem. This MSC proposes enforcing extra requirements on proposals and clarifications that modify these rules in order to ensure that any changes remain consistent and correct.

Proposal

Recently, it has become apparent that there are insufficient guard rails around changes to the specification, such that clarifications have been made to the specification incorrectly. These changes can have security implications on the entire Matrix ecosystem. This proposal proposes that any changes to the following areas of the specification MUST have end-to-end testing in at least one homeserver implementation:

• Room Versions
• Server-Server API:
  • Authentication
  • PDUs
  • Room State Resolution
  • Signing Events
  • Joining Rooms

All of these areas are exclusive to server implementations. This means end-to-end testing MUST test with a real homeserver implementation.

In practice, the ecosystem currently has the following end-to-end testing frameworks:

• Sytest
• Complement

As such, this proposal would force any proposal or clarification which touches the named areas of the specification to have tests in one of these projects. However, any end-to-end testing framework would suffice, at the spec core team's discretion.

Potential issues

This adds extra barriers for contributors, who now need to write code in order to get their proposal landed into the specification. However, this has always been the case as the MSC proposal demands that there is a working client and server implementation. In practice, these implementations should have tests so the actual impact on the MSC process is reduced.

However, this has significant implications on clarifications which typically do not have the same level of rigour applied to them as the proposal process. This would likely result in fewer clarifications being made to these areas, but when they are made they are more likely to be correct. The reduction in the number of clarifications to these areas may outweigh the extra safety guarantees this proposal is attempting to make.

Alternatives

The areas of the specification which need extra guard rails can be changed to reduce friction. However, these areas were chosen because they are critical to the security of Matrix rooms.

The guard rails being applied could be made more or less strict, allowing unit tests to suffice in place of end-to-end tests. Alternatively, a robust justification could be sufficient, provided it is backed with evidence. In cases where clarifications to the specification were incorrect, whilst a robust justification was proposed, it was not backed by evidence to support the claims being made.

Security considerations

This MSC should increase security around the proposals and clarifications process. Psychologically, there is a risk that reviewers may become less vigilant if there are tests. Tests are error-prone and may not be testing what is described in the clarification/proposal.

Affected clarifications

As of this writing, the following clarifications would now be subject to the extra guard rails. This is not an exhaustive list:

• https://github.com/matrix-org/matrix-spec/issues/1708
• https://github.com/matrix-org/matrix-spec/issues/1642
• https://github.com/matrix-org/matrix-spec/issues/1569
• https://github.com/matrix-org/matrix-spec/issues/1515
• https://github.com/matrix-org/matrix-spec/issues/1514
• https://github.com/matrix-org/matrix-spec/issues/1482
• https://github.com/matrix-org/matrix-spec/issues/1373
• https://github.com/matrix-org/matrix-spec/issues/1247
• https://github.com/matrix-org/matrix-spec/issues/1246
• https://github.com/matrix-org/matrix-spec/issues/1244
• https://github.com/matrix-org/matrix-spec/issues/1136
• https://github.com/matrix-org/matrix-spec/issues/1098
• https://github.com/matrix-org/matrix-spec/issues/1061
• https://github.com/matrix-org/matrix-spec/issues/1048
• https://github.com/matrix-org/matrix-spec/issues/1046
• https://github.com/matrix-org/matrix-spec/issues/849