matrix-spec-proposals/proposals/2265-email-lowercase.md

Proposal for mandating case folding when processing e-mail addresses

RFC822 mandates that localparts in e-mail addresses must be processed with the original case preserved. The Matrix spec doesn't mandate anything about processing e-mail addresses, other than the fact that the domain part must be converted to lowercase, as domain names are case insensitive.

On the other hand, most major e-mail providers nowadays process the localparts of e-mail addresses as case insensitive. Therefore, most users expect localparts to be treated case insensitively, and get confused when it's not. Some users, for example, get confused over the fact that registering a 3PID association for john.doe@example.com doesn't mean that the association is valid for John.Doe@example.com, and don't expect to have to remember the exact case they used to initially register the association (and sometimes get locked out of their account because of that). So far we've seen that confusion occur and lead to troubles of various degrees over several deployments of Synapse and Sydent.

Proposal

This proposal suggests changing the specification of the e-mail 3PID type in the Matrix spec appendices to mandate that, before any processing, e-mail addresses must go through a full case folding as described under "Caseless Matching" in chapter 5 of the unicode standard, on top of having their domain lowercased.

This means that Strauß@Example.com must be considered as being the same e-mail address as strauss@example.com.

Other considered solutions

A first look at this issue concluded that there was no need to add such a mention to the spec, and that it can be considered an implementation detail. However, MSC2134 changes this: because hashing functions are case sensitive, we need both clients and identity servers to follow the same policy regarding case sensitivity.

An initial version of this proposal proposed to mandate lowercasing e-mail addresses instead of case folding them, however it was pointed out that this solution might not be the best and most future-proof one.

Unicode normalisation was also looked at but judged unnecessary.

Tradeoffs

Implementing this MSC in identity servers and homeservers might require the databases of existing instances to be updated in a large part to case fold the email addresses of existing associations, in order to avoid conflicts. However, most of this update can usually be done by a background job running at startup, so the UX improvement outweighs this trouble.

Potential issues

Conflicts with existing associations

Some users might already have two different accounts associated with the same e-mail address but with different cases. This appears to happen in a small number of cases, however, and can be dealt with by the identity server's or the homeserver's maintainer.

For example, with Sydent, the process of dealing with such cases could look like:

1. list all MXIDs associated with a variant of the email address, and the timestamp of that association
2. delete all associations except for the most recent one [0]
3. inform the user of the deletion by sending them an email notice to the email address

Storing and querying

Most database engines don't support case folding, therefore querying all e-mail addresses matching a case folded e-mail address might not be trivial, e.g. an identity server querying all associations for strauss@example.com when processing a /lookup request would be expected to also get associations for Strauß@Example.com.

To address this issue, implementation maintainers are strongly encouraged to make e-mail addresses go through a full case folding before storing them.

Implementing case folding

The need for case folding in services on the Internet doesn't seem to be very large currently (probably due to its young age), therefore there seem to be only a few third-party implementation libraries out there. However, both Go, Python 2 and Python 3 support it natively, and a third-party JavaScript implementation exists which, although young, seems to be working.

Footnotes

[0]: This is specific to Sydent because of a bug it has where v1 lookups are already processed case insensitively, which means it will return the most recent association for any case of the given email address, therefore keeping only this association won't change the result of v1 lookups.