matrix-spec-proposals/proposals/MSC4030: Progressive images in Matrix.md

MSC4030: Progressive Images in Matrix

Introduction

Efficient image handling is essential for delivering a seamless user experience within a communication platform like Matrix. This proposal recognizes the potential of modern progressive image formats to enhance image management within Matrix.

While this document will occasionally highlight JPEG XL (referred to as 'jxl') as a notable use case, it is important to note that the concepts discussed herein are applicable to both progressive and non-progressive image formats, including WebP2 and JPEG itself.

Clarifications

In Matrix, there are two types of thumbnails:

• Server-side generated thumbnails available at the thumbnail/ endpoint.
• Event thumbnails, uploaded by clients and referenced using a separate MXC.

This distinction exists because server-side generated thumbnails cannot be served in encrypted rooms or for media types other than m.image.

Reasons behind these changes

The Matrix specification currently does not take into account that clients may benefit from receiving content in formats that align with their capabilities. By introducing media negotiation mechanisms, Matrix can enhance the client experience by serving content in the most efficient way possible.

See Issue #453 for further details.

Encrypted Rooms

Due to the inability to re-encode images server-side, it is necessary to maintain the existing structure and functionality of the current API.

To enhance user experience, it would be ideal loosening the requirements for progressive images in m.image events by eliminating the need for an event thumbnail ("thumbnail_file" entry). This change reduces storage costs, as progressive images do not need thumbnails. For proper functionality, client implementations should support on-the-fly decoding and decryption of media during download.

In summary, if an image is progressive, clients MAY NOT generate the "thumbnail_file" entry (and the associated resources) in m.image messages sent in encrypted rooms.

Unencrypted Rooms

Since we can re-encode the image server-side it would be optimal to serve the image (/download endpoint) and the server-side thumbnails (/thumbnail endpoint) with the best format that align with the client capabilities.

Eliminating Unwanted Behavior

Clients do not acknowledge that, in unencrypted rooms, server-side generated thumbnails are available for image resources. As a result, they unnecessarily generate client-side thumbnails, upload them to the server, and reference them in the m.image event, creating redundant and unnecessary resources.

To fix this issue, clients SHOULD NOT generate the "thumbnail_file" entry (and the associated resources) in m.image messages sent in unencrypted rooms.

Supporting Different Formats

As proposed in MSC4011, media negotiation MUST be performed using the HTTP Content Negotiation mechanism. This mechanism allows the delivery of images in progressive or more modern formats to clients that support such features.

If the Accept header field is unspecified, absent or list unsupported formats, the server MUST provide the image resource as image/png or image/jpeg.

All progressive-capable formats MUST always provide the image as a progressive one, even if the image is uploaded as a non-progressive image in a progressive-capable format. When serving the content, the server MUST provide the image in the same format (and all others) as a progressive image.

If the negotiation is successful and a different non-progressive format is chosen for the image, the server MAY compare the size between the image in the transcoded format and the image in the original format, if the image in the original format is smaller than the transcoded one and is supported by the client, the server could chose to send the image in the original format instead of the transcoded one. However, this behavior SHOULD NOT be implemented for progressive formats.

Clients SHOULD NOT connect to the /thumbnail endpoint if the server can provide images in progressive formats, supported by the client, at the /download endpoint.

Enabling this feature allows the server to store images in more efficient formats such as JPEG XL, reducing storage costs. These images can be quickly served as JPEG for clients that do not support the jxl format. Additionally, JPEG XL can perform lossless conversion of PNG and JPEG, further reducing storage costs and bandwidth usage.

Alternatives

MSC4011 - Thumbnail media negotiation.