From d3f107f7d03f8caba9a03a8dcb0feba846907c00 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sat, 12 Nov 2022 18:46:13 +0000 Subject: [PATCH] Support content blocks system --- proposals/3553-extensible-events-video.md | 204 ++++++++-------------- 1 file changed, 68 insertions(+), 136 deletions(-) diff --git a/proposals/3553-extensible-events-video.md b/proposals/3553-extensible-events-video.md index 242d4f608..9d98aa5bb 100644 --- a/proposals/3553-extensible-events-video.md +++ b/proposals/3553-extensible-events-video.md @@ -1,19 +1,19 @@ # MSC3553: Extensible Events - Videos [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail, -though deliberately does not include schemas for non-text messaging types. This MSC covers only videos. +though deliberately does not include schemas for some messaging types. This MSC covers only videos. *Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in -stages without blocking other pieces of the overall idea. For example, an issue with the way images +stages without blocking other pieces of the overall idea. For example, an issue with the way videos are represented should not block the overall schema from going through. This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and -parts of [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552) for thumbnails. +[MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552). ## Proposal -Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` primary -event type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo). +Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` event +type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo). An example is: @@ -21,124 +21,93 @@ An example is: { "type": "m.video", "content": { - "m.text": "Upload: matrix.mp4 (12 KB)", // or other m.message-like event + "m.markup": [ + // Format of the fallback is not defined, but should have enough information for a text-only + // client to do something with the video, just like with plain file uploads. + {"body": "matrix.mp4 (12 KB, 1:30) https://example.org/_matrix/media/v3/download/example.org/abcd1234"} + ], "m.file": { "url": "mxc://example.org/abcd1234", "name": "matrix.mp4", "mimetype": "video/mp4", "size": 12345 }, - "m.thumbnail": [ - { - // an inline m.file, minus `name` - "url": "mxc://example.org/efgh5678", - "mimetype": "image/jpeg", - "size": 123, - - // an inline m.image - "width": 160, - "height": 120 + "m.video_details": { // optional + "width": 640, + "height": 480, + "duration": 90 + }, + "m.thumbnail": [ // optional + { + // A thumbnail is an m.file+m.image, or a small image + "m.file": { + "url": "mxc://exmaple.org/efgh5678", + "mimetype": "image/jpeg", + "size": 123 + + // "name" is optional in this scenario }, - // ... - ], - "m.caption": [ - // array of m.message objects - { "m.text": "matrix demo" }, - { "body": "matrix demo", "mimetype": "text/html" } + "m.image_details": { + "width": 160, + "height": 120 + } + }, + // ... ], - "m.video": { - "width": 1280, - "height": 720, - "duration": 90000 // milliseconds + "m.caption": { // optional - goes above/below video + "m.markup": [{"body": "Look at this cool animated Matrix logo"}] } } } ``` -All details shown above except `m.video` in the content are inherited from [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552). - -`m.video` is simply a container for the video-specific metadata that is not already covered by other MSCs. -Note that the event does not have an `m.image` - clients should use the first image-like thumbnail as a poster -for the video. It is reasonable for a client to include a lower quality copy (eg: 480p) as a thumbnail in the -event. +The newly introduced blocks are: -Note that videos can be encrypted using the same approach as `m.file`. +* `m.video_details` - Similar to `m.image_details` from MSC3552, optional information about the video. + `width` and `height` are required, while `duration` (length in seconds of the video) is optional. -## Potential issues +Together with content blocks from other proposals, an `m.video` is described as: -The schema duplicates some of the information into the text fallback, though this is unavoidable. +* **Required** - An `m.markup` block to act as a fallback for clients which can't process videos. +* **Required** - An `m.file` block to contain the video itself. Clients use this to show the video. +* **Optional** - An `m.video_details` block to describe any video-specific metadata, such as dimensions. + Like with existing `m.room.message` events today, clients should keep videos within a set of + reasonable bounds, regardless of sender-supplied values. For example, keeping videos at a minimum + size and within a maximum size. +* **Optional** - An `m.thumbnail` block (array) to describe "poster images" for the video. +* **Optional** - An `m.caption` block to represent any text that should be shown above or below the + video. Currently this MSC does not describe a way to pick whether the text goes above or below, + leaving this as an implementation detail. A future MSC may investigate ways of representing this, + if needed. -## Alternatives +The above describes the minimum requirements for sending an `m.video` event. Senders can add additional +blocks, however as per the extensible events system, receivers which understand video events should not +honour them. Such examples might include an `m.audio` block for "audio-only" mode (podcasts, etc) or +an `m.image` to represent the video as a GIF (or similar). -No significant alternatives known. +Note that `m.file` supports encryption and therefore it's possible to encrypt thumbnails and videos +too. -## Security considerations +If a client does not support rendering videos inline, the client would instead typically represent +the event as a plain file upload, then fall further back to a plain text message. An image fallback +is not neccessarily possible, despite all the required blocks being possible. This is due to the file +having a video mimetype, hopefully indicating to the client that an `` (or similar) is not +appropriate for this event. -The same considerations which currently apply to files, videos, and extensible events also apply here. +## Potential issues -## Transition +The schema duplicates some of the information into the text fallback, though this is unavoidable +and intentional for fallback considerations. -The same transition introduced by extensible events is also applied here: +## Alternatives -```json5 -{ - "type": "m.room.message", - "content": { - "body": "matrix.mp4", - "msgtype": "m.video", - "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", - "info": { - "duration": 90000, - "mimetype": "video/mp4", - "size": 12345, - "w": 1280, - "h": 720, - "thumbnail_url": "mxc://example.org/elsewhere", - "thumbnail_info": { - "size": 123, - "mimetype": "image/jpeg", - "w": 160, - "h": 120 - } - }, +No significant alternatives known. - // Extensible Events - "m.text": "matrix.mp4", // or other m.message-like event - "m.file": { - "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", - "name": "matrix.mp4", - "mimetype": "video/mp4", - "size": 12345 - }, - "m.thumbnail": [ - { - // an inline m.file, minus `name` - "url": "mxc://example.org/elsewhere", - "mimetype": "image/jpeg", - "size": 123, - - // an inline m.image - "width": 160, - "height": 120 - }, - // ... - ], - "m.caption": [ - // array of m.message objects - { "m.text": "matrix demo" }, - { "body": "matrix demo", "mimetype": "text/html" } - ], - "m.video": { - "width": 1280, - "height": 720, - "duration": 90000 // milliseconds - } - } -} -``` +## Security considerations -The event details are copied and quite verbose, however this is best to ensure compatibility with the -extensible events format. +The same considerations which currently apply to files, videos, and extensible events also +apply here. For example, bounds on video size, assuming sender-provided details about the file are +false, etc. ## Unstable prefix @@ -146,41 +115,4 @@ While this MSC is not considered stable, implementations should use `org.matrix. place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than the namespace of this MSC - this is deliberate. -Example: -```json5 -{ - "type": "org.matrix.msc1767.video", - "content": { - "org.matrix.msc1767.text": "matrix.mp4", // or other m.message-like event - "org.matrix.msc1767.file": { - "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", - "name": "matrix.mp4", - "mimetype": "video/mp4", - "size": 12345 - }, - "org.matrix.msc1767.thumbnail": [ - { - // an inline m.file, minus `name` - "url": "mxc://example.org/elsewhere", - "mimetype": "image/jpeg", - "size": 123, - - // an inline m.image - "width": 160, - "height": 120 - }, - // ... - ], - "org.matrix.msc1767.caption": [ - // array of m.message objects - { "m.text": "matrix demo" }, - { "body": "matrix demo", "mimetype": "text/html" } - ], - "org.matrix.msc1767.video": { - "width": 1280, - "height": 720, - "duration": 90000 // milliseconds - } - } -} -``` +Note that extensible events should only be used in an appropriate room version as well.