From 30bddac30806f46e6b9c569ceb98258a704d3014 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 7 Dec 2021 16:28:41 -0700 Subject: [PATCH 1/6] Extensible Events - Videos --- proposals/3553-extensible-events-video.md | 187 ++++++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 proposals/3553-extensible-events-video.md diff --git a/proposals/3553-extensible-events-video.md b/proposals/3553-extensible-events-video.md new file mode 100644 index 000000000..f94fc2bb8 --- /dev/null +++ b/proposals/3553-extensible-events-video.md @@ -0,0 +1,187 @@ +# MSC3553: Extensible Events - Videos + +[MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail, +though deliberately does not include schemas for non-text messaging types. This MSC covers only images +and stickers. + +*Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in +stages without blocking other pieces of the overall idea. For example, an issue with the way images +are represented should not block the overall schema from going through. + +This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and +parts of [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552) for thumbnails. + +## Proposal + +Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` primary +event type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo). + +An example is: + +```json5 +{ + "type": "m.video", + "content": { + "m.text": "Upload: matrix.mp4 (12 KB)", // or other m.message-like event + "m.file": { + "url": "mxc://example.org/abcd1234", + "name": "matrix.mp4", + "mimetype": "video/mp4", + "size": 12345 + }, + "m.thumbnail": [ + { + // an inline m.file, minus `name` + "url": "mxc://example.org/efgh5678", + "mimetype": "image/jpeg", + "size": 123, + + // an inline m.image + "width": 160, + "height": 120 + }, + // ... + ], + "m.caption": [ + // array of m.message objects + { "m.text": "matrix demo" }, + { "body": "matrix demo", "mimetype": "text/html" } + ], + "m.video": { + "width": 1280, + "height": 720, + "duration": 90000 // milliseconds + } + } +} +``` + +All details shown above except `m.video` in the content are inherited from [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552). + +`m.video` is simply a container for the video-specific metadata that is not already covered by other MSCs. +Note that the event does not have an `m.image` - clients should use the first image-like thumbnail as a poster +for the video. It is reasonable for a client to include a lower quality copy (eg: 480p) as a thumbnail in the +event. + +Note that videos can be encrypted using the same approach as `m.file`. + +## Potential issues + +The schema duplicates some of the information into the text fallback, though this is unavoidable. + +## Alternatives + +No significant alternatives known. + +## Security considerations + +The same considerations which currently apply to files, videos, and extensible events also apply here. + +## Transition + +The same transition introduced by extensible events is also applied here: + +```json5 +{ + "type": "m.room.message", + "content": { + "body": "matrix.mp4", + "msgtype": "m.video", + "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", + "info": { + "duration": 90000, + "mimetype": "video/mp4", + "size": 12345, + "w": 1280, + "h": 720, + "thumbnail_url": "mxc://example.org/elsewhere", + "thumbnail_info": { + "size": 123, + "mimetype": "image/jpeg", + "w": 160, + "h": 120 + } + }, + + // Extensible Events + "m.text": "matrix.mp4", // or other m.message-like event + "m.file": { + "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", + "name": "matrix.mp4", + "mimetype": "video/mp4", + "size": 12345 + }, + "m.thumbnail": [ + { + // an inline m.file, minus `name` + "url": "mxc://example.org/elsewhere", + "mimetype": "image/jpeg", + "size": 123, + + // an inline m.image + "width": 160, + "height": 120 + }, + // ... + ], + "m.caption": [ + // array of m.message objects + { "m.text": "matrix demo" }, + { "body": "matrix demo", "mimetype": "text/html" } + ], + "m.video": { + "width": 1280, + "height": 720, + "duration": 90000 // milliseconds + } + } +} +``` + +The event details are copied and quite verbose, however this is best to ensure compatibility with the +extensible events format. + +## Unstable prefix + +While this MSC is not considered stable, implementations should use `org.matrix.msc1767.*` as a prefix in +place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than +the namespace of this MSC - this is deliberate. + +Example: +```json5 +{ + "type": "org.matrix.msc1767.video", + "content": { + "org.matrix.msc1767.text": "matrix.mp4", // or other m.message-like event + "org.matrix.msc1767.file": { + "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", + "name": "matrix.mp4", + "mimetype": "video/mp4", + "size": 12345 + }, + "org.matrix.msc1767.thumbnail": [ + { + // an inline m.file, minus `name` + "url": "mxc://example.org/elsewhere", + "mimetype": "image/jpeg", + "size": 123, + + // an inline m.image + "width": 160, + "height": 120 + }, + // ... + ], + "org.matrix.msc1767.caption": [ + // array of m.message objects + { "m.text": "matrix demo" }, + { "body": "matrix demo", "mimetype": "text/html" } + ], + "org.matrix.msc1767.video": { + "width": 1280, + "height": 720, + "duration": 90000 // milliseconds + } + } +} +``` From 39d7395590dcac3b5bc517e9239ade419437a98d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 7 Dec 2021 16:39:56 -0700 Subject: [PATCH 2/6] Remove evidence that this was copy/pasted --- proposals/3553-extensible-events-video.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/3553-extensible-events-video.md b/proposals/3553-extensible-events-video.md index f94fc2bb8..242d4f608 100644 --- a/proposals/3553-extensible-events-video.md +++ b/proposals/3553-extensible-events-video.md @@ -1,8 +1,7 @@ # MSC3553: Extensible Events - Videos [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail, -though deliberately does not include schemas for non-text messaging types. This MSC covers only images -and stickers. +though deliberately does not include schemas for non-text messaging types. This MSC covers only videos. *Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in stages without blocking other pieces of the overall idea. For example, an issue with the way images From d3f107f7d03f8caba9a03a8dcb0feba846907c00 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sat, 12 Nov 2022 18:46:13 +0000 Subject: [PATCH 3/6] Support content blocks system --- proposals/3553-extensible-events-video.md | 204 ++++++++-------------- 1 file changed, 68 insertions(+), 136 deletions(-) diff --git a/proposals/3553-extensible-events-video.md b/proposals/3553-extensible-events-video.md index 242d4f608..9d98aa5bb 100644 --- a/proposals/3553-extensible-events-video.md +++ b/proposals/3553-extensible-events-video.md @@ -1,19 +1,19 @@ # MSC3553: Extensible Events - Videos [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail, -though deliberately does not include schemas for non-text messaging types. This MSC covers only videos. +though deliberately does not include schemas for some messaging types. This MSC covers only videos. *Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in -stages without blocking other pieces of the overall idea. For example, an issue with the way images +stages without blocking other pieces of the overall idea. For example, an issue with the way videos are represented should not block the overall schema from going through. This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and -parts of [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552) for thumbnails. +[MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552). ## Proposal -Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` primary -event type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo). +Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` event +type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo). An example is: @@ -21,124 +21,93 @@ An example is: { "type": "m.video", "content": { - "m.text": "Upload: matrix.mp4 (12 KB)", // or other m.message-like event + "m.markup": [ + // Format of the fallback is not defined, but should have enough information for a text-only + // client to do something with the video, just like with plain file uploads. + {"body": "matrix.mp4 (12 KB, 1:30) https://example.org/_matrix/media/v3/download/example.org/abcd1234"} + ], "m.file": { "url": "mxc://example.org/abcd1234", "name": "matrix.mp4", "mimetype": "video/mp4", "size": 12345 }, - "m.thumbnail": [ - { - // an inline m.file, minus `name` - "url": "mxc://example.org/efgh5678", - "mimetype": "image/jpeg", - "size": 123, - - // an inline m.image - "width": 160, - "height": 120 + "m.video_details": { // optional + "width": 640, + "height": 480, + "duration": 90 + }, + "m.thumbnail": [ // optional + { + // A thumbnail is an m.file+m.image, or a small image + "m.file": { + "url": "mxc://exmaple.org/efgh5678", + "mimetype": "image/jpeg", + "size": 123 + + // "name" is optional in this scenario }, - // ... - ], - "m.caption": [ - // array of m.message objects - { "m.text": "matrix demo" }, - { "body": "matrix demo", "mimetype": "text/html" } + "m.image_details": { + "width": 160, + "height": 120 + } + }, + // ... ], - "m.video": { - "width": 1280, - "height": 720, - "duration": 90000 // milliseconds + "m.caption": { // optional - goes above/below video + "m.markup": [{"body": "Look at this cool animated Matrix logo"}] } } } ``` -All details shown above except `m.video` in the content are inherited from [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552). - -`m.video` is simply a container for the video-specific metadata that is not already covered by other MSCs. -Note that the event does not have an `m.image` - clients should use the first image-like thumbnail as a poster -for the video. It is reasonable for a client to include a lower quality copy (eg: 480p) as a thumbnail in the -event. +The newly introduced blocks are: -Note that videos can be encrypted using the same approach as `m.file`. +* `m.video_details` - Similar to `m.image_details` from MSC3552, optional information about the video. + `width` and `height` are required, while `duration` (length in seconds of the video) is optional. -## Potential issues +Together with content blocks from other proposals, an `m.video` is described as: -The schema duplicates some of the information into the text fallback, though this is unavoidable. +* **Required** - An `m.markup` block to act as a fallback for clients which can't process videos. +* **Required** - An `m.file` block to contain the video itself. Clients use this to show the video. +* **Optional** - An `m.video_details` block to describe any video-specific metadata, such as dimensions. + Like with existing `m.room.message` events today, clients should keep videos within a set of + reasonable bounds, regardless of sender-supplied values. For example, keeping videos at a minimum + size and within a maximum size. +* **Optional** - An `m.thumbnail` block (array) to describe "poster images" for the video. +* **Optional** - An `m.caption` block to represent any text that should be shown above or below the + video. Currently this MSC does not describe a way to pick whether the text goes above or below, + leaving this as an implementation detail. A future MSC may investigate ways of representing this, + if needed. -## Alternatives +The above describes the minimum requirements for sending an `m.video` event. Senders can add additional +blocks, however as per the extensible events system, receivers which understand video events should not +honour them. Such examples might include an `m.audio` block for "audio-only" mode (podcasts, etc) or +an `m.image` to represent the video as a GIF (or similar). -No significant alternatives known. +Note that `m.file` supports encryption and therefore it's possible to encrypt thumbnails and videos +too. -## Security considerations +If a client does not support rendering videos inline, the client would instead typically represent +the event as a plain file upload, then fall further back to a plain text message. An image fallback +is not neccessarily possible, despite all the required blocks being possible. This is due to the file +having a video mimetype, hopefully indicating to the client that an `` (or similar) is not +appropriate for this event. -The same considerations which currently apply to files, videos, and extensible events also apply here. +## Potential issues -## Transition +The schema duplicates some of the information into the text fallback, though this is unavoidable +and intentional for fallback considerations. -The same transition introduced by extensible events is also applied here: +## Alternatives -```json5 -{ - "type": "m.room.message", - "content": { - "body": "matrix.mp4", - "msgtype": "m.video", - "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", - "info": { - "duration": 90000, - "mimetype": "video/mp4", - "size": 12345, - "w": 1280, - "h": 720, - "thumbnail_url": "mxc://example.org/elsewhere", - "thumbnail_info": { - "size": 123, - "mimetype": "image/jpeg", - "w": 160, - "h": 120 - } - }, +No significant alternatives known. - // Extensible Events - "m.text": "matrix.mp4", // or other m.message-like event - "m.file": { - "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", - "name": "matrix.mp4", - "mimetype": "video/mp4", - "size": 12345 - }, - "m.thumbnail": [ - { - // an inline m.file, minus `name` - "url": "mxc://example.org/elsewhere", - "mimetype": "image/jpeg", - "size": 123, - - // an inline m.image - "width": 160, - "height": 120 - }, - // ... - ], - "m.caption": [ - // array of m.message objects - { "m.text": "matrix demo" }, - { "body": "matrix demo", "mimetype": "text/html" } - ], - "m.video": { - "width": 1280, - "height": 720, - "duration": 90000 // milliseconds - } - } -} -``` +## Security considerations -The event details are copied and quite verbose, however this is best to ensure compatibility with the -extensible events format. +The same considerations which currently apply to files, videos, and extensible events also +apply here. For example, bounds on video size, assuming sender-provided details about the file are +false, etc. ## Unstable prefix @@ -146,41 +115,4 @@ While this MSC is not considered stable, implementations should use `org.matrix. place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than the namespace of this MSC - this is deliberate. -Example: -```json5 -{ - "type": "org.matrix.msc1767.video", - "content": { - "org.matrix.msc1767.text": "matrix.mp4", // or other m.message-like event - "org.matrix.msc1767.file": { - "url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35", - "name": "matrix.mp4", - "mimetype": "video/mp4", - "size": 12345 - }, - "org.matrix.msc1767.thumbnail": [ - { - // an inline m.file, minus `name` - "url": "mxc://example.org/elsewhere", - "mimetype": "image/jpeg", - "size": 123, - - // an inline m.image - "width": 160, - "height": 120 - }, - // ... - ], - "org.matrix.msc1767.caption": [ - // array of m.message objects - { "m.text": "matrix demo" }, - { "body": "matrix demo", "mimetype": "text/html" } - ], - "org.matrix.msc1767.video": { - "width": 1280, - "height": 720, - "duration": 90000 // milliseconds - } - } -} -``` +Note that extensible events should only be used in an appropriate room version as well. From a08441b8f14b71c1aae84d25b1bb23c449e4a028 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 14 Nov 2022 09:41:12 -0700 Subject: [PATCH 4/6] Spelling --- proposals/3553-extensible-events-video.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3553-extensible-events-video.md b/proposals/3553-extensible-events-video.md index 9d98aa5bb..582f7f403 100644 --- a/proposals/3553-extensible-events-video.md +++ b/proposals/3553-extensible-events-video.md @@ -90,7 +90,7 @@ too. If a client does not support rendering videos inline, the client would instead typically represent the event as a plain file upload, then fall further back to a plain text message. An image fallback -is not neccessarily possible, despite all the required blocks being possible. This is due to the file +is not necessarily possible, despite all the required blocks being possible. This is due to the file having a video mimetype, hopefully indicating to the client that an `` (or similar) is not appropriate for this event. From 758d33d2622a065a3bc135dda34e0cdf0b90f031 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 18 Apr 2023 14:32:29 -0600 Subject: [PATCH 5/6] Update for MSC1767 --- proposals/3553-extensible-events-video.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/3553-extensible-events-video.md b/proposals/3553-extensible-events-video.md index 582f7f403..74f81f28f 100644 --- a/proposals/3553-extensible-events-video.md +++ b/proposals/3553-extensible-events-video.md @@ -21,7 +21,7 @@ An example is: { "type": "m.video", "content": { - "m.markup": [ + "m.text": [ // Format of the fallback is not defined, but should have enough information for a text-only // client to do something with the video, just like with plain file uploads. {"body": "matrix.mp4 (12 KB, 1:30) https://example.org/_matrix/media/v3/download/example.org/abcd1234"} @@ -55,7 +55,7 @@ An example is: // ... ], "m.caption": { // optional - goes above/below video - "m.markup": [{"body": "Look at this cool animated Matrix logo"}] + "m.text": [{"body": "Look at this cool animated Matrix logo"}] } } } @@ -68,7 +68,7 @@ The newly introduced blocks are: Together with content blocks from other proposals, an `m.video` is described as: -* **Required** - An `m.markup` block to act as a fallback for clients which can't process videos. +* **Required** - An `m.text` block to act as a fallback for clients which can't process videos. * **Required** - An `m.file` block to contain the video itself. Clients use this to show the video. * **Optional** - An `m.video_details` block to describe any video-specific metadata, such as dimensions. Like with existing `m.room.message` events today, clients should keep videos within a set of From 64188eb05d9d8913d5dab5c5663a44f15d4d912b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 17 Apr 2023 21:33:45 -0600 Subject: [PATCH 6/6] Remove excess artifact --- themes/docsy | 1 - 1 file changed, 1 deletion(-) delete mode 160000 themes/docsy diff --git a/themes/docsy b/themes/docsy deleted file mode 160000 index 5023a2914..000000000 --- a/themes/docsy +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 5023a2914528e012ecf3ec85a56028c00ee97dd2