Support content blocks system

travis/msc/extev/video
Travis Ralston 3 years ago
parent 39d7395590
commit d3f107f7d0

@ -1,19 +1,19 @@
# MSC3553: Extensible Events - Videos # MSC3553: Extensible Events - Videos
[MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail, [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail,
though deliberately does not include schemas for non-text messaging types. This MSC covers only videos. though deliberately does not include schemas for some messaging types. This MSC covers only videos.
*Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in *Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in
stages without blocking other pieces of the overall idea. For example, an issue with the way images stages without blocking other pieces of the overall idea. For example, an issue with the way videos
are represented should not block the overall schema from going through. are represented should not block the overall schema from going through.
This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and
parts of [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552) for thumbnails. [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552).
## Proposal ## Proposal
Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` primary Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` event
event type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo). type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo).
An example is: An example is:
@ -21,124 +21,93 @@ An example is:
{ {
"type": "m.video", "type": "m.video",
"content": { "content": {
"m.text": "Upload: matrix.mp4 (12 KB)", // or other m.message-like event "m.markup": [
// Format of the fallback is not defined, but should have enough information for a text-only
// client to do something with the video, just like with plain file uploads.
{"body": "matrix.mp4 (12 KB, 1:30) https://example.org/_matrix/media/v3/download/example.org/abcd1234"}
],
"m.file": { "m.file": {
"url": "mxc://example.org/abcd1234", "url": "mxc://example.org/abcd1234",
"name": "matrix.mp4", "name": "matrix.mp4",
"mimetype": "video/mp4", "mimetype": "video/mp4",
"size": 12345 "size": 12345
}, },
"m.thumbnail": [ "m.video_details": { // optional
{ "width": 640,
// an inline m.file, minus `name` "height": 480,
"url": "mxc://example.org/efgh5678", "duration": 90
"mimetype": "image/jpeg", },
"size": 123, "m.thumbnail": [ // optional
{
// an inline m.image // A thumbnail is an m.file+m.image, or a small image
"width": 160, "m.file": {
"height": 120 "url": "mxc://exmaple.org/efgh5678",
"mimetype": "image/jpeg",
"size": 123
// "name" is optional in this scenario
}, },
// ... "m.image_details": {
], "width": 160,
"m.caption": [ "height": 120
// array of m.message objects }
{ "m.text": "matrix demo" }, },
{ "body": "<b>matrix</b> demo", "mimetype": "text/html" } // ...
], ],
"m.video": { "m.caption": { // optional - goes above/below video
"width": 1280, "m.markup": [{"body": "Look at this cool animated Matrix logo"}]
"height": 720,
"duration": 90000 // milliseconds
} }
} }
} }
``` ```
All details shown above except `m.video` in the content are inherited from [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552). The newly introduced blocks are:
`m.video` is simply a container for the video-specific metadata that is not already covered by other MSCs.
Note that the event does not have an `m.image` - clients should use the first image-like thumbnail as a poster
for the video. It is reasonable for a client to include a lower quality copy (eg: 480p) as a thumbnail in the
event.
Note that videos can be encrypted using the same approach as `m.file`. * `m.video_details` - Similar to `m.image_details` from MSC3552, optional information about the video.
`width` and `height` are required, while `duration` (length in seconds of the video) is optional.
## Potential issues Together with content blocks from other proposals, an `m.video` is described as:
The schema duplicates some of the information into the text fallback, though this is unavoidable. * **Required** - An `m.markup` block to act as a fallback for clients which can't process videos.
* **Required** - An `m.file` block to contain the video itself. Clients use this to show the video.
* **Optional** - An `m.video_details` block to describe any video-specific metadata, such as dimensions.
Like with existing `m.room.message` events today, clients should keep videos within a set of
reasonable bounds, regardless of sender-supplied values. For example, keeping videos at a minimum
size and within a maximum size.
* **Optional** - An `m.thumbnail` block (array) to describe "poster images" for the video.
* **Optional** - An `m.caption` block to represent any text that should be shown above or below the
video. Currently this MSC does not describe a way to pick whether the text goes above or below,
leaving this as an implementation detail. A future MSC may investigate ways of representing this,
if needed.
## Alternatives The above describes the minimum requirements for sending an `m.video` event. Senders can add additional
blocks, however as per the extensible events system, receivers which understand video events should not
honour them. Such examples might include an `m.audio` block for "audio-only" mode (podcasts, etc) or
an `m.image` to represent the video as a GIF (or similar).
No significant alternatives known. Note that `m.file` supports encryption and therefore it's possible to encrypt thumbnails and videos
too.
## Security considerations If a client does not support rendering videos inline, the client would instead typically represent
the event as a plain file upload, then fall further back to a plain text message. An image fallback
is not neccessarily possible, despite all the required blocks being possible. This is due to the file
having a video mimetype, hopefully indicating to the client that an `<img />` (or similar) is not
appropriate for this event.
The same considerations which currently apply to files, videos, and extensible events also apply here. ## Potential issues
## Transition The schema duplicates some of the information into the text fallback, though this is unavoidable
and intentional for fallback considerations.
The same transition introduced by extensible events is also applied here: ## Alternatives
```json5 No significant alternatives known.
{
"type": "m.room.message",
"content": {
"body": "matrix.mp4",
"msgtype": "m.video",
"url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35",
"info": {
"duration": 90000,
"mimetype": "video/mp4",
"size": 12345,
"w": 1280,
"h": 720,
"thumbnail_url": "mxc://example.org/elsewhere",
"thumbnail_info": {
"size": 123,
"mimetype": "image/jpeg",
"w": 160,
"h": 120
}
},
// Extensible Events ## Security considerations
"m.text": "matrix.mp4", // or other m.message-like event
"m.file": {
"url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35",
"name": "matrix.mp4",
"mimetype": "video/mp4",
"size": 12345
},
"m.thumbnail": [
{
// an inline m.file, minus `name`
"url": "mxc://example.org/elsewhere",
"mimetype": "image/jpeg",
"size": 123,
// an inline m.image
"width": 160,
"height": 120
},
// ...
],
"m.caption": [
// array of m.message objects
{ "m.text": "matrix demo" },
{ "body": "<b>matrix</b> demo", "mimetype": "text/html" }
],
"m.video": {
"width": 1280,
"height": 720,
"duration": 90000 // milliseconds
}
}
}
```
The event details are copied and quite verbose, however this is best to ensure compatibility with the The same considerations which currently apply to files, videos, and extensible events also
extensible events format. apply here. For example, bounds on video size, assuming sender-provided details about the file are
false, etc.
## Unstable prefix ## Unstable prefix
@ -146,41 +115,4 @@ While this MSC is not considered stable, implementations should use `org.matrix.
place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than
the namespace of this MSC - this is deliberate. the namespace of this MSC - this is deliberate.
Example: Note that extensible events should only be used in an appropriate room version as well.
```json5
{
"type": "org.matrix.msc1767.video",
"content": {
"org.matrix.msc1767.text": "matrix.mp4", // or other m.message-like event
"org.matrix.msc1767.file": {
"url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35",
"name": "matrix.mp4",
"mimetype": "video/mp4",
"size": 12345
},
"org.matrix.msc1767.thumbnail": [
{
// an inline m.file, minus `name`
"url": "mxc://example.org/elsewhere",
"mimetype": "image/jpeg",
"size": 123,
// an inline m.image
"width": 160,
"height": 120
},
// ...
],
"org.matrix.msc1767.caption": [
// array of m.message objects
{ "m.text": "matrix demo" },
{ "body": "<b>matrix</b> demo", "mimetype": "text/html" }
],
"org.matrix.msc1767.video": {
"width": 1280,
"height": 720,
"duration": 90000 // milliseconds
}
}
}
```

Loading…
Cancel
Save