Support content blocks system

travis/msc/extev/video
Travis Ralston 3 years ago
parent 39d7395590
commit d3f107f7d0

@ -1,19 +1,19 @@
# MSC3553: Extensible Events - Videos
[MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail,
though deliberately does not include schemas for non-text messaging types. This MSC covers only videos.
though deliberately does not include schemas for some messaging types. This MSC covers only videos.
*Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in
stages without blocking other pieces of the overall idea. For example, an issue with the way images
stages without blocking other pieces of the overall idea. For example, an issue with the way videos
are represented should not block the overall schema from going through.
This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and
parts of [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552) for thumbnails.
[MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552).
## Proposal
Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` primary
event type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo).
Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new `m.video` event
type is introduced to replace the [`m.video` `msgtype`](https://spec.matrix.org/v1.1/client-server-api/#mvideo).
An example is:
@ -21,124 +21,93 @@ An example is:
{
"type": "m.video",
"content": {
"m.text": "Upload: matrix.mp4 (12 KB)", // or other m.message-like event
"m.markup": [
// Format of the fallback is not defined, but should have enough information for a text-only
// client to do something with the video, just like with plain file uploads.
{"body": "matrix.mp4 (12 KB, 1:30) https://example.org/_matrix/media/v3/download/example.org/abcd1234"}
],
"m.file": {
"url": "mxc://example.org/abcd1234",
"name": "matrix.mp4",
"mimetype": "video/mp4",
"size": 12345
},
"m.thumbnail": [
"m.video_details": { // optional
"width": 640,
"height": 480,
"duration": 90
},
"m.thumbnail": [ // optional
{
// an inline m.file, minus `name`
"url": "mxc://example.org/efgh5678",
// A thumbnail is an m.file+m.image, or a small image
"m.file": {
"url": "mxc://exmaple.org/efgh5678",
"mimetype": "image/jpeg",
"size": 123,
"size": 123
// an inline m.image
// "name" is optional in this scenario
},
"m.image_details": {
"width": 160,
"height": 120
}
},
// ...
],
"m.caption": [
// array of m.message objects
{ "m.text": "matrix demo" },
{ "body": "<b>matrix</b> demo", "mimetype": "text/html" }
],
"m.video": {
"width": 1280,
"height": 720,
"duration": 90000 // milliseconds
"m.caption": { // optional - goes above/below video
"m.markup": [{"body": "Look at this cool animated Matrix logo"}]
}
}
}
```
All details shown above except `m.video` in the content are inherited from [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551) and [MSC3552](https://github.com/matrix-org/matrix-doc/pull/3552).
The newly introduced blocks are:
`m.video` is simply a container for the video-specific metadata that is not already covered by other MSCs.
Note that the event does not have an `m.image` - clients should use the first image-like thumbnail as a poster
for the video. It is reasonable for a client to include a lower quality copy (eg: 480p) as a thumbnail in the
event.
* `m.video_details` - Similar to `m.image_details` from MSC3552, optional information about the video.
`width` and `height` are required, while `duration` (length in seconds of the video) is optional.
Note that videos can be encrypted using the same approach as `m.file`.
Together with content blocks from other proposals, an `m.video` is described as:
## Potential issues
* **Required** - An `m.markup` block to act as a fallback for clients which can't process videos.
* **Required** - An `m.file` block to contain the video itself. Clients use this to show the video.
* **Optional** - An `m.video_details` block to describe any video-specific metadata, such as dimensions.
Like with existing `m.room.message` events today, clients should keep videos within a set of
reasonable bounds, regardless of sender-supplied values. For example, keeping videos at a minimum
size and within a maximum size.
* **Optional** - An `m.thumbnail` block (array) to describe "poster images" for the video.
* **Optional** - An `m.caption` block to represent any text that should be shown above or below the
video. Currently this MSC does not describe a way to pick whether the text goes above or below,
leaving this as an implementation detail. A future MSC may investigate ways of representing this,
if needed.
The schema duplicates some of the information into the text fallback, though this is unavoidable.
The above describes the minimum requirements for sending an `m.video` event. Senders can add additional
blocks, however as per the extensible events system, receivers which understand video events should not
honour them. Such examples might include an `m.audio` block for "audio-only" mode (podcasts, etc) or
an `m.image` to represent the video as a GIF (or similar).
## Alternatives
Note that `m.file` supports encryption and therefore it's possible to encrypt thumbnails and videos
too.
No significant alternatives known.
## Security considerations
The same considerations which currently apply to files, videos, and extensible events also apply here.
If a client does not support rendering videos inline, the client would instead typically represent
the event as a plain file upload, then fall further back to a plain text message. An image fallback
is not neccessarily possible, despite all the required blocks being possible. This is due to the file
having a video mimetype, hopefully indicating to the client that an `<img />` (or similar) is not
appropriate for this event.
## Transition
## Potential issues
The same transition introduced by extensible events is also applied here:
The schema duplicates some of the information into the text fallback, though this is unavoidable
and intentional for fallback considerations.
```json5
{
"type": "m.room.message",
"content": {
"body": "matrix.mp4",
"msgtype": "m.video",
"url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35",
"info": {
"duration": 90000,
"mimetype": "video/mp4",
"size": 12345,
"w": 1280,
"h": 720,
"thumbnail_url": "mxc://example.org/elsewhere",
"thumbnail_info": {
"size": 123,
"mimetype": "image/jpeg",
"w": 160,
"h": 120
}
},
## Alternatives
// Extensible Events
"m.text": "matrix.mp4", // or other m.message-like event
"m.file": {
"url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35",
"name": "matrix.mp4",
"mimetype": "video/mp4",
"size": 12345
},
"m.thumbnail": [
{
// an inline m.file, minus `name`
"url": "mxc://example.org/elsewhere",
"mimetype": "image/jpeg",
"size": 123,
No significant alternatives known.
// an inline m.image
"width": 160,
"height": 120
},
// ...
],
"m.caption": [
// array of m.message objects
{ "m.text": "matrix demo" },
{ "body": "<b>matrix</b> demo", "mimetype": "text/html" }
],
"m.video": {
"width": 1280,
"height": 720,
"duration": 90000 // milliseconds
}
}
}
```
## Security considerations
The event details are copied and quite verbose, however this is best to ensure compatibility with the
extensible events format.
The same considerations which currently apply to files, videos, and extensible events also
apply here. For example, bounds on video size, assuming sender-provided details about the file are
false, etc.
## Unstable prefix
@ -146,41 +115,4 @@ While this MSC is not considered stable, implementations should use `org.matrix.
place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than
the namespace of this MSC - this is deliberate.
Example:
```json5
{
"type": "org.matrix.msc1767.video",
"content": {
"org.matrix.msc1767.text": "matrix.mp4", // or other m.message-like event
"org.matrix.msc1767.file": {
"url": "mxc://example.org/9af6bae1ae9cacc93058ae386028c52f28e41d35",
"name": "matrix.mp4",
"mimetype": "video/mp4",
"size": 12345
},
"org.matrix.msc1767.thumbnail": [
{
// an inline m.file, minus `name`
"url": "mxc://example.org/elsewhere",
"mimetype": "image/jpeg",
"size": 123,
// an inline m.image
"width": 160,
"height": 120
},
// ...
],
"org.matrix.msc1767.caption": [
// array of m.message objects
{ "m.text": "matrix demo" },
{ "body": "<b>matrix</b> demo", "mimetype": "text/html" }
],
"org.matrix.msc1767.video": {
"width": 1280,
"height": 720,
"duration": 90000 // milliseconds
}
}
}
```
Note that extensible events should only be used in an appropriate room version as well.

Loading…
Cancel
Save