Merge b5ca351dbd
into ecf996389f
commit
e0132ee46f
@ -0,0 +1,269 @@
|
||||
# MSC2545: Image Packs (Emoticons & Stickers)
|
||||
|
||||
Emoticons, or short emotes...we need them!
|
||||
|
||||
We also need proper stickers, too!
|
||||
|
||||
## Terminology
|
||||
### Emoticons
|
||||
Since there is a lot of confusion of how this relates to `m.emote`, why this isn't called "custom emoji"
|
||||
etc,:
|
||||
|
||||
`m.emote` is for emotion - and it has been incorrectly named this way. `m.action` would have been more
|
||||
appropriate, as you use it to describe *actions*, not *emotions*. E.g. "/me is walking to the gym", or
|
||||
"/me is happy" and *not* "/me happy".
|
||||
|
||||
That, however, is *not* what this MSC is about. Instead, it is about emoticons, also known in short as
|
||||
emotes.
|
||||
|
||||
Emoticons are just little images or text describing emotions or other things. Emoji are a subset of
|
||||
emoticons, namely those found within unicode. Custom emoji here would actually refer to a custom emoji
|
||||
font, that is your own rendering of 🦊, 🐱, etc., *not* new images. New images is what custom emoticons
|
||||
are for.
|
||||
|
||||
Now, a client may choose to name these however they like. We already have a naming disparity between
|
||||
spec and clients with groups vs communities. It is, however, imperative to name things in the spec
|
||||
accurately after what they are.
|
||||
|
||||
### Stickers
|
||||
Stickers already exist in Matrix. They are reusable images one can send, usually as a reaction to
|
||||
something sent in the timeline. This MSC adds a way to distribute and define a source for a client to
|
||||
send them.
|
||||
|
||||
## Proposal
|
||||
### Emoticons in the formatted body
|
||||
Emoticons have at least a shortcode and a mxc uri. They are sent as `<img>` tags, which are currently in
|
||||
the spec. As such, many existing clients are able to render them.
|
||||
To allow clients to distinguish emoticons from other inline images, a new
|
||||
property, `data-mx-emoticon`, is introduced. A client can choose to ignore the size attributes of emoticons
|
||||
when rendering, and instead pick the size based on other circumstances. This could e.g. be used to
|
||||
display messages with only emoticons and emoji as larger than usual, which is commonly found in
|
||||
messengers. Such an `<img>` tag of a shortcode `emote` and a mxc uri `mxc://example.org/emote`
|
||||
could look as follows:
|
||||
|
||||
```html
|
||||
<img data-mx-emoticon src="mxc://example.org/emote" alt=":emote:" title=":emote:" height="32" />
|
||||
```
|
||||
|
||||
Both the `alt` and the `title` attributes are specified as they serve different purposes: `alt` is
|
||||
displayed if e.g. the emote does not load. `title` is displayed e.g. on mouse hover over the emote.
|
||||
Thus, specifying both the `alt` and `title` attributes is required.
|
||||
|
||||
The `height` is just a height that looks good on most devices with the normal, default font size.
|
||||
No width is displayed as to not weirdly squish non-square emotes. In order to maintain backwards-compatibility
|
||||
with clients not supporting emotes, specifying the `height` is required.
|
||||
|
||||
If the new `data-mx-emoticon` attribute has a value, it is ignored. Due to limitations of some libraries
|
||||
the attribute may even look like `data-mx-emoticon=""`.
|
||||
|
||||
The `src` attribute *must* be a mxc url. Other URIs, such as `https`, `mailto` etc. are not allowed.
|
||||
|
||||
### Sending stickers
|
||||
To send stickers, the already spec'd `m.sticker` is used.
|
||||
|
||||
### Image types
|
||||
Emoticons are recommended to have a size of about 128x128 pixels. Even though the fallback specifies
|
||||
a height of 32, this is to ensure that the emotes still look good on higher DPI screens.
|
||||
|
||||
Stickers are recommended to have a size of up to 512x512 pixels.
|
||||
|
||||
Furthermore, these images should either have a mimetype of `image/png`, `image/gif` or `image/webp`.
|
||||
They can be animated.
|
||||
|
||||
Due to the low resolution of emotes, `image/jpg` and `image/jpeg` have been purposefully excluded from this
|
||||
list.
|
||||
|
||||
### Image pack event
|
||||
The image pack event has a type of `m.image_pack`. It contains a key `images`, which is a map from a
|
||||
short code to an image object. It may also contain a key `pack`, which is a pack object, described in
|
||||
the following section.
|
||||
|
||||
#### Pack object
|
||||
The pack object consists of the following keys:
|
||||
- `display_name`: (String, optional) A display name for the pack. Defaults to the room name, if the
|
||||
image pack event is in the room. This does not have to be unique from other packs in a room.
|
||||
- `avatar_url`: (String, optional) The mxc uri of an avatar/icon to display for the pack. Defaults
|
||||
to the room avatar, if the pack is in the room. Otherwise, the pack does not have an avatar.
|
||||
- `usage`: (String[], optional) An array of the usages for this pack. Possible usages are `"emoticon"`
|
||||
and `"sticker"`. If the usage is absent or empty, a usage for all possible usage types is to be assumed.
|
||||
- `attribution`: (String, optional) The attribution of this pack.
|
||||
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"display_name": "Awesome Pack",
|
||||
"usage": ["emoticon"]
|
||||
}
|
||||
```
|
||||
|
||||
#### Image object
|
||||
The image object consists of the following keys:
|
||||
- `url`: (String, required) The mxc URL for this image.
|
||||
- `body`: (String, optional) An optional text body for this image. Useful for the sticker body text or
|
||||
the emote alt text. Defaults to the shortcode.
|
||||
- `info`: (`ImageInfo`, optional) The already spec'd `ImageInfo` object used for the `info` block of
|
||||
`m.sticker` events.
|
||||
- `usage`: (String[], optional) An array of the usages for this image. The possible values match those
|
||||
of the `usage` key of a pack object. If present and non-empty, this overrides the usage defined at
|
||||
pack level for this particular image. This is useful to e.g. have one pack contain mixed emotes and
|
||||
stickers. Additionally, as there is only a single account data level image pack, this is required to
|
||||
have a mixture of emotes and stickers available in account data.
|
||||
|
||||
#### Example image pack event
|
||||
Taking all of this into account, an example pack event may look like the following:
|
||||
```json
|
||||
{
|
||||
"images": {
|
||||
"myemote": {
|
||||
"url": "mxc://example.org/blah"
|
||||
},
|
||||
"mysticker": {
|
||||
"url": "mxc://example.org/sticker",
|
||||
"usage": ["sticker"]
|
||||
}
|
||||
},
|
||||
"pack": {
|
||||
"display_name": "Awesome Pack",
|
||||
"usage": ["emoticon"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Image sources
|
||||
There are several places where a client is expected to look for events with type `m.image_pack`, mainly
|
||||
in their own account data and in room state.
|
||||
|
||||
#### User image packs
|
||||
Each user can have their own personal image pack defined in their account data, under the
|
||||
`m.image_pack` key. The user is expected to be presented with these images in all rooms.
|
||||
|
||||
#### Room image packs
|
||||
A room can have an unlimited amount of image packs, by specifying the `m.image_pack` state event with
|
||||
different state keys. By default, the user is expected to be presented with these images only in the room they
|
||||
they are defined in. To enable them to be presented in all rooms, see the "Image pack rooms" section.
|
||||
E.g. a discord bridge could set as state key
|
||||
`de.sorunome.mx-puppet-bridge.discord` and have all the bridged emotes in said state event, keeping
|
||||
bridged emotes from matrix emotes separate.
|
||||
|
||||
#### Space image packs
|
||||
Clients should suggest image packs of a room's canonical space, if the user is also in that space.
|
||||
This should be done recursively on canonical spaces. So, if a room has a canonical space and
|
||||
that space again has a canonical space, the clients should suggest image packs of both of those spaces.
|
||||
|
||||
#### Image pack rooms
|
||||
While room image packs are specific to a room, they can be made accessible from anywhere by setting
|
||||
the `m.image_pack.rooms` key in a user's account data. The value is an object, with a `room` key containing
|
||||
a map of room ids to the state keys to an object. If a room id / state key combination is provided in this form,
|
||||
clients should make the corresponding room image pack globally accessible in all rooms.
|
||||
|
||||
Note that while this MSC does not define any keys for the bottom-level object, defining it as an object means greater
|
||||
flexibility in the case of future changes.
|
||||
|
||||
The contents of `m.image_pack.rooms` could look like the following:
|
||||
|
||||
```json
|
||||
{
|
||||
"rooms": {
|
||||
"!someroom:example.org": {
|
||||
"": {},
|
||||
"de.sorunome.mx-puppet-bridge.discord": {}
|
||||
},
|
||||
"!someotherroom:example.org": {
|
||||
"": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Here three emote packs are globally accessible to the user: Two defined in `!someroom:example.org`
|
||||
(one with blank state key and one with state key `de.sorunome.mx-puppet-bridge.discord`) and one in
|
||||
`!someotherroom:example.org` (with a blank state key).
|
||||
|
||||
### Image pack source priority and deduplicating
|
||||
If a client gives image suggestions (emotes, stickers) to the user in some ordered fashion (e.g. an
|
||||
ordered list where you click an entry), the order of the images should be predictable between rooms.
|
||||
A suggestion for clients of image pack ordering is as follows:
|
||||
1. User image pack (defined in your own account data)
|
||||
2. Image pack rooms (defined in the `m.image_pack.rooms` account data object)
|
||||
3. Space image packs (defined in the hierarchy of canonical spaces for the current room)
|
||||
4. Room image packs (defined in the currently open room's state)
|
||||
|
||||
Furthermore, clients are expected to de-duplicate images so that same images are not displayed multiple
|
||||
times. Such scenarios include:
|
||||
1. when viewing a room that has a pack defined in the `m.image_pack.rooms` account data object, and
|
||||
2. a bot which syncs emotes over multiple rooms.
|
||||
|
||||
This could be done by deduplicating by the mxc URI.
|
||||
|
||||
### Sending
|
||||
#### Emoticons
|
||||
For emoticons a client could add delimiters (e.g. `:`) around the image shortcode, meaning
|
||||
that if an image has a shortcode of `emote`, the user can enter `:emote:` to send it. If there are
|
||||
multiple emoticons with the same shortcode in a room, the client could e.g. slugify the packs display
|
||||
name and then have the user enter `:slug~emote:`. As slugs typically match `^[\w-]+$`, that should
|
||||
ensure complete-ability.
|
||||
|
||||
The alt / title text for the `<img>` tag is expected to be the `body` of the emote. If absent, its
|
||||
shortcode should be used instead.
|
||||
|
||||
#### Stickers
|
||||
When sending a sticker, the `body` of the `m.sticker` event should be set to the `body` defined for that
|
||||
image, or if absent, its shortcode.
|
||||
|
||||
The `info` object of the `m.sticker` event should be set to the `info` object of the image, or if absent,
|
||||
an empty object.
|
||||
|
||||
## Security Considerations
|
||||
When sending an `<img>` tag in an encrypted room, the client will make a request to fetch
|
||||
said image, in this case an emote. As there is no way to encrypt content behind `<img>` tags yet,
|
||||
this could potentially leak part of the conversation. This is **not** a new security consideration,
|
||||
it already exists. This, however, isn't much different from posting someone a link in an e2ee chat and
|
||||
the recipient opens the link. Additionally, images, and thus emotes, are often cached by the client,
|
||||
not even necessarily leading to a http query.
|
||||
|
||||
Related issue: https://github.com/matrix-org/matrix-doc/issues/2418
|
||||
|
||||
## Unstable prefix
|
||||
The `m.image_pack` in the account data is replaced with `im.ponies.user_emotes`. The `m.image_pack` in
|
||||
the room state is replaced with `im.ponies.room_emotes`. The `m.image_pack.rooms` is replaced with
|
||||
`im.ponies.emote_rooms`.
|
||||
|
||||
Some existing implementations using `im.ponies.user_emotes` and `im.ponies.room_emotes` currently use
|
||||
a dict called `short` which is just a map of the shortcode to the mxc url.
|
||||
|
||||
Some other implementations currently also call the `images` key `emoticons`.
|
||||
|
||||
## Alternatives
|
||||
One can easily think of near infinite ways to implement emotes. The aspect of sending an `<img>` tag
|
||||
and marking it as an emote with `data-mx-emoticon` seems to be pretty universal though, so the question
|
||||
is mainly on different emote sources. For that, a separate MSC, [MSC1951](https://github.com/matrix-org/matrix-doc/pull/1951)
|
||||
already exists, so it is reasonable to compare this one with that one:
|
||||
|
||||
### Comparison with MSC1951
|
||||
MSC1951 defines a dedicated room as the only image pack source. This MSC, however, also allows you to bind image packs
|
||||
to your own account, offering greater flexibility. In MSC1951 there can also only be a single image pack
|
||||
in a room. This could be problematic in e.g. bridged rooms: You set some emotes or stickers from the matrix
|
||||
side and a discord bridge would plop all the discord emotes and stickers in another pack in the same room.
|
||||
|
||||
The original sharing-focused idea of MSC1951 is still preserved: Once room types are a thing, you could
|
||||
still easily have an image pack-only room.
|
||||
|
||||
MSC1951 defines a way to recommend using a pack of a different room - this MSC does not have an equivalent
|
||||
to that. Instead, this MSC allows multiple image packs for a room, typically one where you already
|
||||
chat in anyways. Furthermore, it allows you to enable an image pack to be globally available for yourself
|
||||
across all rooms you are in.
|
||||
|
||||
The core difference is how MSC1951 and this MSC define the image packs themselves: In MSC1951 you have to
|
||||
set one state event *per image*. While this might seem like a nice idea on the surface, it doesn't
|
||||
scale well. There are people who easily use and want hundreds or even thousands of image packs accessible.
|
||||
A simple dict of shortcode to mxc URI seems more appropriate for this.
|
||||
|
||||
In general, MSC1951 feels like a heavier approach to image pack sources, while this MSC is more lightweight.
|
||||
|
||||
## Looking further
|
||||
A couple of interesting points have been raised in discussions of this MSC that tangentially touch
|
||||
custom emoticons. Each warrant an MSC for themselves however, as they touch more on how `<img>` is works.
|
||||
- Figuring out how `<img>` should work with encrypted media.
|
||||
- Allow SVGs in the `<img>` tag. Current problem: Clients typically try to thumbnail the mxc URL,
|
||||
and most media repositories can't thumbnail SVGs. Possible solution: Somehow embed the mimetype.
|
||||
- For stickers: Recommend rendering sizes / resolutions
|
Loading…
Reference in New Issue