see images in mastodon as image not as link

[albjeremias]

For front end issues, use lemmy-ui

Is your proposal related to a problem?

Yeah, i'd love to see images in mastodon

Describe the solution you'd like

When I follow a community from mastodon I'd like that when people make a post from lemmy in a community with a picture, the picture is rendered in the frontend of mastodon. not a link to the image.

Describe alternatives you've considered

.

Additional context

https://docs.joinmastodon.org/spec/activitypub/#PropertyValue
dariusk/rss-to-activitypub#16

According to the documentation above, seems simple, could be something like:

"attachment": [{
       "type": "Document",
       "mediaType": "image/jpeg",
       "url": "https://img.pawoo.net/media_attachments/files/013/041/272/original/99ce3e980221e458.jpeg",
       "name": null,
       "focalPoint": [0.01, 0.6]
   }, 

[dessalines]

@Nutomic this wouldn't be too bad right? Its just checking for attachments in the incoming Comment Note, checking if they're an image type, and then appending them as markdown images to the comment text before inserting them into the DB.

[albjeremias]

@Nutomic this wouldn't be too bad right? Its just checking for attachments in the incoming Comment Note, checking if they're an image type, and then appending them as markdown images to the comment text before inserting them into the DB.

you dont need to do any change on the db.. is only on the activitypub api node, which the json payload needs to be more readable for other activitypub consumers :)

[Nutomic]

At the moment we send attachments like this:

  "attachment": [
    {
      "type": "Link",
      "href": "https://enterprise.lemmy.ml/pictrs/image/eOtYb9iEiB.png"
    }
  ],

Im not sure if Mastodon requires the document type specifically, but I guess its enough to add the mediaType. The reason why its missing is because Lemmy doesnt store that in the database. We could make a HEAD request to determine the url media type every time the post is serialized, but that could be very inefficient. So it should really be added to the db.

[albjeremias]

At the moment we send attachments like this:

  "attachment": [
    {
      "type": "Link",
      "href": "https://enterprise.lemmy.ml/pictrs/image/eOtYb9iEiB.png"
    }
  ],

Im not sure if Mastodon requires the document type specifically, but I guess its enough to add the mediaType. The reason why its missing is because Lemmy doesnt store that in the database. We could make a HEAD request to determine the url media type every time the post is serialized, but that could be very inefficient. So it should really be added to the db.

got it! :) I'd like if you show me those PRs or PRs where i can get some inspiration.. i will browse a bit anyway

[Nutomic]

The first thing you need to do is create a new sql migration in migrations folder which adds a column post.link_media_type, should be a nullable text. Then add that column to schema.rs and post.rs. The compiler will tell you where things need to be adjusted, but mainly you need to write that new field whenever a post with link is created or updated. For federation you need to add the media type here. Or maybe we can send as type Document instead of Link, i will check that later.

[dessalines]

This could be an opportunity to talk about removing the post.link field, and having multiple post links, but have them as attachments with content types also.

[albjeremias]

This could be an opportunity to talk about removing the post.link field, and having multiple post links, but have them as attachments with content types also.

i prefer to do two mini PRs.. i like to keep PRs as small as possible

[db0]

+1 for this. I have a bot which generates images via mastodon. They look like this on mastodon https://sigmoid.social/@stablehorde_generator/110542776904219863 but when used as comments on lemmy, they don't appear at all https://lemmy.dbzer0.com/comment/48168

[wont-work]

this (?) is a part of the reason why federation with misskey and it's forks are broken. misskey expects a type:Image* with a url field, but lemmy sends a type:Link with a href field instead

*: misskey actually has another issue where it sends type:Link attachments into it's image code, which means it tries to parse non-image links as images, completely breaking post fetching (misskey-dev/misskey#12243). that's not lemmy's fault but worth mentioning as if it weren't for this then posts would still be fetched, just with images appearing (presumably) as direct links.

(that said lemmy should still try to federate images as images in my opinion)

[Nutomic]

@ShittyKopper Thats good to know, thanks for pointing it out. I added a commit to #4035 to federate images as Activitypub Image type instead of Link.

[wont-work]

Thats good to know, thanks for pointing it out. I added a commit to #4035 to federate images as Activitypub Image type instead of Link.

If you need any testing I have a local branch of Sharkey that works around the Misskey side of the problem (not submitted upstream because I'm not sure if it breaks anything else) and can try to fetch posts. Non-image link posts will still be unfetchable until Misskey (and hardforks like Firefish) fix the type:Link issue but there isn't anything Lemmy can do there (And text-only posts work just fine)

(ps: It may make sense to federate type:Video and perhaps type:Audio as well, the underlying issue really is the href vs url difference)

[Nutomic]

Lemmy cant currently receive attachments with type Video or Audio so that would cause problems with backwards compatibility. For now the PR needs to be merged first, after that it can be tested.