Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Link previews #327

Closed
felicio opened this issue Jan 9, 2023 · 37 comments
Closed

Link previews #327

felicio opened this issue Jan 9, 2023 · 37 comments

Comments

@felicio
Copy link
Collaborator Author

felicio commented Jan 9, 2023

My notes

  • Issue
  • Notes
  • Questions
  • Tables
    • Fields
    • Content opics
    • Encryption keys
    • Protobuffer
    • Update frequency
  • Proposals
  • Mentions/pocs
  • Branch/poc
    • Image, base64
    • Keys, base58btc
    • Url schema
      • Same as deep/app url
      • #
      • Servers
      • Hosts
      • IM
      • Browser
      • Update spec
    • Emoji hash
    • Clock
    • Cache
    • Loading
    • Error
      • Connection
      • Not found
    • QR
  • Follow-ups
    • Internal (in-app) native and interactive previews

@felicio
Copy link
Collaborator Author

felicio commented Jan 12, 2023

Community

COMMUNITY_DESCRIPTION
Banner Yes
Photo Yes
Name Yes
Description Yes
Members count Yes (but computed)
Active users No
App link No (but computed)
Tags Yes (but computed)
Color Yes

Channel

COMMUNITY_DESCRIPTION CHAT_IDENTITY
Banner No
Photo No
Emoji Yes
Name Yes
Description Yes
App link No (but computed)
Color Yes
Community photo Yes
Community name Yes

User

CONTACT_CODE_ADVERTISEMENT
Banner No
Photo Yes
Name Yes
Description Yes
Emoji hash No (but computed)
Identicon No (but computed)
Social links Yes
App link No (but computed)
Color No

Group

MEMBERSHIP_UPDATE_MESSAGE
Banner No
Photo
Emoji
Name
Members count
App link No (but computed)
Color

@felicio
Copy link
Collaborator Author

felicio commented Jan 12, 2023

@endulab @cammellos could you please confirm that we're currently not publicly broadcasting group chat details (i.e. photo, emoji, name, members count, color), and if so help me create and estimate issues for related repos?

Cc @pedro-et (for sanity check on the required fields)

@cammellos
Copy link

@felicio @pedro-et @John-44 we had conversations about this already, and we haven't settled on a decision, private group chat as it is now should not leak any metadata. We agreed that this is something that we would be discussing for post mvp.

@endulab
Copy link

endulab commented Jan 12, 2023

@felicio we will handle only 1. community, 2. channel and 3. user links.

@John-44
Copy link

John-44 commented Jan 12, 2023

@felicio @pedro-et @John-44 we had conversations about this already, and we haven't settled on a decision, private group chat as it is now should not leak any metadata. We agreed that this is something that we would be discussing for post mvp.

Sharing links to ad-hoc group chats (with group chat name, image, etc...) is out of scope for MVP, but highly likely to be something we will do shortly after MVP (while preserving the current fully private group chat functionality of course - users would effectively have the choice of converting a private ad-hoc group chat into a publicly sharable ad-hoc group chat if they wanted to). The design for such post-MVP 'sharable ad-hoc group chat' functionality is tbd.

Post MVP we will also be introducing many more types of URL links (e.g. links to request money including a message, links to individual messages in communities or 'open' ad-hoc group chats, etc..). But we don't need to worry about this for now, because we have more than enough items that are in-scope for MVP that we need to direct our energies towards.

1 similar comment
@John-44
Copy link

John-44 commented Jan 12, 2023

@felicio @pedro-et @John-44 we had conversations about this already, and we haven't settled on a decision, private group chat as it is now should not leak any metadata. We agreed that this is something that we would be discussing for post mvp.

Sharing links to ad-hoc group chats (with group chat name, image, etc...) is out of scope for MVP, but highly likely to be something we will do shortly after MVP (while preserving the current fully private group chat functionality of course - users would effectively have the choice of converting a private ad-hoc group chat into a publicly sharable ad-hoc group chat if they wanted to). The design for such post-MVP 'sharable ad-hoc group chat' functionality is tbd.

Post MVP we will also be introducing many more types of URL links (e.g. links to request money including a message, links to individual messages in communities or 'open' ad-hoc group chats, etc..). But we don't need to worry about this for now, because we have more than enough items that are in-scope for MVP that we need to direct our energies towards.

@felicio
Copy link
Collaborator Author

felicio commented Jan 13, 2023

@felicio we will handle only 1. community, 2. channel and 3. user links.

@endulab please,

  • Do you have an issue open in which you're addressing this feature on your end too?
  • Do you have designs for displaying the previews within app? For example, after clicking the deep link outside the app (e.g. user not in contacts) or sharing the link in chat (e.g. not yet joined community).
  • Did you consider dropping the status-im:// URL scheme and supporting only universal links (see https://developer.apple.com/documentation/xcode/defining-a-custom-url-scheme-for-your-app)?
  • Do you also require community's public key to retrieve channel details?

@endulab
Copy link

endulab commented Jan 13, 2023

@felicio

  • There is the epic issue here: Deep Links Epic status-desktop#7840
  • There are no strict designs: 1. /u/ opens user profile dialog 2. /c/ opens community or if you are not a member it allows you to spectate. You can send request to join then. /cc/ does the same as /c/ but opens channel additionally.
  • Which one ? Can you elaborate ? Installation package already registers custom link "status-im". I think this is a common practice, eg. discord is registered as discord://. FYI, we want to use status-app instead of status-im (Use https:// deep links instead of status-im:// - MacOS status-desktop#7957)
  • Hmm, right now we do not expose links for channels. The link requires only channel id which contains also community id. In this case we do not need community key because we get it from channel key. @xAlisher how do we want to expose channel link in the application ? Also, we need to think about how the exposed key should/may look like: compressed ? Or maybe consist of both communityId and channelId ? FYI @cammellos

@felicio
Copy link
Collaborator Author

felicio commented Jan 13, 2023

@endulab thank you, and to follow up

Can you elaborate ? Installation package already registers custom link "status-im".

Let me try to rephrase then. Will, or can, desktop open and handle links starting with https:// like mobile does? Thus rendering status-im:// scheme obsolete for cases like "Open in Status" (see status-im/status-desktop#6612) and redirecting users straight to the app if installed, and landing on the website only if not.

which contains also community id

Which technically is the community's compressed public key later then used for content topic and symmetric key, correct? If not, could you make a reference to the code?

@endulab
Copy link

endulab commented Jan 16, 2023

@endulab thank you, and to follow up

Can you elaborate ? Installation package already registers custom link "status-im".

Let me try to rephrase then. Will, or can, desktop open and handle links starting with https:// like mobile does? Thus rendering status-im:// scheme obsolete for cases like "Open in Status" (see status-im/status-desktop#6612) and redirecting users straight to the app if installed, and landing on the website only if not.

Oh, I see. The app can't do it right now, but we can try to do it this way. However, I understand that if we have universal links, the app will be opened directly w/o need of our website. Is it the workflow we want to have ? I will have to do some research how to do this. Also, we need this functionality on Windows. Let me know if we need this feature; I will create the ticket then.

which contains also community id

Which technically is the community's compressed public key later then used for content topic and symmetric key, correct? If not, could you make a reference to the code?

Yes, the community key in the community channel key is a 33-bytes compressed public key. However, in order to expose keys, we use a different compression, which is called "multiformat" compression - zQ3sh........
We use multiformat to expose userId and communityId. Eg. Status community link is: https://join.status.im/c/zQ3shf8EvhRKHSc3skgtupULuLdWh59S5czZkvS69P6rJ7aed
To keep consistency we'll have to use multiformat in channel id:
Instead of: "0x036f5a79a5..." have "zQ3shf8Ev"
I hope that makes sense.

@felicio
Copy link
Collaborator Author

felicio commented Jan 16, 2023

@John-44 @pedro-et could you please confirm that desktop should open and handle links starting with https:// like mobile does? Thus redirecting users straight to the app if installed, and landing on the website only if not.

Cc @endulab

@John-44
Copy link

John-44 commented Jan 16, 2023

@John-44 @pedro-et could you please confirm that desktop should open and handle links starting with https:// like mobile does? Thus redirecting users straight to the app if installed, and landing on the website only if not.

Cc @endulab

Yes, both desktop and mobile should open and handle links starting with https:// , it's really important that they do this.

@endulab the URL format is going to change, @felicio @prichodko are working on some improvements to the URL format so I'll set up a call so we can discuss later in the week

@felicio
Copy link
Collaborator Author

felicio commented Feb 1, 2023

@John-44 please, could you review the individual "Matrix: Community" and "Matrix: Channel" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing and let me know if data encoded and total url lengths, from the product point of view, are worth not using and exposing public keys to servers?

While the final encoding may undo some of the compression causing the total number to go up or down by ~5%, I do believe it can be stabilized and further reduced (e.g. LZ compression, text trimming) but still placing it somewhere between 150-250 characters.

Cc @prichodko @fryorcraken

@felicio
Copy link
Collaborator Author

felicio commented Feb 8, 2023

felicio please check out https://github.com/waku-org/js-waku-examples/blob/master/examples/store-js/index.html
Note that currently the API is such that more JS code is downloaded that what is striclty necessary, I want to improve the way the bundles are split so that minimum code needs to be downloaded by the user.

– Status, web, 0x6916cc7e7db8fe0ecbe2731a9ed9c7758a5dc15f4f1629e29fbe288eb94be1e0

@fryorcraken.eth | Waku Hi, I assume this has to do with today's call where I mentioned the use of store protocol within a lambda like env (i.e. server, node.js, relatively enough space) and where the exec should take only couple of seconds to fetch the preview details. My question was if you guys would consider it a valid use case for a waku node to support REST API. And if you would have any concerns regarding the load/scalability even with caching configured (e.g. at max 1 request per community/channel/user per let's say 15 mins., depending on users sharing and clicking the reference link). But thank you for the above info anyway. Cc LNSDev

– Status, web, 0xe52f8415d9b20c12aacda745763fcc42a8c02480869f05969c4d28b3c54c4c7c

@felicio
Copy link
Collaborator Author

felicio commented Feb 8, 2023

Ok. let's wait @felicio.eth 's input because in Status Desktop he mentioned the usage of caching and NodeJS so I am a bit confused as the usage of js-waku would permit a frontend/client/browser only solution.

https://discord.com/channels/864066763682218004/1067545488934588426/1068355176173424670

@fryorcraken.eth | Waku these links are related to

  1. previews (e.g. on social or other IMs displaying meta info and an image)
  2. website (for environments without native app and when circumventing deep linking)
  3. deep linking

and it is the previews that I mentioned for which some work on the server side is needed. And if they were to be based on any data from waku now (e.g. images), which hasn't been definitively ruled out yet but explored for the purposes of the feature analysis nevertheless, it'd require such set-up.

backend logic makes the solution quite centralized and Status-dependent
Hosted, or self-hosted that I understand to be the case even with the current solution (https://github.com/status-im/universal-links-handler) served from our servers.

Are you aware of a decentralized way that can dynamically generate a html doc prior returning it to the client? Would you have an example for reference? I've heard about ipns (not ifps), but didn't spend time researching it.

data from a store query

The rest api comment I made, so the peer connection and disconnection wouldn't have to be awaited for such a short-lived op.

link points to a bridge

Could you please elaborate?

I believe that the resulting approach will depend on following factors:

  • url length with some data already encoded
  • practical use
  • speed
  • design
  • privacy
  • level of centralization and control
  • and whether any part puts us in the role of a content provider

As soon as I post the summary to aid with the decision in #327, I'll mention you as well.

Does this make things more clear?

https://discord.com/channels/864066763682218004/1067545488934588426/1068477948023296032

link points to a bridge
Could you please elaborate?

felicio I am talking about hosting JS code on IPFS and have the invite link pointing to this code, possibly via a IPFS bridge. However, I guess this is a level of decentralization that is not necessary for Status MVP.

Does this make things more clear?

Yes thanks, especially regarding social preview.

As soon as I post the summary to aid with the decision in #327, I'll mention you as well.

Sounds good.

https://discord.com/channels/864066763682218004/1067545488934588426/1069444784042807407

@felicio
Copy link
Collaborator Author

felicio commented Feb 8, 2023

@Samyoul gave me a great feedback:

  • Compress text size by dropping 8th bits and concatenating bytes.
    • Which for [A-Za-z0-9_-.\u0020] input, and a corresponding alphabet of 66 characters could theoretically compress the byte size by up to 12.5%.
    • Something @Samyoul already proved (edit: that's not going to help due to final encoding).
  • Consider IPFS instead.
    • I don't know what would it entail, but I'm thinking
      • Making sure our IPFS gateway https://ipfs.status.im/api/v0/pin/ls scales with the user base too and can handle the additional pins and queries.
      • For the time being, it'd mean content duplication since the same data is already in waku.
      • Not IPFS's CIDs, which would probably have to be stored and changed on update in waku too.
      • But IPNS's hashes of public keys instead, probably of those used for chatting.
      • It wouldn't be more performant and reliable than waku for this use case.
  • Sign content within the url.
    • I totally overlooked it even though counting with it for the case of fetching rest of the content from waku.
    • Signature would increase the resulting url length.
    • And if the signed content wouldn't be stored in waku (another duplication), the only scenario I can picture is having two sets of links. One when shared by the owner/setter of its content and another when shared by someone else who cannot encode and sign the content on behalf of the owner. Not preferable, I think.

@Samyoul , @cammellos , @richard-ramos , @iurimatias , @saledjenic , @jakubgs , @fryorcraken, @prichodko I was going to propose an extension to status-im/specs#159 in form of a single url search parameter carrying the serialized, compressed and encoded content as per "Matrix:*" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing. However after considering the above, I now propose

For the url scheme:

  • Keep user and community as is
  • Change channel to /cc/[compressed_community_public_key]/[channel_uuid] for clarity

For clients:

And for the infrastructure:

Please, would you agree?


NOTE: What I'm proposing already has some POCs, so if you'd like to validate something let me know.

@jakubgs
Copy link
Member

jakubgs commented Feb 8, 2023

If we are re-doing deep links the we need to revisit using hash arguments to preserve privacy.

Using paths or URL arguments means the paths end up on the backend side. If we use hash arguments we prevent whatever is hosting the deep-linking UI from knowing what people are actually linking.

@jakubgs
Copy link
Member

jakubgs commented Feb 8, 2023

As for the /cc/[compressed_community_public_key]/[channel_uuid] format. As long as we can generate it on the client side in the browser and use hash arguments that's fine.

@felicio
Copy link
Collaborator Author

felicio commented Feb 9, 2023

@jakubgs thank you,

revisit using hash arguments to preserve privacy
...
If we use hash arguments [/c/#[compressed_community_public_key]] we prevent whatever is hosting the deep-linking UI from knowing what people are actually linking.

that is true, even for whatever backend/server is generating the previews, however without an access to the public key I cannot fetch content like profile images from Waku nor verify the encoded content in url. Because if it contained a signature the public key could be recovered anyway.

On the other hand, and considering "IPNS's hashes of public keys instead, probably of those used for chatting" I've mentioned above, I think the verification could be done by comparing the hashes after visiting the urls (and in our apps even before clicking or resolving).

But previews outside our apps would still need to be considered untrusted, I believe.

@rymnc @s1fr0 it might be a reach mentioning you on this issue, but I'm convinced your take on the matter:

  • Storing somewhere user's content/metadata
  • Tying it to its identity
  • Verifying it in code where we can prior displaying
  • And all that without revealing the real identity/public key to servers

would be helpful.

@rymnc
Copy link
Member

rymnc commented Feb 9, 2023

@felicio, thanks for the tag - let me get up to speed with this issue and let you know soon

@felicio
Copy link
Collaborator Author

felicio commented Feb 14, 2023

Following has been decided to address above mentioned concerns in short term,

  • to encode some data in url until waku response is instant and considered reliable enough for this use case
  • a verifying state while loading until encoded data is matched against waku response 🕵️

limiting proposals to following two,

  • sharing public key with hosting server 🔓
    • Examples
      • Min
        • Community: https://status.app/c/zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU?d=iwWAGgFMQgcjNDM2MERGAw== (test data)
        • Channel: https://status.app/cc/zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU/30804ea7-bd66-4d5d-91eb-b2dcfe2515b3?d=iw-AEgFsKgTwn6uFMgcjRUFCNzAwOgwaAUxCByM0MzYwREYD (test data)
        • User: https://status.app/u/zQ3shUHp2rAM1yqBYeo6LhFbtrozG5mZeA6cRoGohsudtsieT?d=iweAEgVMb3JlbToHI0VBQjcwMAM= (test data)
      • Mid
        • Community: https://status.app/c/zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU?d=G2UA-J0Htk1qG1d7YfVJhrl-0BBikkagwtbcoqXJocUgXzQ1QxrbIQ9uG_t_MGFsiHoQhUCCShIAwjqn3CcrdFhUMGQT03NTR6r0E7k3Y3Z6TKSiQy1re8FPXfCwgA7-LOTotWW_rww= (test data)
        • Channel: https://status.app/cc/zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU/30804ea7-bd66-4d5d-91eb-b2dcfe2515b3?d=i0aAEgxsb3JlbS1pcHN1bWEqDfCfkoLigI3imYDvuI8yByNFQUI3MDA6ZhoPTG9yZW0gaXBzdW0gZXUuIkZMb3JlbSBpcHN1bSBkb2xvciBzaXQgYW1ldCwgY29uc2VjdGV0dXIgYWRpcGlzY2luZyBlbGl0LiBTZWQgZXQgZXggZXUuKKDCHkIHIzQzNjBERgM= (test data)
        • User: https://status.app/u/zQ3shUHp2rAM1yqBYeo6LhFbtrozG5mZeA6cRoGohsudtsieT?d=G5AAYBwJdmzJcOxhPUKlJ4MEk6lL4QNedFJp9LSDbcKdfT-WnaoBAxtw4JBRIDpttPDVyOSjpLrzeqQZ3LTNcvqj_PdKg8zJHMK6aAqWrB2QyI0auBSkidbj2zQk_GiS0I5FoFJoRnVbh4o5mc7FIXOhb4pJQtMaU-zv3fnzeF4A (test data)
      • Max
        • Community: https://status.app/c/zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU?d=G7sAgK0OzJOmo4yssBOBo6ekybWGRw-TDqBpImRzStN66S8_fBx3Wi8-CYpBaDEC98GMkJGEsSFyjBOFTRcZeFtvHzxkKV8QJcbjHpzQfWpGfDia4OjNJC4AIZP5w7_FMpkVkK5bz6NrgKbZ9pt3szpYvtBL8WoKITWKsP-sCRMPMiEP9l-CGQLv6g153IqXJ_Uuk4T2ffHRPA== (test data)
        • Channel: https://status.app/cc/zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU/30804ea7-bd66-4d5d-91eb-b2dcfe2515b3?d=G_8AYKwKeDJ8lNFUDUMv91LnB3zVmJqbOncP-lRBw2oL2sim1DYaJmLlSOBnEj6C2pzStPxw8-d13Gm9-CQoDUIriLEOhKAmx78H5owyTSTBTlue5zD9csuh3xpC1HlwNX1OrTZcY9tlJt_Pc9t_C9RripV4uVRGtp3ugIgmxDAyjiY4RzgLJnA0spg_PMu4TGYFZOij59Y9QdPsx6DvZnWwfWOU4tUUQmoW0cVnLZh4kAm58fwlmCHwod5V63EpXp7Uq8wL3boV8b3QLms= (test data)
        • User https://status.app/u/zQ3shUHp2rAM1yqBYeo6LhFbtrozG5mZeA6cRoGohsudtsieT?d=GxUBoJwHdsOHgITK206z2oTI6cn2pEtSkxmSJyFKmBF4myq27J1Oe-VxJ_YlRcI4D_AFouMYJ6wzv66LnJ8PjkNGz1Pk70qr7Qlu6Ju9l74z4W0mn-EsI7Ugm_X1x5OzWB2Qpa_mVH9A0xqmcxK3ul6doFFv-kKHy3BcYGvus1A068Q_jaZ6MVZv7GPMUg4UTrokunCH-qNmgtU6VGtCetYJ_MlteVwghXn8aQawLdzvCbw_CVbIwBOX4nqHxmqUOtIB (test data)
  • and not sharing 🔒
    • Examples
      • Min
        • Community: https://status.app/c?d=iwWAGgFMQgcjNDM2MERGAw==#zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU
        • Channel: https://status.app/cc/30804ea7-bd66-4d5d-91eb-b2dcfe2515b3?d=iw-AEgFsKgTwn6uFMgcjRUFCNzAwOgwaAUxCByM0MzYwREYD#zQ3shY7r4cAdg4eUF5dfcuCqCFzWmdjHW4SX5hspM9ucAarfU
        • User: https://status.app/u?d=iweAEgVMb3JlbToHI0VBQjcwMAM=#zQ3shUHp2rAM1yqBYeo6LhFbtrozG5mZeA6cRoGohsudtsieT
      • ditto (public key is moved after "#")

@John-44 please, could you confirm with 👍 or otherwise comment below? Also, do let me know if I should schedule a call and clarify anything.

Cc @prichodko


Note: Mind that some social platforms or IMs vissualy trim the urls, so urls would actually not be displayed in full length (see "Limits:*" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing)

Note: Without sharing public keys with hosting servers some data like profile photos or banners cannot be included in the previews (see "Matrix:*" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing)

Note: encoding small data has actually an opposite effect due to serialization and compression headers, which could however be optimized to shave off couple of characters but left as-is for simplicity

This was referenced Feb 14, 2023
@felicio
Copy link
Collaborator Author

felicio commented Feb 14, 2023

At the beginning I pointed out,

  • broken desktop deep link resolution without status-im:// but with https:// instead
  • support for utf8 in display names and descriptions while products requirements expects only [A-Za-z0-9_-.\u0020], or like, character class
  • missing color field for user

Please, where are these being addressed/tracked now?

Cc @John-44

@endulab
Copy link

endulab commented Feb 15, 2023

@felicio I created 2 tasks for MacOS and Windows:
status-im/status-desktop#9549
status-im/status-desktop#7957
I will catch you later to discuss details.

@felicio
Copy link
Collaborator Author

felicio commented Feb 20, 2023

In addition to #327 (comment),

Cc @John-44

@John-44
Copy link

John-44 commented Feb 20, 2023

@endulab @cammellos probably obvious (so apologies for stating the obvious) , and I'm not sure if this is the correct place to mention this, but the generation of this URL needs to happen in status-go so the same URL generation code can be used by both Status Desktop and Status Mobile

@felicio
Copy link
Collaborator Author

felicio commented Mar 2, 2023

@felicio I created 2 tasks for MacOS and Windows:
status-im/status-desktop#9549
status-im/status-desktop#7957
I will catch you later to discuss details.

@iurimatias @cammellos, please could you create the tasks too but for our supported mobile versions and share the links here afterwards? See status-im/specs#159 for changes to the URL scheme.

Cc @pedro-et

@cammellos
Copy link

@felicio apologies for the delay, created issue on mobile side status-im/status-mobile#15325

@jakubgs
Copy link
Member

jakubgs commented Mar 13, 2023

I see we have introduced a <signature> that is behind a # hash argument. Not sure what's the purpose of the signature, but that still means we would still be receiving things like [compressed_channel_key] or [compressed_user_key or ens_name] via the URL on the backend/storage side of the preview page. Are we sure we want to do it like that?

The goal is preserve annonimity of users, and this scheme does allow for correlating the IP with most of the URL on backend.

@felicio
Copy link
Collaborator Author

felicio commented Mar 15, 2023

@jakubgs

<signature> that is behind a # hash argument

That, as well as public keys or ENS names.

what's the purpose of the signature

To ensure integrity of the encoded content. More on that at status-im/specs#169 (comment).

we would still be receiving things like [compressed_channel_key] or [compressed_user_key or ens_name] via the URL on the backend/storage side of the preview page.

No, as per above. Also, since we're using secp256k1, public keys can be recovered on the client side from the encoded content and the signature.

correlating the IP with most of the URL

  1. Was your concern only public keys? Which has been hopefully addressed above.
  2. What other content before the hash do you consider not preserving of anonymity?
  3. Who did you discuss this with? And what agreement have you reached?

The now encoded content could also be moved after the hash. Which would however mean that none of it could be used in social previews/cards.

Cc @John-44

@jakubgs
Copy link
Member

jakubgs commented Mar 15, 2023

Was your concern only public keys? Which has been hopefully addressed above.
What other content before the hash do you consider not preserving of anonymity?

Any data that identifies what channel/person/group the link is for.

Who did you discuss this with? And what agreement have you reached?

I discussed it with myself. My understanding is that the things before the hash still reveal information about what channel/person/group is being linked and that can be correlated with IP and also client data.

Is my understanding correct then that this decision is between preserving privacy and not being able to do previews, and not preserving privacy but having fancy previews?

@felicio
Copy link
Collaborator Author

felicio commented Mar 15, 2023

@jakubgs

as we have discussed several times.

– status-im/specs#159 (review)

Based on the above, I thought you have discussed this with others thus could provide more context as to who and why exactly.

If not, no worries.

My other questions were just to figure out whether your concern was only public keys or other (encoded) content too, and simply note the implication.

@John-44 please, could you leave a comment here addressing the following concern @jakubgs has raised? I could then make changes accordingly, if necessary.

correlating the IP with most of the URL ([not public keys but display name or description, for example]) on backend

– #327 (comment)

@John-44
Copy link

John-44 commented Mar 15, 2023

Yes there is a trade between privacy / adoption here, because yes a sufficiently determined attacker could perform a correlation attack (however to do so in this instance would not be trivial). This has already been discussed a bunch, and it was from this discussion that the decision to not have images in unrolls (so the keys would be shielded by the # from the server) but to have URL specific metadata in unrolls (at the cost of exposing this metadata to the server) was made.

It might be a good idea to log a feature request in https://status-desktop.featureupvote.com/ for post MVP that is titled something like "give users option to share URLs without metadata for privacy purposes". Then we can give the user the choice of whether or not they want to share A) URLs that unroll nicely with meta info but where the server also sees this meta info or B) URLs that don't contain the meta info but which also don't contain any URL specific content in the unroll.

Such as feature would not be for MVP because we are busy trying to cut scope, and we really can't afford to add scope at the moment. For MVP the decision has been made to go with the trade I've described above

@jakubgs
Copy link
Member

jakubgs commented Mar 15, 2023

Very well, if this is a conscious trade off that people are ware of, fair enough.

@jakubgs
Copy link
Member

jakubgs commented Mar 15, 2023

I think it's fine if we add an option to have the privacy preserving URLs without nice previews for users to pick in the UI later on, as long as we actually remember to do that. And not just keep it as a "nice to have" forever.

@felicio
Copy link
Collaborator Author

felicio commented Jun 1, 2023

Closing in favor of https://github.com/orgs/status-im/projects/91 for tracking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants