[RFC] Magic URL Payload Format for Twitter (v1.0.0.0)

[neruthes]

Background

Twitter has a strict length restriction for tweets which we must find ways to bypass in order to publish armored payloads within one tweet.

Basic Idea

URLs on Twitter will be converted to t.co short links and only the length of the t.co short link is considered actually occupying characters in the tweet. We add a prefix before the actual payload to make it look like a real URL for Twitter, in order to take advantage of this feature.

Prefix

We maintain a list of popular websites (like Alexa top 100) and use a random prefix to avoid basic pattern detection.

Encoding

Since there are restrictions on which characters may be used in URI, according to IETF and W3C specifications, we use a subtly different payload encoding method.

Refer to specifications of location object in DOM API for JavaScript.

Prefix

The prefix part always look like https://www.amazon.com/item/233.html.

It has protocol, host, and path.

It does not have user, password, search, hash.

It may or may or have port.

Prefix Randomization (PR)

Prefix Randomization is not mandatory but recommended. For now, it is ok to maintain a simple list of prefixes.

The prefix has a static part and a dynamic part.

Static part includes host.

Dynamic part includes:

Part Name	Value Range	Examples
protocol	...	http, https
host	domain names, IPv4 addresses	twitter.com
port	implied, 8000, 8080, 9527-12315
path	\/[\w-_\.]{0, 24}/	/_233_/

Update: URIs whose host is an IPv4 address will not be converted to short links on Twitter.

Base64 Alternative Characters

In Base64 encoding, we need +, /, and =. These characters need to be replaced in order to ensure the reliability of URI detection on Twitter. We should follow RFC 4648 for URI-safe Base64 codec.

From	To
+	-
/	_
=	=

Padding characters may optionally be discarded.

Header Token

We use %20 to mark the payload sequence starting.

Footer Token

We use %40 to mark the payload sequence ending.

Garbage Bytes (GB)

Random garbage bytes may be added after footer token.

Garbage Bytes is not mandatory but recommended.

Separation Token

We use . to mark the separation between two adjacent fields.

---

Discussion wanted.

[Tedko]

I think this approach is too hacky. Considering the fact that we will support iMgs / files sooner or later, let’s rather use IPFS or some storage solutions directly. It’s important to make our solutions elegant and don’t have any single point failure — imagine what if twitter decides to modify the URL policy, all of our users post lose permanently. I’m strongly against this way. It’s worse than BaseCJK/Emoji or store in IPFS and put hash pointers in tweet.

…

On Fri, Sep 20, 2019 at 02:26 Neruthes 0x5200DF38 ***@***.***> wrote: Background Twitter has a strict length restriction for tweets which we must find ways to bypass in order to publish armored payloads within one tweet. Basic Idea URLs on Twitter will be converted to t.co short links and only the length of the t.co short link is considered actually occupying characters in the tweet. We add a prefix before the actual payload to make it look like a real URL for Twitter, in order to take advantage of this *feature*. Prefix We maintain a list of popular websites (like Alexa top 100) and use a random prefix to avoid basic pattern detection. Encoding Since there are restrictions on which characters may be used in URI, according to IETF and W3C specifications, we use a subtly different payload encoding method. Refer to specifications of location object in DOM API for JavaScript. Prefix The prefix part always look like https://www.amazon.com/item/233.html. It has protocol, host, and path. It does not have user, password, search, hash. It may or may or have port. Prefix Randomization (PR) Prefix Randomization is not mandatory but recommended. For now, it is ok to maintain a simple list of prefixes. The prefix has a static part and a dynamic part. Static part includes host. Dynamic part includes: Part Name Value Range Examples protocol ... http, https host domain names, IPv4 addresses twitter.com port implied, 8000, 8080, 9527-12315 path \/[\w-_\.]{0, 24}/ /_233_/ Base-64 Alternative Characters In Base-64 encoding, we need +, /, and =. These characters need to be replaced in order to ensure the reliability of URI detection on Twitter. From To + - / / = _ Header Token We use %20 to mark the payload sequence starting. Footer Token We use %40 to mark the payload sequence ending. Garbage Bytes (GB) Random garbage bytes may be added after footer token. Garbage Bytes is not mandatory but recommended. Separation Token We use .to mark the separation between two adjacent fields. ------------------------------ Discussion wanted. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#198?email_source=notifications&email_token=ABTAVTKZ74IEQR2MRVCNNULQKSJLDA5CNFSM4IYU2LBKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HMUDQMQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABTAVTLUFBPKBABT34FJIMDQKSJLDANCNFSM4IYU2LBA> .

[Tedko]

Over all we’re building a protocol and a product. This hacky solution may help the product in short term but will harm the protocol largely.

…

On Fri, Sep 20, 2019 at 02:35 Suji Yan ***@***.***> wrote: I think this approach is too hacky. Considering the fact that we will support iMgs / files sooner or later, let’s rather use IPFS or some storage solutions directly. It’s important to make our solutions elegant and don’t have any single point failure — imagine what if twitter decides to modify the URL policy, all of our users post lose permanently. I’m strongly against this way. It’s worse than BaseCJK/Emoji or store in IPFS and put hash pointers in tweet. On Fri, Sep 20, 2019 at 02:26 Neruthes 0x5200DF38 < ***@***.***> wrote: > Background > > Twitter has a strict length restriction for tweets which we must find > ways to bypass in order to publish armored payloads within one tweet. > Basic Idea > > URLs on Twitter will be converted to t.co short links and only the > length of the t.co short link is considered actually occupying > characters in the tweet. We add a prefix before the actual payload to make > it look like a real URL for Twitter, in order to take advantage of this > *feature*. > Prefix > > We maintain a list of popular websites (like Alexa top 100) and use a > random prefix to avoid basic pattern detection. > Encoding > > Since there are restrictions on which characters may be used in URI, > according to IETF and W3C specifications, we use a subtly different payload > encoding method. > > Refer to specifications of location object in DOM API for JavaScript. > Prefix > > The prefix part always look like https://www.amazon.com/item/233.html. > > It has protocol, host, and path. > > It does not have user, password, search, hash. > > It may or may or have port. > Prefix Randomization (PR) > > Prefix Randomization is not mandatory but recommended. For now, it is ok > to maintain a simple list of prefixes. > > The prefix has a static part and a dynamic part. > > Static part includes host. > > Dynamic part includes: > Part Name Value Range Examples > protocol ... http, https > host domain names, IPv4 addresses twitter.com > port implied, 8000, 8080, 9527-12315 > path \/[\w-_\.]{0, 24}/ /_233_/ Base-64 Alternative Characters > > In Base-64 encoding, we need +, /, and =. These characters need to be > replaced in order to ensure the reliability of URI detection on Twitter. > From To > + - > / / > = _ Header Token > > We use %20 to mark the payload sequence starting. > Footer Token > > We use %40 to mark the payload sequence ending. > Garbage Bytes (GB) > > Random garbage bytes may be added after footer token. > > Garbage Bytes is not mandatory but recommended. > Separation Token > > We use .to mark the separation between two adjacent fields. > ------------------------------ > > Discussion wanted. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#198?email_source=notifications&email_token=ABTAVTKZ74IEQR2MRVCNNULQKSJLDA5CNFSM4IYU2LBKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HMUDQMQ>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABTAVTLUFBPKBABT34FJIMDQKSJLDANCNFSM4IYU2LBA> > . >

[Misaka-0x447f]

for insider preview, it will be implemented as this:
https://google.com/%203/4|${data.ownersAESKeyEncrypted}|${data.iv}|${data.encryptedText}|${data.signature}%40

[Tedko]

So no other feedbacks?
I'm wondering do we have other ways besides this? like the IPFS/etc way i mentioned.

[neruthes]

If you want to delay to February.

[neruthes]

We may initiate a project to examine the availability of these solutions and arrange implementations accordingly. Also, I prefer not to put backward compatibility at risk, unless we revert the banner-removal commit and pretend it is always early-access beta test version.

[neruthes]

These will require much amount of work and I prefer putting these resources on features which have greater priority, including new dashboard, automatic recipient amending, and Misakanet.

[Tedko]

These will require much amount of work and I prefer putting these resources on features which have greater priority, including new dashboard, automatic recipient amending, and Misakanet.

@neruthes
Let's do this:

• For making Mask twitter available for users soon, so

1. automatic recipient amending
2. 'Shared' recipient (?) for PR and growth hack
3. URL hack
4. base Emoji research
5. then some image/file decentralized storage hack? (such as IPFS I mentioned)

3->5 need to be relatively fast since we don't want to got banned etc.

[neruthes]

These will require much amount of work and I prefer putting these resources on features which have greater priority, including new dashboard, automatic recipient amending, and Misakanet.

@neruthes
Let's do this:

• For making Mask twitter available for users soon, so

1. automatic recipient amending
2. 'Shared' recipient (?) for PR and growth hack
3. URL hack
4. base Emoji research
5. then some image/file decentralized storage hack? (such as IPFS I mentioned)

3->5 need to be relatively fast since we don't want to got banned etc.

IPFS can be an option for fallback, with regards to our Principle of Saturation. I have no idea how much times does it require to build IPFS compatibility.

Base-Emoji may have difficulties. Keep watching #139.

And explain 'Shared' recipient (?) for PR and growth hack, please.

[Artoria2e5]

Instead of encoding it as a path, it is also very viable to encode it as a #-fragment. This has the benefit of allowing native base64. For example:

https://prettier.io/playground/#N4Igxg9gdgLgprEAuEB6VACASgMQMIYAsAbIQBwA6UYANgIYDODGAQo3KQKpYAyGwVDBnQZODOBgAWMGAAcGSdAHMAljEkBXAEYA6SAFtUWgE4q6UGhABucBgHcIxgCYMj7UhuM0MKqA3h0TjpUAL4gADQgELIwKtAMyKB0xsYQdgAKyQgJKHRWECpOESAmdGAA1nAwAMqyZb5KyDDGGnCR0vo0AOqSarZ1YHDV2WoqVmoAnsjgTMW+4sYw6cZ0Svp0yABmdDTikQBWDAAeLCsVVdV0+nA8vnBbO3sgh0fVDTRwAIoaEPAPu20QHVjAtpjAJrJbGBTDFirJTLAuoV1MgyAAGSLwiDiLorWTTKBwGzGYoARx+8GW0RyIEYAFpCXAnEzisY4OSVGzlqt1v8nuJ9Co+YCGO8vhT7khtgDIjA6FokU4UUgAEyylYqGgNPAQfS8lBQaD3SIacQAFXlOWl4hCISAA

Another thing we can use for an alternative-base64 is the RFC 4648 base64url. The equal-signs can be discarded.

[Tedko]

@Artoria2e5 good idea.

[neruthes]

Updated with link to RFC 4648.

I would still recommend not to rely on location.hash. It creates extra reliance, although the extra risk may be little. Since we always struggle to build a stable software, it is reasonable to avoid this reliance even when it is small.

[neruthes]

I appears that this RFC is open for long enough and a lot of improvements have been merged. I will move this RFC to become a current technical specification on Tuesday. Later on, all suggestions for modification will be difficult.

[Tedko]

These will require much amount of work and I prefer putting these resources on features which have greater priority, including new dashboard, automatic recipient amending, and Misakanet.

@neruthes
Let's do this:

• For making Mask twitter available for users soon, so

1. automatic recipient amending
2. 'Shared' recipient (?) for PR and growth hack
3. URL hack
4. base Emoji research
5. then some image/file decentralized storage hack? (such as IPFS I mentioned)

3->5 need to be relatively fast since we don't want to got banned etc.

IPFS can be an option for fallback, with regards to our Principle of Saturation. I have no idea how much times does it require to build IPFS compatibility.

Base-Emoji may have difficulties. Keep watching #139.

And explain 'Shared' recipient (?) for PR and growth hack, please.

@neruthes similar functionality like 'all maskbook user can see' etc. As we discussed before in the chat. This is mainly for growth. Imagine some KOL post something encrypted.

[Tedko]

IPFS can be an option for fallback, with regards to our Principle of Saturation. I have no idea how much times does it require to build IPFS compatibility.

@neruthes I will talk to our friends 👬 who share similar vision and using ipfs now. Will try to talk to textile as well.

[neruthes]

These will require much amount of work and I prefer putting these resources on features which have greater priority, including new dashboard, automatic recipient amending, and Misakanet.

@neruthes
Let's do this:

• For making Mask twitter available for users soon, so

1. automatic recipient amending
2. 'Shared' recipient (?) for PR and growth hack
3. URL hack
4. base Emoji research
5. then some image/file decentralized storage hack? (such as IPFS I mentioned)

3->5 need to be relatively fast since we don't want to got banned etc.

IPFS can be an option for fallback, with regards to our Principle of Saturation. I have no idea how much times does it require to build IPFS compatibility.
Base-Emoji may have difficulties. Keep watching #139.
And explain 'Shared' recipient (?) for PR and growth hack, please.

@neruthes similar functionality like 'all maskbook user can see' etc. As we discussed before in the chat. This is mainly for growth. Imagine some KOL post something encrypted.

For this matter, we may amend UserGroup Abstraction Model (#12). It is not within the scope of RFC 198.

[Tedko]

These will require much amount of work and I prefer putting these resources on features which have greater priority, including new dashboard, automatic recipient amending, and Misakanet.

@neruthes
Let's do this:

• For making Mask twitter available for users soon, so

1. automatic recipient amending
2. 'Shared' recipient (?) for PR and growth hack
3. URL hack
4. base Emoji research
5. then some image/file decentralized storage hack? (such as IPFS I mentioned)

3->5 need to be relatively fast since we don't want to got banned etc.

IPFS can be an option for fallback, with regards to our Principle of Saturation. I have no idea how much times does it require to build IPFS compatibility.
Base-Emoji may have difficulties. Keep watching #139.
And explain 'Shared' recipient (?) for PR and growth hack, please.

@neruthes similar functionality like 'all maskbook user can see' etc. As we discussed before in the chat. This is mainly for growth. Imagine some KOL post something encrypted.

For this matter, we may amend UserGroup Abstraction Model (#12). It is not within the scope of RFC 198.

ACK

[neruthes]

IPFS can be an option for fallback, with regards to our Principle of Saturation. I have no idea how much times does it require to build IPFS compatibility.

@neruthes I will talk to our friends 👬 who share similar vision and using ipfs now. Will try to talk to textile as well.

Good to hear that. But I recommend moving the IPFS middleware to the next milestone as we all see the risk of introducing extra delay.

[neruthes]

The internal payload structure may be subject to refactoring as we design #329

[neruthes]

Ratified.