Cannot sign raw payloads

[ntn-x2]

Apparently the signer does not support or it is not fully compatible with raw payload signing.

I have had issues in two different cases, with my wallet-managed account linked to Polkadot extension as external:

1. Polkassembly asks me to sign in with my account. I sign in the challenge. Polkassembly says signature invalid. The same process goes through with an account that lives entirely inside the extension.
2. On a website that calls page.signer.signRaw(), which triggers the Polkadot extension, the payload cannot be parsed by the signer, even though both the website and the signer are using the latest chain metadata.

I have had zero problems with extrinsic signing so far, so I guess the problem lays somewhere in the raw payload signing. Is this a known issue? If not, I could provide with additional screenshot/context about both errors.

[varovainen]

What version of Signer do you use?
Also, what exactly do you mean by raw payload: is it a text message, an opaque transaction, something else?
A bit more details would help.

[ntn-x2]

Latest version, 5.0.0.

What I mean by raw payload is, as I recently found out, the function that the polkadot extension exposes in https://github.com/polkadot-js/extension/blob/master/packages/extension-base/src/background/RequestBytesSign.ts. What it does, is to wrap the provided payload in<Bytes>...</Bytes>. We found that out, and changed our code to account for this padding.
Now, if I try to sign such a wrapped payload with the extension using an external account handled by the signer app, it fails to parse the QRCode with the error I mentioned above. Please find a (not very indicative) screenshot of the error below.

[Slesarew]

Oh, I see. You are attempting to send a "sign message" payload, but there are non-UTF8 symbols in the payload. "Send message" interface was designed for text messages and the Signer needs a way to visualize it. If you are indeed using this interface to send binary data, we need to figure out how to represent it and make sure that this does not result in blind signing (which is strictly forbidden) and that it does not result with potential attacker being able to sneak a transaction for signing while making the user believe that it is a binary payload.

[ntn-x2]

I thought that is why the extension wraps the payload in the <Bytes> tag, right? So that it is impossible to get a transaction signed in this way. I was under the assumption that the signer would follow the same process.

[EDIT]
Of course I am not making the rules, but it seems that more and more applications, and also the extensions, are using something different than what the signer supports, so I would say it makes sense for the signer to support these use cases, including signing raw payloads properly wrapped to avoid the issue you mentioned above.

[Slesarew]

That part is actually similar between transaction and messages. What differs is that messages do not have extensions like nonce and lifetime. So this works in most networks (there are some that do not have extensions, but those are special cases).

If what you are trying to sign is not a valid transaction but has binary format indeed, can you encode it somehow uft-friendly? Like base64 of something like that. Users will see encoded transaction and could maybe compare it against something this way. Although I would strongly advise against using payloads that could not be rendered into something human-understandable, as this requires certain trust in a generally trustless system. With messages, this is not forbidden, but users will (hopefully) recognize the danger and I would expect them to have less overall willingness to use the system.

Signing non-UTF payloads (with a warning) could be allowed if we just show binary content of payload (hex-encoded?) on failure to decode message as string. In my opinion, this is just weird (sloppy?) - why not include this functionality properly in metadata and make proper transactions? Or at least spell clearly what the thing signed by user means? - but I understand that there are unimaginable uses for the system and we can indeed consider adding this feature. Let me just show this to @kirushik for security review.

[ntn-x2]

This is what happens in websites I and my team have no control over, for instance Polkassembly. I guess what they ask the extension to sign is not UTF-8 encoded, as otherwise I am understanding it would be parsed by the signer app.
So my point is that at least the extension and the signer should support the same features, and right now there seems to be a disconnect between the two, so that signer-handled accounts cannot be used everywhere extension-handled accounts can.

[ntn-x2]

Is this issue relevant? Does it make sense for me to try to find some spare time a give it a try to fix it? Never browsed the codebase before, so not sure I could even make an estimate. It is really a pity that I can't use the Signer everywhere I would like to, given it's really cool.

[Slesarew]

Is it still present? #1014 seems to be related to this (possibly duplicate)

[ntn-x2]

So I am using the same benchmark as before (i.e., https://kilt.polkassembly.network/).
While when I opened this issue the challenge could not be signed at all, now it can indeed be signed (see screenshot attached), but verification still fails on Polkassembly side. I guess this is now something that they have to fix?

[Slesarew]

As far as I understand, you are correct. This is improper payload formatting, for some reason they did not scale-encode the message properly and/or used an external signer raw payload interface that should not have been used as is for this purpose.

[ntn-x2]

So the proper way to create challenges that are "QRCodable" is by doing something like scaleEncode(toU8a('<Bytes>challenge</Bytes>'))?

[Slesarew]

Right, but maybe without toU8a conversion. On Signer side it was meant to just follow "sign message" protocol, just sign a string.

[ntn-x2]

Ok, I will try to reach out to the Polkassembly people to investigate the issue. If they can get it fixed, I will close this issue.

[Tbaut]

I'm saddened to inform you that reg. polkadot-js/extension#1053

1. What this does actually is that it drops support for QR signing altogether. To me this is a big bummer, what @ntn-x2 tries to do will simply not be possible any more. Here's a screenshot of the master build of the extension when signing something with a QR code account screenshot
2. There has been no release of the extension since December, any PR that has been merged since is not live, so there's no way you can test this unless you actually build it.

[ntn-x2]

It is indeed a big bummer. It is basically rendering the Parity Signer useless, if not for signing transactions. We have seen that more and more generic payload signing is an important feature for logins and similar use cases. Are we entirely sure there is no way around it than simply dropping support altogether? It seems quite limiting to me.

[Slesarew]

On contrary, this was removed for fixing as far as I know. It was not supposed to be possible there in current state. I'm pretty sure support will come back once it is implemented properly, it's just that someone must go and implement it.

[Slesarew]

Either way, this is just something that should be fixed outside of Signer. I'll keep this issue open to help building pressure on that front.

[ntn-x2]

I was not able to find a tracking issue for this, and would love to see one including a suggestion on how to proceed further and welcome external contributions 😁

ping @Tbaut

[Tbaut]

I'm pretty sure support will come back once it is implemented properly, it's just that someone must go and implement it.

Yup, someone needs to do it. I was the one bringing the QR signing to the extension, without scale encoding. I'm definitely happy to be the one to bring it back the right way, but I need to understand if there is a right way and how. I opened an issue in the extension repo polkadot-js/extension#1070
Can you please comment there @Slesarew to let me know your thoughts?

[Slesarew]

Ok, looks like we can adopt 2 formats of signing freeform messages in Signer simultaneously.

1. Messages enveloped in <Bytes>...</Bytes>. These are used in pjs extension and Stylo. These are protected from slipping transactions into them by the fact that no encoded transaction could start with <Bytes> (well, theoretically it is possible but in the end will require that the call 121 in pallet 66 is 15 symbol long and extensions end with </Bytes>, maybe we should add a test for it).

2. Messages encoded by SCALE. These were used in some projects that are not actual anymore. These are protected from being transactions by the fact that all payload's length is encoded, which implies that there are no signed extensions in this network - thus there is no nonce and block hash and blind signing is probably not the worst issue of this network. This does not have to be supported by pjs, but since someone used it before, we can just keep it.

Distinguishing between the 2 formats is quite simple. We can give them slightly different representation. Most importantly - whole payload should be just signed as is and Signer should only allow limited number of permitted formats.

We can keep adding allowed formats later if needed.

I feel like there is a more general solution to distinguish a valid transaction from not-transaction-for-all-possible-networks, but given that there is no network identification in this kind of message, this is quite dangerous feature.

@kirushik please check if this makes sense.

[Tbaut]

Any news reg. this? It'd be great to have it in before the next release.

[prybalko]

Fixed in #1332

[ntn-x2]

Nice! When will this be released?

[prybalko]

We expect a release in a month or two.
If you're ok with using betas, feel free to use our test builds.
https://testflight.apple.com/join/s4Y3n7L3

[wischli]

What is the first version of the Polkadot Vault which supports this feature? The changelogs don't reveal this information. Using v6.0.2 does not seem to support it 😢

[prybalko]

What is the first version of the Polkadot Vault which supports this feature? The changelogs don't reveal this information. Using v6.0.2 does not seem to support it 😢

Polkadot Vault does support raw payload signing since v6. The problem is in the pjs extension
polkadot-js/extension#1268

[Tbaut]

Use Talisman @wischli it's implemented there and works well with Vault.

[wischli]

Use Talisman @wischli it's implemented there and works well with Vault.

I tried but none of my QR signer addresses are listed as a linkable options on Polkassembly. Fixed by building the extension locally and running in dev mode.

[Tbaut]

Because you need to "connect" on Talisman the accounts with the Dapp. I just did it successfully.
There's a "X accounts connected" at the top of Talisman, pjs master has the same BTW. If you added new accounts to it, you need to tell Talisman to connect this new account too.

[wischli]

Because you need to "connect" on Talisman the accounts with the Dapp. I just did it successfully. There's a "X accounts connected" at the top of Talisman, pjs master has the same BTW. If you added new accounts to it, you need to tell Talisman to connect this new account too.

You are absolutely right. Thanks a lot for taking the time to educate me. Next Decoded or Sub0 event I'll get you a beer 🍺