fix child keys derivation adding bytes padding #1139

gravityblast · 2018-08-06T23:45:40Z

Problem

While deriving a child key, if i ≥ 2^31 (hardened), the resulting private key is padded with a 0x00 to make it 33 bytes.
The specs says:

Check whether i ≥ 2^31 (whether the child is a hardened key).
    If so (hardened child): let I = HMAC-SHA512(Key = cpar, Data = 0x00 || ser256(kpar) || ser32(i)). (Note: The 0x00 pads the private key to make it 33 bytes long.)

The important part is: The 0x00 pads the private key to make it 33 bytes long.
If we pad it to always be 33 bytes, we should also add more zeros when the key data is less than 32 bytes.

In this specific case the derived private key was 31 bytes.

That private key is anyway right, but the problem raises when deriving its child key.
One of the step to derive the child key is calling our splitHMAC that returns a secretKey and a chainCode.
Inside this function we make a sha512 of a seed that is a 37 bytes with:

1 byte with 0x00
32 bytes for the key data
4 bytes for the child key index

In our case, if the key was less then 32 bytes, it was shifted to the left of that 32 bytes space, resulting in a different seed, and a different data returned from the sha512 call.

Changes

I added a third test vector that has been added to the BIP32 spec after this part of the code has been written (taken from the test vector 3 section of BIP32), though our old codebase without this patch is not failing with those.

I added unit test to test a specific mnemonic key that generates a key of 31 bytes during one of the steps of the derivation of a key with path m/44'/60'/0'/0/0.

The main change is padding the key data with zeros to always have 32 bytes in the field KeyData of ExtendedKey.

Links

dshulyak · 2018-08-07T02:52:20Z

extkeys/hdkey.go

-		child.KeyData = keyBigInt.Bytes()
+		keyData := keyBigInt.Bytes()
+		if len(keyData) < 32 {
+			extra := make([]byte, 32-len(keyData))


if you want you could do a single allocation instead of 2

extra := make([]byte, 32-len, 32)

@dmitryn yes good point I'll do it!

dshulyak · 2018-08-07T10:32:38Z

@pilu do you have a link to relevant bip where this condition is described?

gravityblast · 2018-08-07T10:44:04Z

@pilu do you have a link to relevant bip where this condition is described?

For now I just compared to other implementation. But looking at Private parent key → private child key, the private key should always be 0x00 + 32 bytes, which is not in our case when it's a number smaller then 32 bytes. What do you think @dshulyak ?

gravityblast · 2018-08-07T10:45:14Z

@dshulyak so the problem is not in that key (the one smaller than 32 bytes), because it's still a valid one. but in the generation of the child, because the parent private key is smaller than 32 bytes

gravityblast · 2018-08-07T10:48:52Z

@dshulyak in fact I think the same logic should be used even when k.IsPrivate is false

gravityblast · 2018-08-07T23:22:21Z

in fact I think the same logic should be used even when k.IsPrivate is false

actually if it's public is not actually needed because we call SerializeCompressed that is already calling paddedAppend
https://github.com/status-im/status-go/blob/develop/vendor/github.com/btcsuite/btcd/btcec/pubkey.go#L134

mandrigin

since it is very non-trivial to roll out these updates, we need

unit-tests for that particular case to prove that it works correctly
we need the old version of this code still available with a flag or a separate method for migration purposes
we need investigations if all our Instabug reports about this problem are actually fixed by that, so we don’t have anything else that slipped in

adambabik · 2018-08-08T09:55:26Z

@mandrigin

we need the old version of this code still available with a flag or a separate method for migration purposes

We can only check this when recovering an account. If that happens and we discover that the addresses are different, how the process will look like? If we decide to support both versions in the UI, this will root deep into the codebase responsible for keys and accounts and it will make it harder to reason about.

I would try to make avoid that. For example, we can make an announcement first and then release a version that fixes it without supporting the bugged version. If someone misses the window, they should be able to download an old release that will allow to move founds.

status-im-bot · 2018-08-08T12:34:05Z

Can one of the admins verify this patch?

corpetty · 2018-08-08T13:02:15Z

they should be able to download an old release that will allow to move founds.

If this is the route we go with only going with the new proper method, I suggest a how-to guide to go with this announcement.

Who should lead that?

I also agree that we should go this route as it would lead to address/identity issues across the app.

corpetty

as key-generation is pivotal to many parts of Status, I would also like to see unit-tests to help give confidence of proper behavior.

gravityblast · 2018-08-08T13:04:27Z

@corpetty yes for sure I'll do as soon as I continue with this. I just opened the PR to share that part of the code but it's just a start, so that we can look at it together.

mandrigin · 2018-08-08T14:50:02Z

To avoid having these situations in the future, let's try to find a good test suite for key derivation.
The vast majority of blockchain works is open source, this BIP was around since 2014, maybe there are good test sets for the implementation.

If there aren't, then we should make one and share it with crypto community, I think.

gravityblast · 2018-08-09T16:43:24Z

@mandrigin @adambabik @dshulyak @corpetty I added tests and documentation in the PR description

corpetty · 2018-08-09T16:46:53Z

extkeys/hdkey_test.go

@@ -111,6 +111,21 @@ func TestBIP32Vectors(t *testing.T) {
 			"xpub6FnCn6nSzZAw5Tw7cgR9bi15UV96gLZhjDstkXXxvCLsUXBGXPdSnLFbdpq8p9HmGsApME5hQTZ3emM2rnY5agb9rXpVGyy3bdW6EEgAtqt",
 			"xprvA2nrNbFZABcdryreWet9Ea4LvTJcGsqrMzxHx98MMrotbir7yrKCEXw7nadnHM8Dq38EGfSh6dqA9QWTyefMLEcBYJUuekgW4BYPJcr9E7j",
 		},
+		// Test vector 3


can you reference where this came from for future folks looking at this?

good point @corpetty, I'm adding it to the PR description, it's from here:

https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#test-vector-3

PR descriptions/commit messages are pretty ephemeral. Let's put all the references in the code so no refactorings can destroy it!

@mandrigin I added more comments in the code taken from the PR description

hey @mandrigin @corpetty is there anything else you want to change in this PR?

adambabik · 2018-08-11T22:10:07Z

Please merge develop. The CI should pass normally for this PR before merging.

add bytes padding in derived keys

5a5328a

gravityblast added the security label Aug 6, 2018

dshulyak reviewed Aug 7, 2018

View reviewed changes

mandrigin suggested changes Aug 8, 2018

View reviewed changes

oskarth requested a review from corpetty August 8, 2018 09:25

corpetty requested changes Aug 8, 2018

View reviewed changes

gravityblast added 3 commits August 9, 2018 17:39

add 3rd test vector taken from BIP32 specs

19d1bed

test specific case where keyData is less then 32 bytes

ac3e8b0

add comment on key data padding snippet

a55d387

gravityblast changed the title ~~[WIP] fix child keys derivation adding bytes padding~~ fix child keys derivation adding bytes padding Aug 9, 2018

corpetty reviewed Aug 9, 2018

View reviewed changes

add more docs comments

edd3416

adambabik approved these changes Aug 11, 2018

View reviewed changes

Merge branch 'develop' into fix/bip32-keys-derivation

69f0b15

goranjovic mentioned this pull request Aug 14, 2018

test status-go bip32 fix on status-react side status-im/status-mobile#5495

Closed

corpetty approved these changes Aug 14, 2018

View reviewed changes

adambabik merged commit fc3978a into develop Sep 4, 2018

adambabik deleted the fix/bip32-keys-derivation branch September 4, 2018 07:42

chadyj mentioned this pull request Sep 6, 2018

Recovering the same account with a different password is stuck status-im/status-mobile#5745

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix child keys derivation adding bytes padding #1139

fix child keys derivation adding bytes padding #1139

gravityblast commented Aug 6, 2018 •

edited

Loading

dshulyak Aug 7, 2018

gravityblast Aug 7, 2018

dshulyak commented Aug 7, 2018

gravityblast commented Aug 7, 2018

gravityblast commented Aug 7, 2018 •

edited

Loading

gravityblast commented Aug 7, 2018

gravityblast commented Aug 7, 2018 •

edited

Loading

mandrigin left a comment

adambabik commented Aug 8, 2018

status-im-bot commented Aug 8, 2018

corpetty commented Aug 8, 2018

corpetty left a comment

gravityblast commented Aug 8, 2018

mandrigin commented Aug 8, 2018

gravityblast commented Aug 9, 2018

corpetty Aug 9, 2018

gravityblast Aug 9, 2018

mandrigin Aug 9, 2018

gravityblast Aug 10, 2018

gravityblast Aug 14, 2018

adambabik commented Aug 11, 2018

fix child keys derivation adding bytes padding #1139

fix child keys derivation adding bytes padding #1139

Conversation

gravityblast commented Aug 6, 2018 • edited Loading

Problem

Changes

Links

dshulyak Aug 7, 2018

Choose a reason for hiding this comment

gravityblast Aug 7, 2018

Choose a reason for hiding this comment

dshulyak commented Aug 7, 2018

gravityblast commented Aug 7, 2018

gravityblast commented Aug 7, 2018 • edited Loading

gravityblast commented Aug 7, 2018

gravityblast commented Aug 7, 2018 • edited Loading

mandrigin left a comment

Choose a reason for hiding this comment

adambabik commented Aug 8, 2018

status-im-bot commented Aug 8, 2018

corpetty commented Aug 8, 2018

corpetty left a comment

Choose a reason for hiding this comment

gravityblast commented Aug 8, 2018

mandrigin commented Aug 8, 2018

gravityblast commented Aug 9, 2018

corpetty Aug 9, 2018

Choose a reason for hiding this comment

gravityblast Aug 9, 2018

Choose a reason for hiding this comment

mandrigin Aug 9, 2018

Choose a reason for hiding this comment

gravityblast Aug 10, 2018

Choose a reason for hiding this comment

gravityblast Aug 14, 2018

Choose a reason for hiding this comment

adambabik commented Aug 11, 2018

gravityblast commented Aug 6, 2018 •

edited

Loading

gravityblast commented Aug 7, 2018 •

edited

Loading

gravityblast commented Aug 7, 2018 •

edited

Loading