Fix signature buffer size for RSA keys

[cmurphy]

When using the pyOpenSSL crypto module to sign data using a large key,
e.g. 8192 bit, a memory allocation error occurs. A test case to show
this, which comes from OpenStack Glance, is:

  $ openssl genrsa -out server.key 8192
  $ ...
  $ cat test.py
  from OpenSSL import crypto
  import uuid
  key_file = 'server.key'
  with open(key_file, 'r') as keyfile:
      key_str = keyfile.read()
  key = crypto.load_privatekey(crypto.FILETYPE_PEM, key_str)
  data = str(uuid.uuid4())
  digest = 'sha256'
  crypto.sign(key, data, digest)
  $ python test.py
  *** Error in `python': free(): invalid next size (normal): 0x0000000002879050 ***
  Aborted

Other errors that may appear to the user are:

  Segmentation Fault

  *** Error in `python': double free or corruption (!prev): 0x0000000001245300 ***
  Aborted

  *** Error in `python': munmap_chunk(): invalid pointer: 0x0000000001fde540 ***
  Aborted

The reason this happens is that the sign function of the crypto module
hard-codes the size of the signature buffer to 512 bytes (4096 bits).
An RSA key generates a signature that can be up to the size of the
private key modulus, so for an 8192 bit key, a buffer for a 4096 bit
signature is too short and causes a memory allocation error.

Technically the maximum size key this code should be able to handle is
4096 bits, but due to memory allocation alignment the problem only
becomes apparent for keys of at least 4161 bits.

This patch does two things. First, it determines the correct size of
the signature buffer, in bytes, based on the real size of the private
key, and passes that the buffer allocation instead of the static number
512. Second, it no longer passes in a signature length. This is because
the OpenSSL EVP_SignFinal function uses this argument as an output and
completely ignores it as an input[1], so there is no need for us to set
it.

This is only a problem for RSA keys, and this patch only affects RSA
keys. For DSA keys, the key size is restricted to 1024 bits (128
bytes), and the signature a DSA key will generate will be about 46
bytes, so this buffer will still be big enough for DSA signatures.

[1] https://github.com/openssl/openssl/blob/349807608f31b20af01a342d0072bb92e0b036e2/crypto/evp/p_sign.c#L74

[alex]

oof, this is awful. can you add a test case for this?

[alex]

(To be clear, the awful thing is our code). Thanks for working on this!

[alex]

I think probably you want pkey_length = (pkey.bits() + 7) // 8, otherwise rounding down will get you for a key that's not a multiple of 8 bits.

[cmurphy]

Thanks for checking this @alex. I've added a test and fixed the rounding error you pointed out.

[reaperhulk]

LGTM, once tests pass we can merge and it will be in the next release (which should be soon!).

Thanks!

[codecov-io]

Current coverage is 87.72%

Merging #428 into master will not affect coverage as of a2308b4

@@            master    #428   diff @@
======================================
  Files            7       7       
  Stmts         2045    2045       
  Branches       377     377       
  Methods          0       0       
======================================
  Hit           1794    1794       
  Partial        121     121       
  Missed         130     130       

Review entire Coverage Diff as of a2308b4

Powered by Codecov. Updated on successful CI builds.

[Lukasa]

LGTM as well, great catch @cmurphy!

[cmurphy]

Thanks!

[hynek]

Can someone please add a change log entry? (And maybe replace :py:obj: thru something more specific without a py prefix?)

I'd do it myself but I'm on a phone in the Moroccan desert.

[nagyv]

Hi,

indeed a great catch, and fix. Thanks! I'm eagerly waiting for it being released.

Is there a planned release date with the above fix being included?

[Lukasa]

@nagyv I believe there are plans to release 0.16 soon™. Last I heard you might get something late this week or early next, but we promise nothing. =)

[hynek]

It’s 16.0 and we’re waiting for cryptography 1.3 because we need its bindings. :)