Disabling of compiler optimizations needed to prevent non-null pReserved when invoking C_Initialize().

[ximon18]

Environment

Reproducible with Rust pkcs11 crate version v0.5.0 on:

• SoftHSM 2.6.1-2 on Ubuntu 21.10 installed with apt install softhsm, or
• SoftHSM 2.6.1 on Alpine Linux 3.12 installed with RUN apk add softhsm from a Dockerfile

Steps to reproduce

Calling Ctx::Initialize(Some(args)) where args is either CK_C_INITIALIZE_ARGS::default() or:

let mut args = CK_C_INITIALIZE_ARGS::new();
args.CreateMutex = None;
args.DestroyMutex = None;
args.LockMutex = None;
args.UnlockMutex = None;
args.flags = CKF_OS_LOCKING_OK;
ctx.initialize(Some(args)

(note that CK_C_INITIALIZE_ARGS::new() sets args.pReserved = ptr::null_mut() and that CK_C_INITIALIZE_ARGS::default() calls CK_C_INITIALIZE_ARGS::new())

Works in debug builds

When built with cargo build my application works without error.

Fails in release builds

When built with cargo build --release my application fails with error CKR_ARGUMENTS_BAD and in /var/log/ via Syslog logging from SoftHSM I see error message SoftHSM.cpp(436): pReserved must be set to NULL_PTR.

Disabling compiler optimizations helps

Adding this to Cargo.toml and rebuilding my application with cargo build --release "solves" the problem:

[profile.release.package.pkcs11]
opt-level = 0

[jhagborgftx]

I encountered a similar issue today, but more severe. C_Initialize ends up getting called with garbage parameters. Here's the portion of memory where the CK_C_INITIALIZE_ARGS should be, right before C_Initialize is called:

(gdb) x/6a $rdi
0x7fffffffd238: 0x7ffff7b7ee30 <C_OpenSession>  0x7ffff7c29020
0x7fffffffd248: 0x7fffffffd560  0x7fffffffd300
0x7fffffffd258: 0x7ffff7dba670 <__memmove_avx_unaligned_erms>   0x555555692ba0 <_ZN4clap3app6parser6Parser7add_arg17h6a546dc72bff8a9cE>

Not only that, right before C_Initialize gets called, we've already clobbered the stack:

#0  0x00005555556d8f80 in pkcs11::Ctx::initialize ()
#1  0x00007ffff7b7ede0 in ?? () from /usr/lib/softhsm/libsofthsm2.so
#2  0x00007ffff7b7ee30 in ?? () from /usr/lib/softhsm/libsofthsm2.so
#3  0x00007ffff7c29020 in ?? () from /usr/lib/softhsm/libsofthsm2.so
#4  0x00007fffffffd560 in ?? ()
#5  0x00007fffffffd300 in ?? ()
#6  0x00007ffff7dba670 in ?? () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:221 from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000555555692ba0 in ?? ()
#8  0x0000555555977b70 in ?? ()
#9  0x00007fffffffd5d8 in ?? ()

Things seemed to be fine before Ctx::initialize, so I'm suspecting that function has some UB somehow.

This is also on x86_64 Linux, and also only happens with optimizations on.

[mheese]

Adding this to this issue as well: please switch to the cryptoki crate: https://github.com/parallaxsecond/rust-cryptoki