pkg-config file should preserve CFLAGS

[deldotbrain]

Prerequisites

• Checked the developer manual
• Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=repo%3Alibtom%2Flibtomcrypt
• Checked that your issue isn't due to the fact that you're using asymmetric cryptography and you forgot linking in and/or setting an MPI provider (usually this causes either random crashes or runtime errors like LTC_ARGCHK 'ltc_mp.name != NULL' failure ...). c.f. Ch. "Math Descriptors" of the developer manual.
• Checked that your issue isn't related to TomsFastMath's limitation that PK operations can by default only be done with max. 2048bit keys

Description

I opened NixOS/nixpkgs#346707 against nixpkgs while packaging an application that used its shared libtomcrypt library. Before I hack around the issue in nixpkgs, I thought I should report it here.

The pkg-config file installed by libtomcrypt doesn't capture the CFLAGS that LTC was built with. Applications using pkg-config to link against a shared LTC library will not automatically build with the same CFLAGS as LTC, which can cause problems. For some flags (e.g LTM_DESC in the application I was packaging), that's a nuisance. (Edit: I see that since the release of 1.18.2, a change was made so the LTM_DESC flag is captured in the pkg-config file)

However, for the LTC_PTHREAD flag specifically, it can cause a major bug. Since LTC_PTHREAD changes the size of struct prng_state, building LTC with it enabled (as nixpkgs does), then using pkg-config to build an application against it causes e.g. rng_make_prng() to overwrite glibc's malloc metadata and crash the program.

Steps to Reproduce

$ make CFLAGS="-DUSE_LTM -DLTM_DESC -DLTC_PTHREAD" EXTRALIBS="-ltommath" -f makefile.shared install

$ cat <<EOF >test.c
#include <stdio.h>
#include <tomcrypt.h>

int main(int argc, char **argv) {
  // Initialize the yarrow PRNG, including mutexes for which no space was
  // allocated:
  int yarrow_wprng = register_prng(&yarrow_desc);
  prng_state* yarrow_state = malloc(sizeof(prng_state));
  if (rng_make_prng(128, yarrow_wprng, yarrow_state, NULL) == -1) {
    fprintf(stderr, "error initializing prng");
    return 1;
  }

  // LTC has now accidentally corrupted glibc's malloc metadata for
  // yarrow_state. Doing another allocation will detect it.
  malloc(1);

  // If prng_state were stack-allocated instead, SSP would detect a
  // stack smashing attempt when main() returns.
  return 0;
}
EOF

$ gcc $(pkg-config --cflags --libs) test.c && ./a.out

The test program will crash and burn with a failed assertion from glibc.

Version

libtomcrypt 1.18.2

GCC 13.3.0 with glibc 2.39-52 on NixOS unstable (24.11)

[levitte]

I've thought of this discrepancy in the last few weeks, but haven't had the peace of mind to figure out a solution.

I'm not at all convinced that the user CFLAGS should be the same as the build CFLAGS. it shouldn't be necessary... but perhaps we'll have to in some sort of interim.