Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.9.0: tests make build non-reproducible #2324

Closed
1 of 3 tasks
dvzrv opened this issue Jul 29, 2020 · 12 comments · Fixed by #2560
Closed
1 of 3 tasks

3.9.0: tests make build non-reproducible #2324

dvzrv opened this issue Jul 29, 2020 · 12 comments · Fixed by #2560
Assignees
Labels
kind: bug solution: proposed fix a fix for the issue has been proposed and waits for confirmation

Comments

@dvzrv
Copy link

dvzrv commented Jul 29, 2020

When building 3.9.0 as a package (using the test-data 3.0.0) for Arch Linux I ran our reproducible builds tools against the resulting package.
Beforehand I had to apply #2318 as install of the pkgconfig file fails otherwise.

What is the issue you have?

Running the tests (ctest --output-on-failure -LE git_required) after build renders the package unreproducible and alters the pkgconfig file to change cflags from -I/usr/include to -I/usr/local/include (amongst others).

Please describe the steps to reproduce the issue.

  cmake -DCMAKE_INSTALL_PREFIX='/usr' \
        -DCMAKE_INSTALL_LIBDIR='/usr/lib' \
        -DCMAKE_BUILD_TYPE='None' \
        -DBUILD_TESTING=ON \
        -DJSON_BuildTests=ON \
        -DJSON_MultipleHeaders=ON \
        -Wno-dev \
        -B build \
        -S .
  make VERBOSE=1 -C build
  ctest --output-on-failure -LE git_required
  make DESTDIR="${pkgdir}" install -C build

After a package has been created (building in a clean systemd-nspawn container), the package is rebuilt using the exact same dependencies with the repro tool.

Can you provide a small but working code example?

n/a

What is the expected behavior?

Package is reproducible. Tests do not alter the build and resulting installed files.

And what is the actual behavior instead?

This is the output of diffoscope on the two packages:

--- nlohmann-json-3.9.0-1-any.pkg.tar.zst
+++ build/nlohmann-json-3.9.0-1-any.pkg.tar.zst
├── nlohmann-json-3.9.0-1-any.pkg.tar
│ ├── file list
│ │ @@ -1,9 +1,9 @@
│ │  -rw-r--r--   0 root         (0) root         (0)     4659 2020-07-29 06:45:59.000000 .BUILDINFO
│ │ --rw-r--r--   0 root         (0) root         (0)     3854 2020-07-29 06:45:59.000000 .MTREE
│ │ +-rw-r--r--   0 root         (0) root         (0)     3858 2020-07-29 06:45:59.000000 .MTREE
│ │  -rw-r--r--   0 root         (0) root         (0)      337 2020-07-29 06:45:59.000000 .PKGINFO
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/include/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/include/nlohmann/
│ │  -rw-r--r--   0 root         (0) root         (0)     1504 2020-07-29 06:45:59.000000 usr/include/nlohmann/adl_serializer.hpp
│ │  -rw-r--r--   0 root         (0) root         (0)     4633 2020-07-29 06:45:59.000000 usr/include/nlohmann/byte_container_with_subtype.hpp
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/include/nlohmann/detail/
│ │ @@ -50,17 +50,17 @@
│ │  -rw-r--r--   0 root         (0) root         (0)    79525 2020-07-29 06:45:59.000000 usr/include/nlohmann/thirdparty/hedley/hedley.hpp
│ │  -rw-r--r--   0 root         (0) root         (0)     4981 2020-07-29 06:45:59.000000 usr/include/nlohmann/thirdparty/hedley/hedley_undef.hpp
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/cmake/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/
│ │  -rw-r--r--   0 root         (0) root         (0)      620 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonConfig.cmake
│ │  -rw-r--r--   0 root         (0) root         (0)      675 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonConfigVersion.cmake
│ │ --rw-r--r--   0 root         (0) root         (0)     3599 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
│ │ +-rw-r--r--   0 root         (0) root         (0)     3293 2020-07-29 06:45:59.000000 usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/lib/pkgconfig/
│ │ --rw-r--r--   0 root         (0) root         (0)       97 2020-07-29 06:45:59.000000 usr/lib/pkgconfig/nlohmann_json.pc
│ │ +-rw-r--r--   0 root         (0) root         (0)       91 2020-07-29 06:45:59.000000 usr/lib/pkgconfig/nlohmann_json.pc
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/doc/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/doc/nlohmann-json/
│ │  -rw-r--r--   0 root         (0) root         (0)     3213 2020-07-29 06:45:59.000000 usr/share/doc/nlohmann-json/CODE_OF_CONDUCT.md
│ │  -rw-r--r--   0 root         (0) root         (0)    92324 2020-07-29 06:45:59.000000 usr/share/doc/nlohmann-json/README.md
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/licenses/
│ │  drwxr-xr-x   0 root         (0) root         (0)        0 2020-07-29 06:45:59.000000 usr/share/licenses/nlohmann-json/
│ ├── .MTREE
│ │ ├── .MTREE-content
│ │ │ @@ -1,11 +1,11 @@
│ │ │  #mtree
│ │ │  /set type=file uid=0 gid=0 mode=644
│ │ │  ./.BUILDINFO time=1596005159.0 size=4659 md5digest=63c650b55cdce35f19383793997e9f66 sha256digest=ddc54c9966f2bdd79f897309ed4fef322b885cfa94ba0b69e244e6c5c20cf8bc
│ │ │ -./.PKGINFO time=1596005159.0 size=337 md5digest=62f9e8e0245fd2e8ecebde2373d1bce5 sha256digest=051095f02018bb81a86be4e3baebd3089f975e329d1edf7fc5edc0ed3a48f5f0
│ │ │ +./.PKGINFO time=1596005159.0 size=337 md5digest=cbe4f20312aee609d4c4299824b7e7a0 sha256digest=619dae38f84bdc895d1a043e3247602896bdd37ff635affef83dd3f36f9746cd
│ │ │  /set mode=755
│ │ │  ./usr time=1596005159.0 type=dir
│ │ │  ./usr/include time=1596005159.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/include/nlohmann time=1596005159.0 mode=755 type=dir
│ │ │  ./usr/include/nlohmann/adl_serializer.hpp time=1596005159.0 size=1504 md5digest=6681125504fe517fb4349916c7ca3ad5 sha256digest=9a09ae189930ebf891942ce0fb3b10674a0ae8d6e3c1db618d46c5357960c526
│ │ │  ./usr/include/nlohmann/byte_container_with_subtype.hpp time=1596005159.0 size=4633 md5digest=3773ad72c762eeada43f95ac0a0351ed sha256digest=9ceb0600c83ae0ec49701a6fc0aaa2f4e30f4dfd1120fb7d513f15b4c6c9cc37
│ │ │ @@ -55,17 +55,17 @@
│ │ │  /set mode=755
│ │ │  ./usr/lib time=1596005159.0 type=dir
│ │ │  ./usr/lib/cmake time=1596005159.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/lib/cmake/nlohmann_json time=1596005159.0 mode=755 type=dir
│ │ │  ./usr/lib/cmake/nlohmann_json/nlohmann_jsonConfig.cmake time=1596005159.0 size=620 md5digest=19b85ca4287a9d187a335feafbf45568 sha256digest=c7f377920b46cd6c89c6f2f1bccbd1cf7380addf545db0bf7046a27920e2bdd3
│ │ │  ./usr/lib/cmake/nlohmann_json/nlohmann_jsonConfigVersion.cmake time=1596005159.0 size=675 md5digest=bfc688b969c471dd5dcb38f6d9e823c7 sha256digest=ca77a7cb7c35cff0fd7e180b5f5a90955d0fa6a3b72909eccb4bcd62b36f449b
│ │ │ -./usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake time=1596005159.0 size=3599 md5digest=fc37818baa390ff80790e74713840ee7 sha256digest=427752e51feeefacb23ce487de7f888a552ea3181e0951600683814e68f676dd
│ │ │ +./usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake time=1596005159.0 size=3293 md5digest=a408eb50f0acc70cadd0e0cab904f0d5 sha256digest=cc206c5bd4cc8ec064d6174d7664377f8dcd5ed280476c4b7473154d0996bb11
│ │ │  ./usr/lib/pkgconfig time=1596005159.0 mode=755 type=dir
│ │ │ -./usr/lib/pkgconfig/nlohmann_json.pc time=1596005159.0 size=97 md5digest=8bdec2b26a228ff226cc21cf26937e84 sha256digest=a30c5a41abc1efef47ed270154cdeddf9790c838e219b164b03700b318c4690b
│ │ │ +./usr/lib/pkgconfig/nlohmann_json.pc time=1596005159.0 size=91 md5digest=8a8567f3f10d22c334c5804521a33788 sha256digest=eb5f758dfd576d338954329a2eb4e9b6f8518bf107126b8acae19b5c33388a8e
│ │ │  /set mode=755
│ │ │  ./usr/share time=1596005159.0 type=dir
│ │ │  ./usr/share/doc time=1596005159.0 type=dir
│ │ │  /set mode=644
│ │ │  ./usr/share/doc/nlohmann-json time=1596005159.0 mode=755 type=dir
│ │ │  ./usr/share/doc/nlohmann-json/CODE_OF_CONDUCT.md time=1596005159.0 size=3213 md5digest=0227ed5d32159cf5923d5b71e8c82c12 sha256digest=3f82f9935dbdd87919ec79bf0f2a43232f2cfe82040236900652458ffff77783
│ │ │  ./usr/share/doc/nlohmann-json/README.md time=1596005159.0 size=92324 md5digest=07df80351c6c3d2f1aeb8b5a0ef87f1d sha256digest=174c18ebde64965cac98c147204e1e2c33714f4d93242bf375a2923ea9a37f71
│ ├── .PKGINFO
│ │ @@ -3,12 +3,12 @@
│ │  pkgname = nlohmann-json
│ │  pkgbase = nlohmann-json
│ │  pkgver = 3.9.0-1
│ │  pkgdesc = JSON for Modern C++
│ │  url = https://github.com/nlohmann/json
│ │  builddate = 1596005159
│ │  packager = David Runge <[email protected]>
│ │ -size = 1025469
│ │ +size = 1025157
│ │  arch = any
│ │  license = MIT
│ │  makedepend = cmake
│ │  makedepend = git
│ ├── usr/lib/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
│ │ @@ -37,22 +37,16 @@
│ │    message(FATAL_ERROR "Some (but not all) targets in this export set were already defined.\nTargets Defined: ${_targetsDefined}\nTargets not yet defined: ${_targetsNotDefined}\n")
│ │  endif()
│ │  unset(_targetsDefined)
│ │  unset(_targetsNotDefined)
│ │  unset(_expectedTargets)
│ │  
│ │  
│ │ -# Compute the installation prefix relative to this file.
│ │ -get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
│ │ -get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
│ │ -get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
│ │ -get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
│ │ -if(_IMPORT_PREFIX STREQUAL "/")
│ │ -  set(_IMPORT_PREFIX "")
│ │ -endif()
│ │ +# The installation prefix configured by this project.
│ │ +set(_IMPORT_PREFIX "/usr")
│ │  
│ │  # Create imported target nlohmann_json::nlohmann_json
│ │  add_library(nlohmann_json::nlohmann_json INTERFACE IMPORTED)
│ │  
│ │  set_target_properties(nlohmann_json::nlohmann_json PROPERTIES
│ │    INTERFACE_COMPILE_DEFINITIONS "JSON_USE_IMPLICIT_CONVERSIONS=\$<BOOL:ON>"
│ │    INTERFACE_COMPILE_FEATURES "cxx_std_11"
│ ├── usr/lib/pkgconfig/nlohmann_json.pc
│ │ @@ -1,4 +1,4 @@
│ │  Name: nlohmann_json
│ │  Description: JSON for Modern C++
│ │  Version: 3.9.0
│ │ -Cflags: -I/usr/local/include
│ │ +Cflags: -I/usr/include

When not running the tests the package is reproducible and the above discrepancy can not be found.

Which compiler and operating system are you using?

  • Compiler: gcc 10.1.0
  • Operating system: Arch Linux
  • cmake: 3.18.0

Which version of the library did you use?

  • latest release version 3.9.0
  • other release - please state the version: ___
  • the develop branch

If you experience a compilation error: can you [compile and run the unit tests]

n/a

@nlohmann
Copy link
Owner

Could you please try with the latest develop version - we recently fixed two bugs introduced with the pkg-config file: #2314 and #2318. This is just a guess.

@dvzrv
Copy link
Author

dvzrv commented Jul 29, 2020

@nlohmann #2318 is applied, as otherwise installation would fail after running the tests.

@nlohmann
Copy link
Owner

So the problem is that the pkg-config and the cmake file contains the name of the binary directory?

@dvzrv
Copy link
Author

dvzrv commented Jul 29, 2020

The problem is, that the tests change the contents of those files. This is even the case, if I make a copy of the build directory (e.g. cp -av build build-test) and run ctest in the copy (i.e. in build-test).
This should not happen.

@nlohmann
Copy link
Owner

Is there a way I can reproduce this without Arch Linux?

@dvzrv
Copy link
Author

dvzrv commented Aug 4, 2020

I found a solution that is not pretty at all, but gets around the problem of the tests interfering with the packaged files:
I am now building twice (once for the installation, once for the tests) and then run the tests against the test build.

AFAICS the tests output things into the destination/output directory which overwrites previously set configuration and/or alters previously set configuration directly (see mention of nlohmann_json.pc in the initial comment). This has also been mentioned in #2318.

Test data should never leak into whatever is installed to the system!

Is there a way I can reproduce this without Arch Linux?

I'm not entirely sure. Other distributions also offer reproducible builds tools. I can't really provide any guidance for other distributions unfortunately.

Apart from that you could look at the nlohmann_json.pc file when setting CMAKE_INSTALL_PREFIX to '/usr'. When not running the tests and installing, it will correctly set the Cflags to -I/usr/include. If the tests are run, the output directory is overwritten with test data and Cflags is set to -I/usr/local/include (ouf).

If you want to try Arch Linux in a virtual machine, we provide it as a vagrant box:

vagrant init archlinux/archlinux
vagrant up
vagrant ssh

In the VM, you can prepare the system for building packages, import your own GPG public key and get the PKGBUILD for the package:

sudo pacman -Syu archlinux-repro asp base-devel devtools
gpg --recv-keys '797167AE41C0A6D9232E48457F3CEA63AE251B69'
asp checkout nlohmann-json
cd nlohmann-json/trunk

Here you'd have to modify the PKGBUILD to again run the tests against the initial build directory (i.e. change to make -k test -C build in the check() function).
Then you can build the package and run repro on it:

extra-x86_64-build
repro -ndf nlohmann-json-3.9.0-1-any.pkg.tar.zst

@nlohmann
Copy link
Owner

nlohmann commented Aug 5, 2020

I found a solution that is not pretty at all, but gets around the problem of the tests interfering with the packaged files:
I am now building twice (once for the installation, once for the tests) and then run the tests against the test build.

AFAICS the tests output things into the destination/output directory which overwrites previously set configuration and/or alters previously set configuration directly (see mention of nlohmann_json.pc in the initial comment). This has also been mentioned in #2318.

Looking at #2318, it seems that the test cmake_add_subdirectory is the culprit then. Is everything fine when you skip the CMake-based tests then? (just comment the last 5 lines from tests/CMakeLists.txt.

@dvzrv
Copy link
Author

dvzrv commented Aug 6, 2020

Yes, removing the add_subdirectory() calls from the file makes the package reproducible as well.

@nlohmann
Copy link
Owner

@dvzrv Can you please check whether 1047d09 fixes the issue? In particular, whether ctest -LE not_reproducible does the job?

@nlohmann nlohmann added the solution: proposed fix a fix for the issue has been proposed and waits for confirmation label Aug 30, 2020
@stale
Copy link

stale bot commented Oct 4, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the state: stale the issue has not been updated in a while and will be closed automatically soon unless it is updated label Oct 4, 2020
@stale stale bot closed this as completed Oct 12, 2020
@nlohmann
Copy link
Owner

@dvzrv Did you check the fix I proposed in #2324 (comment) ?

@nlohmann nlohmann reopened this Dec 30, 2020
@nlohmann nlohmann self-assigned this Dec 31, 2020
@nlohmann nlohmann added this to the Release 3.9.2 milestone Dec 31, 2020
@nlohmann nlohmann removed the state: stale the issue has not been updated in a while and will be closed automatically soon unless it is updated label Dec 31, 2020
@dvzrv
Copy link
Author

dvzrv commented May 8, 2021

@nlohmann sorry for not getting back to you on this! I got swamped with other things and forgot to reply eventually.

Version 3.9.1 definitely fixes this issue. Many thanks for looking into this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind: bug solution: proposed fix a fix for the issue has been proposed and waits for confirmation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants