Compile_tb.py generate options not recogised by xrun

[cern143]

Observed Behavior

compile_tb_stdstreams.log

Expected Behavior

Steps to reproduce the issue

My Environment

EDA tool and version:

Xcelium 23.09-s001

Operating system:

Centos 7.9

Version of the Ibex source code:

6682336

[rswarbrick]

To save other people having to download the log file (which is only 3 lines long!), it contains:

xrun: *SE,BDOPT: Unknown option -rpath,/lib.
Running compile_tb command :
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker -rpath,/lib -lstdc++

[rswarbrick]

I think this form of the -rpath argument is coming from _get_iss_pkgconfig_flags in compile_tb.py. Pinging @hcallahan-lowrisc, who might be able to help (given that he wrote the code there)

[hcallahan-lowrisc]

Hi @cern143

There has been a few issues with CentOS7 support in the past, so this could possibly be related to that. I can't reproduce this locally right now. (I am using NixOS, but I can also confirm it works fine on Ubuntu). But could I just check a few things with you first before we go down that route...

The xrun invocation I get locally looks as follows:

$ make --keep-going IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0

# compile_tb_stdstreams.log
<...>
xrun -64bit -q -f /home/harry/projects/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/harry/projects/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/harry/projects/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/harry/projects/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include/softfloat -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include -Wld,-Xlinker,-rpath,/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib -lstdc++
<...>

Comparing it to your logfile, this suggests to me that your build has not been able to correctly use pkg-config to resolve the build-time flags pointing at your local spike build. Everywhere my build contains the path /nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0 (the path to my spike build), yours has no path there at all.

Also, strangely the -rpath argument looks to have been formed differently between our systems. Mine resolved to -Wld,-Xlinker,-rpath,<spike_path>/lib, whereas yours resolved to -Wld,-Xlinker -rpath,/lib, not only missing the resolved spike_path but also there is now a space instead of a comma between -Xlinker and -rpath.

To get a bit more information, I'll prepare a branch which prints out a bit more information from the build. In the meantime, could you just confirm a few things.

1. What your version of pkg-config is?

[harry@neptune:~/projects/ibex/dv/uvm/core_ibex]$ pkg-config --version
0.29.2

2. From the build instructions page, you have built spike and setup the PKG_CONFIG_PATH correctly, pointing to a path that exists. e.g.

[harry@neptune:~/projects/ibex/dv/uvm/core_ibex]$ echo $PKG_CONFIG_PATH | tr : '\n' | grep spike
/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib/pkgconfig

Thanks

[cern143]

Hi @hcallahan-lowrisc
Thank for your reply. When i check pkg-config version, i got:

 pkg-config --version
0.27.1

and for PKG_CONFIG_PATH:

 echo $PKG_CONFIG_PATH
/usr/lib/pkgconfig

I can notice our environments are different. Is that ok?
Best regard

[hcallahan-lowrisc]

Okay I see the issue. You need to follow the build instructions page, which describes how to build spike and setup your environment variables appropriately. A part of this is setting PKG_CONFIG_PATH to point towards the spike model you have just built. Without doing this, it is not possible to run the simulations, as we need to link against the spike model to perform cosimulation.
Thanks!

[cern143]

Hi @hcallahan-lowrisc, I rebuilt it in a different directory:

echo $PKG_CONFIG_PATH
/opt/lib/pkgconfig

And still get the same result:

xrun: *SE,BDOPT: Unknown option -rpath,/opt/lib.
Running compile_tb command :
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/opt/lib -I/opt/include -I/opt/include/softfloat -Wld,-Xlinker -rpath,/opt/lib -lstdc++

Im pretty sure i didn't specify the prefix at the first time i built it (which is not a smart move), but .pc files content remain the same.
For more information, i also check the version of riscv-isa-sim

 ❯ git rev-parse HEAD
39612f93837122a980395487f55b2d97d29d70c1
❯ git rev-parse --abbrev-ref HEAD
ibex_cosim

Everything seem fine.
I also tried to edit compile.py to use -Wld option of xcelium and it seem to work, howerver the default g++ of xcelium can't understand -rpath option:

ake[2]: Entering directory `/home/usr14/ibex/dv/uvm/core_ibex'
building library run.so
g++: error: unrecognized command line option '-rpath'
make[2]: *** [/home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/xcelium.d/run.lnx8664.23.09.d/librun.so] Error 1
make[2]: Leaving directory `/home/usr14/ibex/dv/uvm/core_ibex'

Can you tell me what to do next? Best regard.

[hcallahan-lowrisc]

Could you try running the command directly, except with a comma between -Xlinker and -rpath instead of a space? See the message above where I pasted the command generated on my system for an example. I believe the xrun tool is parsing the CLI invocation differently due to this change, and hence g++ is being fed different arguments.
This suggests to me that pkg-config on our systems are somehow returning different outputs. I'll try to put together a branch today which prints some logging output to confirm this.

[cern143]

It is still the same problem:

xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker -rpath,/lib -lstdc++
xrun: *SE,BDOPT: Unknown option -rpath,/lib.

I couldn't understand why that doesn't work

[hcallahan-lowrisc]

Hey @cern143
Could you try just running this command directly, and let me know what happens?

xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker,-rpath,/lib -lstdc++

(I just copied the command from your message, but changed the space for a comma before the -rpath argument)
Thanks

[cern143]

I tried your command @hcallahan-lowrisc, and xrun seem ok with those parameters:

❯ xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker,-rpath,/lib -lstdc++
xrun: *E,FILEMIS: Cannot find the provided file /dv/uvm/core_ibex/common/prim/prim_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_assert.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_util_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_count_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_count.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_secded_pkg.sv.
...

There are more log line but i these are enough to point out the problem, xrun can't find file listed in ibex_dv_cosim_dpi.f

[cern143]

Hey @hcallahan-lowrisc, seem like we fixed it. Just need to edit a line in compile.py (remove space between xlinker and formated string):

_flags = "'-Xlinker{}'".format(_flags.replace('-Wl,', ','))

and it will be ok:

make IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0
Build metadata already exists, not recreating from scratch.
Building RTL testbench
Generating core configuration file
Building randomized test generator
Running randomized test generator to create assembly file /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.S
Compiling riscvdv test assembly to create binary at /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.bin
Running RTL simulation at /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1
Collecting simulation results and checking logs of testcase at /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/trr.yaml
Collecting up results of tests into report regr.log
100.00% PASS 1 PASSED, 0 FAILED

[hcallahan-lowrisc]

Thanks for letting me know @cern143, glad to hear its working now. I've just opened #2209, which should fix this a bit more generally. Would you mind testing it out and confirming that it works for you?

[cern143]

Of course @hcallahan-lowrisc, i had ran your new script and it worked completely fine:

make --keep-going IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0
Build metadata already exists, not recreating from scratch.
Building RTL testbench
Generating core configuration file
Building randomized test generator
Running randomized test generator to create assembly file /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.S
Compiling riscvdv test assembly to create binary at /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.bin
Running RTL simulation at /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1
Collecting simulation results and checking logs of testcase at /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/trr.yaml
Collecting up results of tests into report regr.log
100.00% PASS 1 PASSED, 0 FAILED