[Bug-Candidate]: Fail to scan in a project that supports both Hardhat and Foundry

[0xmichalis]

Describe the issue:

We have a project where we use Hardhat and Foundry in parallel. It seems that when Slither detects a foundry.toml file in the project and the Slither config specifies that compilation should be ignored, then if Slither does not find build artifacts produced by Foundry it will fail, although there may be build artifacts by Hardhat. Ideally Slither detects the Hardhat artifacts, or there is a way for us to trigger Slither to look for the Hardhat artifacts.

Code example to reproduce the issue:

git clone https://github.com/ToucanProtocol/contracts
cd contracts
yarn
yarn hardhat compile
slither .

Version:

0.8.3

Relevant log output:

$ slither .
--ignore-compile used, if something goes wrong, consider removing the ignore compile flag
Traceback (most recent call last):
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 744, in main_impl
    ) = process_all(filename, args, detector_classes, printer_classes)
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 76, in process_all
    compilations = compile_all(target, **vars(args))
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 643, in compile_all
    compilations.append(CryticCompile(target, **kwargs))
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 131, in __init__
    self._compile(**kwargs)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 553, in _compile
    self._platform.compile(self, **kwargs)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/platform/foundry.py", line 90, in compile
    hardhat_like_parsing(crytic_compile, self._target, build_directory, self._target)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/platform/hardhat.py", line 49, in hardhat_like_parsing
    os.listdir(build_directory), key=lambda x: os.path.getmtime(Path(build_directory, x))
FileNotFoundError: [Errno 2] No such file or directory: 'out/build-info'
None
Error in .
Traceback (most recent call last):
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 744, in main_impl
    ) = process_all(filename, args, detector_classes, printer_classes)
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 76, in process_all
    compilations = compile_all(target, **vars(args))
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 643, in compile_all
    compilations.append(CryticCompile(target, **kwargs))
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 131, in __init__
    self._compile(**kwargs)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 553, in _compile
    self._platform.compile(self, **kwargs)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/platform/foundry.py", line 90, in compile
    hardhat_like_parsing(crytic_compile, self._target, build_directory, self._target)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/platform/hardhat.py", line 49, in hardhat_like_parsing
    os.listdir(build_directory), key=lambda x: os.path.getmtime(Path(build_directory, x))
FileNotFoundError: [Errno 2] No such file or directory: 'out/build-info'

[0xalpharush]

Hi please upgrade to the latest slither version 0.10.0 as the foundry compilation is more robust. Let me know if that doesn't fix the issue

[0xmichalis]

@0xalpharush it looks like the issue persists in 0.10.0

$ slither .
--ignore-compile used, if something goes wrong, consider removing the ignore compile flag
Traceback (most recent call last):
  File "/home/michalis/.local/bin/slither", line 8, in <module>
    sys.exit(main())
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 727, in main
    main_impl(all_detector_classes=detectors, all_printer_classes=printers)
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 833, in main_impl
    ) = process_all(filename, args, detector_classes, printer_classes)
  File "/home/michalis/.local/lib/python3.9/site-packages/slither/__main__.py", line 96, in process_all
    compilations = compile_all(target, **vars(args))
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 718, in compile_all
    compilations.append(CryticCompile(target, **kwargs))
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 207, in __init__
    self._compile(**kwargs)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/crytic_compile.py", line 629, in _compile
    self._platform.compile(self, **kwargs)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/platform/foundry.py", line 86, in compile
    hardhat_like_parsing(crytic_compile, self._target, build_directory, self._target)
  File "/home/michalis/.local/lib/python3.9/site-packages/crytic_compile/platform/hardhat.py", line 52, in hardhat_like_parsing
    raise InvalidCompilation(txt)
crytic_compile.platform.exceptions.InvalidCompilation: Compilation failed. Can you run build command?
out/build-info is not a directory.

[elopez]

@0xmichalis when both Foundry and Hardhat are present, crytic-compile chooses Foundry as it's usually more robust. You can however override this automatic choice and opt to use Hardhat by using --compile-force-framework hardhat on the command line or compile_force_framework on the config file.

[0xalpharush]

^ Using ignore_compile will disable compilation altogether so it should be removed from the config.
https://github.com/ToucanProtocol/contracts/blob/530baaf585b91ba172ce0ea1b94352c60fc7bf85/slither.config.json#L6

[0xmichalis]

We cannot avoid ignore_compile because of #2026

[elopez]

Both options should work well together, as long as you run a build with the chosen framework before running Slither.

[0xalpharush]

@0xmichalis Your comment references hardhat. I tested with the config removed and slither worked. Is there a reason using foundry for building can not be done instead (as is done by default when both frameworks are detected)?

EDIT:
You can see test files are ignored

'forge clean' running (wd: /Users/alpharush/contracts)
'forge build --build-info --skip */test/** */script/** --force' running (wd: /Users/alpharush/contracts)

[0xmichalis]

@0xmichalis Your comment references hardhat. I tested with the config removed and slither worked. Is there a reason using foundry for building can not be done instead (as is done by default when both frameworks are detected)?

@0xalpharush I guess you are referencing #2026 (comment) ? We can use forge just fine with Slither, it's only Hardhat that does not work.

[0xalpharush]

Can you explain the issue you're experiencing now that it is building? I can run slither . --compile-force-framework hardhat with the ignore_compile removed in addition to forge working.

[0xmichalis]

Can you explain the issue you're experiencing now that it is building? I can run slither . --compile-force-framework hardhat with the ignore_compile removed in addition to forge working.

Ah sorry about that but the code that requires path filtering is not currently pushed to the repo I have provided in the reproducer for the current issue and this really mixes two different issues, right?

[elopez]

@0xmichalis @0xalpharush slither . --compile-force-framework hardhat --ignore-compile should also work if you've run npx hardhat compile manually, or is that not the case? I'm pretty sure ignore_compile only inhibits running npx hardhat compile.

[0xalpharush]

Go ahead and describe the issue with path filtering here, please. To answer your question, we would need some way to tell hardhat not to compile files yet still successfully resolve dependencies and compile to perform filtering ahead of analysis.

Fwiw, I realize it may be annoying that we force recompilation, but often times the build framework retain outdated references (a file offset that was invalidated by editing the file).

[0xmichalis]

@0xmichalis @0xalpharush slither . --compile-force-framework hardhat --ignore-compile should also work if you've run npx hardhat compile manually, or is that not the case? I'm pretty sure ignore_compile only inhibits running npx hardhat compile.

Ah, yes, this works fine with 0.10.0. Pretty sure I had tried with 0.8.3 and it didn't work.

[0xmichalis]

Go ahead and describe the issue with path filtering here, please.

We have a contracts/testing directory that would need to be excluded from scanning because Slither was failing with #1352 before it could scan any contracts but it seems fixed in 0.10.0 so we can most likely get rid of path filtering altogether.

To answer your #2026 (comment), we would need some way to tell hardhat not to compile files yet still successfully resolve dependencies and compile to perform filtering ahead of analysis.

Cool, that explains why tricks like this are not enough and I would still see a Slither error.

[0xmichalis]

Closing as seems that we just need to upgrade to the latest version :)

Thanks for the prompt responses!