scancode: broken today due to a python error an unexpected keyword argument 'update_min_steps

[0xc0170]

Description of defect

Few PRs are red today due to breakage in scancode, see : https://github.com/ARMmbed/mbed-os/pull/15022/checks?check_run_id=3445020605

The error is:

+ scancode -l --json-pp scancode.json SCANCODE
/usr/local/lib/python3.8/dist-packages/cluecode/copyrights.py:3355: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
/usr/local/lib/python3.8/dist-packages/pygmars/lex.py:95: FutureWarning: Possible nested set at position 7
  re.compile(m, flags=re_flags).match
Setup plugins...
Collect file inventory...
Scan files for: licenses with 1 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
  File "/usr/local/bin/scancode", line 8, in <module>
    sys.exit(scancode())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/commoncode/cliutils.py", line 69, in main
    return click.Command.main(
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scancode/cli.py", line 432, in scancode
    success, _results = run_scan(
  File "/usr/local/lib/python3.8/dist-packages/scancode/cli.py", line 865, in run_scan
    scan_success = run_scanners(
  File "/usr/local/lib/python3.8/dist-packages/scancode/cli.py", line 1103, in run_scanners
    scan_success = scan_codebase(
  File "/usr/local/lib/python3.8/dist-packages/scancode/cli.py", line 1202, in scan_codebase
    scans = progress_manager(scans)
  File "/usr/local/lib/python3.8/dist-packages/commoncode/cliutils.py", line 270, in progressmanager
    return progress_class(
TypeError: __init__() got an unexpected keyword argument 'update_min_steps'
Error: Process completed with exit code 1.

scancode-toolkit recent release is 5th of August, so it must be somewhere in its dependencies. Although their requirements are locked to specific versions https://github.com/nexB/scancode-toolkit/blob/develop/requirements.txt

Target(s) affected by this defect ?

all

Toolchain(s) (name and version) displaying this defect ?

all

What version of Mbed-os are you using (tag or sha) ?

mbed-os-6.14

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

not relevant

How is this defect reproduced ?

Create a new pull request and it will become red with this failure.

cc @ARMmbed/mbed-os-test

[jeromecoutant]

If you patch the scancode, what about adding a filter */STM32Cube_FW/* files?
#15022 (comment)

[0xc0170]

That's unrelated. I should create a new issue to fix that SPDX checking , I'll do it today.

[0xc0170]

I am not able to reproduce locally.

My setup:

Python 3.9.7
scancode_toolkit-21.8.4

I executed scancode on mbed-os/cmsis folder scancode -l --json-pp scancode.json cmsis

Scanning done.
Summary:        licenses with 1 process(es)
Errors count:   0
Scan Speed:     0.70 files/sec.
Initial counts: 186 resource(s): 130 file(s) and 56 directorie(s)
Final counts:   186 resource(s): 130 file(s) and 56 directorie(s)
Timings:
  scan_start: 2021-08-31T124938.245222
  scan_end:   2021-08-31T125244.967945
  setup_scan:licenses: 1.77s
  setup: 1.78s
  inventory: 0.32s
  scan: 184.59s
  output:json-pp: 0.14s
  output: 0.14s
  total: 186.87s
Removing temporary files...done.

[0xc0170]

The only similar reference I was able to found: aboutcode-org/scancode-toolkit#636 (comment)

The error comes from sys.exit() in scancode.. Is this related rather to Docker and env there?

Or is it the latest python 3.8.12 release (released 30.8, so 2 days ago, about the time it started being red).

[jamesbeyond]

Seems it is because of the click module :
Scancode will use click 8.x,
not sure why mbed-tools restrict click module to 7.x

ERROR: mbed-tools 7.31.1 has requirement Click<8,>=7.1, but you'll have click 8.0.1 which is incompatible.

after update click scancode works

[jamesbeyond]

This would fix the GHA CI: #15043

[0xc0170]

Fixed now on master