Cannot import safetydb

[rolfschr]

Hi,

I setup everything locally (tests & migrations run fine) and started the import process. I cannot seem to import safetydb. Please find below the error log. The likely culprit is here.

$ DJANGO_DEV=1 python manage.py import --all
Importing data from rust
Successfully imported data from rust
Importing data from alpine
Successfully imported data from alpine
Importing data from archlinux
Successfully imported data from archlinux
Importing data from debian
Successfully imported data from debian
Importing data from safetydb
Traceback (most recent call last):
  File "/nix/store/2ailgd4vdaslsybza4vv7g5phdm48v1x-python3.8-schema-0.7.1/lib/python3.8/site-packages/schema.py", line 393, in validate
    nvalue = Schema(svalue, error=e, ignore_extra_keys=i).validate(value)
  File "/nix/store/2ailgd4vdaslsybza4vv7g5phdm48v1x-python3.8-schema-0.7.1/lib/python3.8/site-packages/schema.py", line 354, in validate
    return type(data)(o.validate(d) for d in data)
  File "/nix/store/2ailgd4vdaslsybza4vv7g5phdm48v1x-python3.8-schema-0.7.1/lib/python3.8/site-packages/schema.py", line 354, in <genexpr>
    return type(data)(o.validate(d) for d in data)
  File "/nix/store/2ailgd4vdaslsybza4vv7g5phdm48v1x-python3.8-schema-0.7.1/lib/python3.8/site-packages/schema.py", line 164, in validate
    raise SchemaError(
schema.SchemaError: Or({'advisory': <class 'str'>, 'cve': Or(None, Regex('CVE-\\d+-\\d+')), 'id': Regex('^pyup.io-\\d'), 'specs': <class 'list'>, 'v': <class 'str'>}) did not validate {'advisory': 'RoundCube in debops 2.0.0 uses the user login and password credentials to authenticate to the SMTP (submission) service before sending e-mail messages. This allows the SMTP server to check the message details, block mail with forged sender address, etc. The default configuration uses encrypted connections to the IMAP and SMTP services to ensure confidentiality and security.', 'cve': '""', 'id': 'pyup.io-26403', 'specs': ['<2.0.0'], 'v': '<2.0.0'}
Key 'cve' error:
Or(None, Regex('CVE-\\d+-\\d+')) did not validate '""'
Regex('CVE-\\d+-\\d+') does not match '""'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/nix/store/a7yb469cn30fjpnvyf7byb6zbq0vdh68-python3.8-django-3.0.7/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/nix/store/a7yb469cn30fjpnvyf7byb6zbq0vdh68-python3.8-django-3.0.7/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/nix/store/a7yb469cn30fjpnvyf7byb6zbq0vdh68-python3.8-django-3.0.7/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/nix/store/a7yb469cn30fjpnvyf7byb6zbq0vdh68-python3.8-django-3.0.7/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/rolfschr/src/vulnerablecode/vulnerabilities/management/commands/import.py", line 69, in handle
    self._import_data(Importer.objects.all(), options['cutoff_date'])
  File "/home/rolfschr/src/vulnerablecode/vulnerabilities/management/commands/import.py", line 105, in _import_data
    ImportRunner(importer, batch_size).run(cutoff_date=cutoff_date)
  File "/home/rolfschr/src/vulnerablecode/vulnerabilities/import_runner.py", line 114, in run
    with data_source:
  File "/home/rolfschr/src/vulnerablecode/vulnerabilities/importers/safety_db.py", line 80, in __enter__
    validate_schema(self._api_response)
  File "/home/rolfschr/src/vulnerablecode/vulnerabilities/importers/safety_db.py", line 62, in validate_schema
    Schema(scheme).validate(advisory_dict)
  File "/nix/store/2ailgd4vdaslsybza4vv7g5phdm48v1x-python3.8-schema-0.7.1/lib/python3.8/site-packages/schema.py", line 397, in validate
    raise SchemaError([message] + x.autos, [e] + x.errors)
schema.SchemaError: Key 'debops' error:
Or({'advisory': <class 'str'>, 'cve': Or(None, Regex('CVE-\\d+-\\d+')), 'id': Regex('^pyup.io-\\d'), 'specs': <class 'list'>, 'v': <class 'str'>}) did not validate {'advisory': 'RoundCube in debops 2.0.0 uses the user login and password credentials to authenticate to the SMTP (submission) service before sending e-mail messages. This allows the SMTP server to check the message details, block mail with forged sender address, etc. The default configuration uses encrypted connections to the IMAP and SMTP services to ensure confidentiality and security.', 'cve': '""', 'id': 'pyup.io-26403', 'specs': ['<2.0.0'], 'v': '<2.0.0'}
Key 'cve' error:
Or(None, Regex('CVE-\\d+-\\d+')) did not validate '""'
Regex('CVE-\\d+-\\d+') does not match '""'

[sbs2001]

@rolfschr you are right I had noticed the issue before too, more precisely https://github.com/pyupio/safety-db/blob/c4f4673861c540659b8995cf23af2915c584e9d3/data/insecure_full.json#L2640 .

A quick fix would be to comment out https://github.com/nexB/vulnerablecode/blob/80f78da812338d5f82a5c12c0561d505df7a327d/vulnerabilities/importers/safety_db.py#L65.

PRoper solution would be to ping safetydb.

[sbs2001]

Created a ticket at safetydb pyupio/safety-db#2323

[sbs2001]

This will be fixed on safetydb's side as mentioned at pyupio/safety-db#2323 (comment) soon

[pombredanne]

I am still facing the issue. Let's keep it open until fixed and IMHO we should either

1. disable pyup for now
2. in all cases we should always be able to cope with problematic data

[sbs2001]

#273 got squashed in c944101 . This commit fixes the importer.

By defaut the safetydb importer is disabled due to licensing issue