Clarify licensing when using pyup.io's security check

[mattip]

Over at pypa/wheel, I have been following @agronholm's trials and tribulations when trying to fix a regex used internally which may have some side effects. The code is not user facing, but the pyup.io service has determined that this is a security vulnerability and pipenv check is marking the older versions of the package as unsafe. The release with this change had to be yanked, and the maintainer is trying to create a valid release with the fix. The problem is that users have opened new issues like pypa/wheel#480, pypa/wheel#481 due to the pyup.io false positive.

I see that the pipenv check feature has been using pyupio since this commit, so about 5 years. I am curious whether pyup.io has a policy of contributing bounties or PRs to projects that it marks as vulnerable, in an attempt to help overburdened maintainers provide timely solutions. I could not find information on their site, nor any information about the connection between the open source project pipenv and pyup.io. Now that pyup has discontinued their free scans, perhaps pypa/pipenv would like to rethink the pipenv check model or at least publicize how the cooperation with the company helps open source projects.

[matteius]

@mattip We had an initiative with pyup.io to upgrade our version of safety and it was around the time that article was published about discontinued free scans. #5218

I will follow up and find out where we are at with this effort to upgrade and continue running against a free version of the database.

[AndreiPaulau]

Hello Gents,

I'm a bit out of problematic, but have this on my scanner. Could you please advise? Who triggers installation while pipenv lock - virtualenv package?
Does this mean, I need to wait until virtualenv update (virtualenv will grab 0.38 wheel version) and then pipenv update pointed to the updated virtualenv?

Many thanks!

[matteius]

@AndreiPaulau I did not quite understand your question.

[AndreiPaulau]

Hi,

I mistyped here not pipenv lock, but pipenv install.
Still the question was about seeded packages. Despite I have installed wheel version 0.38.1, pipenv install outputs added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1. I was able to fix that only by including the wheel package into Pipfile.
So the question was, who does triggers installing seeded wheel package with version 0.37 and do I have an option to influence on that? Hope I was clear

[matteius]

@AndreiPaulau From what I have been able to tell in the past, it is wired into at least a couple spots in pipenv/requirementslib to do:

            if not pyproject_backend and self.setup_cfg is not None:
                setup_dict = SetupInfo.get_setup_cfg(self.setup_cfg)
                pyproject_backend = get_default_pyproject_backend()
                pyproject_requires = setup_dict.get(
                    "build_requires", ["setuptools", "wheel"]
                )  # type: ignore

Additionally, virtualenv I think is the one that prints out the "added seed packages" message. I found this when I searched about it: https://stackoverflow.com/questions/70460921/how-to-change-seed-packages-versions-when-creating-virtualenv

[matteius]

New version of pipenv just released today, which has the updated safety command 2.0.3 -- please check it out.

[mattip]

I am not sure I understand. I asked if there is any kind of agreement that pipenv has entered into with the company behind pyup.io to support the open source projects they scan for vulnerabilities, or any more information about the connection between pyenv pipenv and pyup.io. I don't see any change in the documentation of the safety command. Am I missing something?

Also: there seems to be a contradiction in the note: it states "Each month, PyUp.io updates the safety database of insecure Python packages and makes it available to the community for free. But following that link there is a note "Note that this scan and data is for testing purposes only. The data is not up-to-date, and is not licensed for commercial use." So any commercial user of pipenv that runs the safety check could be in violation of the pyup.io terms of service?

[matteius]

@yeisonvargasf Are you able to seek clarification about the above questions/concerns?

[Jwomers]

Hi @mattip, CEO of PyUp here 👋
Thanks for your questions! They make sense and I will get back to this thread later today with all the information you're looking for :)

[Jwomers]

Hi @mattip, thanks again for raising these concerns!

Since its inception, PyUp has published a delayed version of our vulnerability database for free for the Python community - Safety DB. The goal was to serve and give back to the open-source community and improve the security of the entire Python ecosystem. I think we've been successful in this: Safety DB is widely used across the ecosystem, and our Safety CLI scanner runs millions of free vulnerability scans every month.

This free database was always meant to help open source maintainers keep their codebases and systems secure and so was licensed for non-commercial use. We didn't enforce this, however, and many teams have and still do use the free version of Safety CLI and Safety DB for commercial projects, which technically is against the license terms for that database. We've never taken legal action or any action at all against teams or projects that do this, just hoped and trusted that our paid services were enough of a value to add for some commercial teams to subscribe.

We're making a few changes to our databases, licenses and paid features to make all of this clearer and better for the open-source and general Python communities:

• We have published a new free database that is open-source and commercially usable, which now powers Safety CLI and pipenv check. This means running safety check or now the newly updated pipenv check command uses this new vulnerability database that is free and licensed for commercial use.

• We're simplifying our paid and hosted features. The vast majority of our users and customers are not using features like Safety CI, and we're transitioning away from these in favour of more modern integrations like GitHub Actions that give users more control and are also more secure. This does not impact the free usage of Safety CLI itself and associated tools like pipenv check, which remain free.

PyUp is proud to have served the open source Python community for these 5 years and is committed to continuing to do so! Now Safety CLI (and pipenv check) are commercially usable for free. PyUp's vulnerability database is considerably more comprehensive than other options out there (OSV, GitHub, GitLab, etc), and we're proud to make this data available for the open-source community. This partnership and integration has always added a more comprehensive security scan feature to pipenv, and with the new release, it's also now fully commercially usable too.

Additionally, open-source projects (as well as non-profits) can apply for a 100% discount on PyUp's paid plans too. Our goal has never been and never will be to profit off of the open source community. We want to support the open source community as much as we can through free tools and data, as well as making paid features free for open source too.

Let me know if that answers your questions, and also happy to jump on a call to discuss this; I'm at [email protected] :)

cc @matteius

[mattip]

Thanks so much for allowing use of Safety DB and for explaining the licensing around it.

We have published a new free database

Is there more information about this offering? The link you provided to Safety DB still has a disclaimer about commercial use.

It seems the version of safety was updated in #5217. It might be nice to print out the terms of use and some attribution (to mention pyup.io) in the output as shown in this comment.

[Jwomers]

@mattip great question! Our eng team is ahead of me on publishing our blog post about this new change, but it's coming soon, I'll post it back here when we've released it, probably sometime next week. In the meantime, you can see the license attached to the db that pipenv is now using ( URL: https://d2qjmgddvqvu75.cloudfront.net/aws/safety/pipenv/1.0.0/insecure_full.json - note this URL prompts a download of the jSON) has the license attribution in the meta section of the JSON: attribution: "Licensed under CC-BY-4.0 by pyup.io" - CC-BY-4.0 is the same license that others like OSV are publishing some of their data under.

Let me know if that answers your questions? And I'll loop back here when our blog post is published! :)

cc @matteius @yeisonvargasf

[matteius]

@Jwomers and @mattip Happy Holidays! Justin, Was that blog post published? Matti, does this satisfy your request, and if so can we close it out?

[mattip]

Indeed the URL listed above has a "attribution": "Licensed under CC-BY-4.0 by pyup.io" in the metadata, perhaps the safety cli should output a little more information like

Using: 
    database: https://raw.githubusercontent.com/pyupio/safety-db/master/data/ (local file)
    last updated: Dec 22, 2022
    attribution: Licensed under CC-BY-4.0 by pyup.io
    source URL: https://d2qjmgddvqvu75.cloudfront.net/aws/safety/pipenv/1.0.0/insecure_full.json
    for more information: https://github.com/pyupio/safety-db
    Please report this information if problems are found
Found and scanned 118 packages

I didn't see the blog post.

Of course, the project may decide this is too much and close the issue.

I was curious how pipenv actually did the security check since some users reported pipenv as the source of a warning when using pypa/wheel, and it took more digging than I expected to find out what was going on.

[Jwomers]

Hi! Blog post is not yet published we’re still aligning some other small projects so we can publish at the same time. It will be published in January for sure, thanks for your patience but rest assured that the data used here is Licensed under CC-BY-4.0, as shown by the meta data

[matteius]

I believe the licensing has been clarified.