Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guess unpinned versions in python requirements.txt #1966

Merged
merged 9 commits into from
Jul 27, 2023

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Jul 27, 2023

Expands on #1597 by exposing a new python.guess-unpinned-requirements configuration, allowing loose requirements to be filled in with the lowest expressible version instead of dropping the package altogether. Note: since this is synthesizing version information this is an opt-in feature.

This replaces the v10 schema changes in #1967 since they have not been released yet (a condition specifically allowed in the schema readme). The small change is to allow for some elements to be optional in the requirements metadata (everything except for name and version).

@wagoodman wagoodman added the breaking-change Change is not backwards compatible label Jul 27, 2023
@github-actions
Copy link

github-actions bot commented Jul 27, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                              │ ./.tmp/benchmark-e497ead.txt │%0A                                                              │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                       12.61m ±  1%25%0AImagePackageCatalogers/apkdb-cataloger-2                                        751.0µ ±  2%25%0AImagePackageCatalogers/binary-cataloger-2                                       210.2µ ± 11%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                       623.3µ ±  1%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                   23.36µ ±  1%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                             98.89µ ±  1%25%0AImagePackageCatalogers/java-cataloger-2                                         13.99m ±  1%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                         98.50µ ±  3%25%0AImagePackageCatalogers/javascript-package-cataloger-2                           405.8µ ±  1%25%0AImagePackageCatalogers/nix-store-cataloger-2                                    290.8µ ±  2%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                       834.4µ ±  2%25%0AImagePackageCatalogers/portage-cataloger-2                                      504.6µ ±  1%25%0AImagePackageCatalogers/python-package-cataloger-2                               3.558m ±  1%25%0AImagePackageCatalogers/r-package-cataloger-2                                    219.8µ ±  1%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                       580.1µ ±  3%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                 968.4µ ±  1%25%0AImagePackageCatalogers/sbom-cataloger-2                                         122.8µ ±  1%25%0Ageomean                                                                         509.7µ%0A%0A                                                              │ ./.tmp/benchmark-e497ead.txt │%0A                                                              │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                       5.123Mi ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                        204.9Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                       30.18Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                       169.0Ki ± 0%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                   3.695Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                             9.906Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                         2.824Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                         8.594Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                           94.22Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                    49.14Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                       186.7Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                      119.9Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                               1.003Mi ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                    53.30Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                       180.9Ki ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                 144.1Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                         14.20Ki ± 0%25%0Ageomean                                                                         100.3Ki%0A%0A                                                              │ ./.tmp/benchmark-e497ead.txt │%0A                                                              │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                        87.75k ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                         4.182k ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                         830.0 ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                        3.002k ± 0%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                     132.0 ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                               281.0 ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                          39.88k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                           228.0 ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                            1.342k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                      895.0 ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                        4.080k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                       2.268k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                                16.44k ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                      929.0 ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                        3.989k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                  2.447k ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                           394.0 ± 0%25%0Ageomean                                                                          2.051k

@wagoodman wagoodman removed the breaking-change Change is not backwards compatible label Jul 27, 2023
@wagoodman wagoodman force-pushed the feat-python-requirementstxt-parsing branch from d1231b4 to ad3926b Compare July 27, 2023 14:03
@wagoodman
Copy link
Contributor Author

Note: depends on #1967 before reviewing / merging

@wagoodman
Copy link
Contributor Author

# $ diff -C 2 ./schema/json/schema-9.0.2.json ./schema/json/schema-10.0.0.json
*** ./schema/json/schema-9.0.2.json0.2.jThu Jul 27 08:55:01 2023-10.0.0.json
--- ./schema/json/schema-10.0.0.json    Thu Jul 27 11:30:14 2023
***************
*** 1,5 ****
  {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
!   "$id": "anchore.io/schema/syft/json/9.0.2/document",
    "$ref": "#/$defs/Document",
    "$defs": {
--- 1,5 ----
  {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
!   "$id": "anchore.io/schema/syft/json/10.0.0/document",
    "$ref": "#/$defs/Document",
    "$defs": {
***************
*** 1621,1630 ****
          },
          "markers": {
!           "patternProperties": {
!             ".*": {
!               "type": "string"
!             }
!           },
!           "type": "object"
          }
        },
--- 1621,1625 ----
          },
          "markers": {
!           "type": "string"
          }
        },
***************
*** 1632,1639 ****
        "required": [
          "name",
!         "extras",
!         "versionConstraint",
!         "url",
!         "markers"
        ]
      },
--- 1627,1631 ----
        "required": [
          "name",
!         "versionConstraint"
        ]
      },

@wagoodman wagoodman added breaking-change Change is not backwards compatible and removed breaking-change Change is not backwards compatible labels Jul 27, 2023
Signed-off-by: Alex Goodman <[email protected]>
@wagoodman wagoodman marked this pull request as ready for review July 27, 2023 15:38
@wagoodman wagoodman added the changelog-ignore Don't include this issue in the release changelog label Jul 27, 2023
@wagoodman wagoodman requested a review from a team July 27, 2023 15:40
@wagoodman wagoodman self-assigned this Jul 27, 2023
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
* feat: python requirements.txt parsing inclusive

Signed-off-by: manifestori <[email protected]>

* refactor: parseVersion

Signed-off-by: manifestori <[email protected]>

* add python config for optional requirements version constraint resolution

Signed-off-by: Alex Goodman <[email protected]>

* fix tests

Signed-off-by: Alex Goodman <[email protected]>

* allow for python requirements metadata to be optional

Signed-off-by: Alex Goodman <[email protected]>

* restore cyclonedx dependency

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: manifestori <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Co-authored-by: manifestori <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants