Create utility for package version handling: parsing, normalization, comparison #119
Comments
|
We currently use semantic_version in /scrapers/npm.py it seems fine but I'm not sure whether it supports deb,rpm style versions which include '-'. |
|
A repaste from chat. Regarding https://github.com/repology/libversion is a C program, IMHO they also don't support NPM version specs out of the box I think you should take a look at https://github.com/python-semver/python-semver they have documentation of supported methods here: https://github.com/python-semver/python-semver/blob/master/semver.py Personally I think we should continue to use https://github.com/rbarrois/python-semanticversion |
|
https://github.com/dephell/dephell_specifier also seems promising |
pombredanne
changed the title
|
After some experience with dephell, we want to switch to something else mainly because :
I suggest to we create a wrapper library, which dispatches the comparision according to the ecosystem. |
|
Here are different specs and the libs which will take care of them semver :- Used by rust, npm, ruby . We will delegate this to https://github.com/python-semver/python-semver . |
|
Some comments:
|
|
Here is a proposed approach:
Universal version specifiers syntax: For instance: With these operators and syntactic elements:
The comparison of two versions is scheme-specific. A version scheme covers both The known schemes and their codes are:
class BaseVersion:
# each version value should be comparable e.g. implement functools.total_ordering
scheme = None
value = None
def validate(self):
"""
Validate that the version is valid for its scheme
"""
raise NotImplementedError
class GenericVersion:
scheme = "generic"
def validate(self):
"""
Validate that the version is valid for its scheme
"""
# generic implementation ...
# use 1. of https://github.com/repology/libversion/blob/master/doc/ALGORITHM.md#core-algorithm
# Version is split into separate all-alphabetic or all-numeric components. All other characters are treated as separators. Empty components are not generated.
# 10.2alpha3..patch.4. → 10, 2, alpha, 3, patch, 4
class DebianVersion:
scheme = "debian"
def validate(self):
"""
Validate that the version is valid for its scheme
"""
# debian implementation ...
class SemverVersion:
scheme = "semver"
def validate(self):
"""
Validate that the version is valid for its scheme
"""
# node-semver implementation ...
versions_classes_by_scheme = {
"generic": GenericVersion,
"semver": SemverVersion,
"debian": DebianVersion,
# ....
}
def parse_version(version):
"""
Return a Version object from a scheme-prefixed string
"""
if ':' in version:
scheme, _, version = version.partition(':')
else:
scheme = 'generic'
cls = versions_classes_by_scheme[scheme]
return cls(version)
class VersionRange:
# one of <> >= =< or != or =
operator = ""
range = ""
class VersionSpecifier:
scheme = ""
ranges = []
@classmethod
def from_version_spec_string(cls, value):
"""
Return a VersionSpecifier built from a version spec string, prefixed by
a scheme such as "semver:1.2.3,>=2.0.0"
"""
raise NotImplementedError
@classmethod
def from_scheme_version_spec_string(cls, scheme, value):
"""
Return a VersionSpecifier built from a scheme-specific version spec string and a scheme string.
"""
raise NotImplementedError
def __contains__(self, version):
"""
Return True if this VersionSpecifier contains the ``version``
Version object or scheme-prefixed version string. A version is contained
in a VersionSpecifier if it satisfies all its Range.
"""
if isinstance(version, str):
version = parse_version(version)
# .... magic happens here
raise NotImplementedError
def __str__(self):
"""
Return this VersionSpecifier string using a canonical representation and our universal syntax.
"""
# TODO: sort to make canonic
ranges = ",".join(self.ranges)
return f"{self.scheme}:{ranges}" |
|
@pombredanne I'm doing the implementation at https://github.com/sbs2001/universal-versions , we can move it to nexB GH org once significant progress is made.
|
|
@pombredanne regarding memory/python-dpkg@1a7c022 . I don't want the whole module since it does many other things. I want to extract the method https://github.com/memory/python-dpkg/blob/1a7c022bb0d4b631241c98e526f88f178c243dcc/pydpkg/__init__.py#L529 and all it's dependencies. How do I document it's origins and license ? |
Please use https://github.com/nexB/debian-inspector/ then , which is already a fork of the above https://github.com/nexB/debian-inspector/blob/master/src/debut/version.py I would rather avoid having multiple forks of the same code in our projects and we can evolve it as needed, so please see if you can use this. Wrt. the general question of the documentation of origin and license, see https://github.com/nexB/debian-inspector/blob/master/src/debut/version.py.ABOUT and https://github.com/nexB/debian-inspector/blob/master/src/debut/version.py.LICENSE |
|
And if you want it in a dedicated library such as "debian-version" that's OK and we could extract it if needed as a shared library |
|
@pombredanne I could use https://github.com/nexB/debian-inspector/, but there are 2 issues with it.
|
yes I would like that |
|
@pombredanne you're right about rpms, Using https://github.com/sassoftware/python-rpm-vercmp would be least painful for me. |
|
Implementing nuget and maven style intervals won't be necessary in the context of vulnerablecode, no data source uses it. |
|
In context of vulnerablecode python, semver, deb versions are only needed |
|
@pombredanne We don't need to implement maven and nuget style version ranges here. The datasources which provide data about vulnerable maven and nuget packages ie I need only https://github.com/sassoftware/pymaven/blob/1e69f50dc9ce3ac1ae3980e070f09e2505aa89f7/pymaven/versioning.py#L343 . |
|
@pombredanne the issue with https://github.com/nexB/pymaven is the same with |
|
PHP composer most likely uses semver, see https://getcomposer.org/doc/articles/versions.md#testing-version-constraints , it mentions a semver validation utility . See https://github.com/composer/semver , and composer is deendent on it see https://packagist.org/packages/composer/semver/dependents?order_by=downloads |
|
To fork: VulnerableCode doesn't require to compare rpm versions, but since this is going to be used outside of vulnerablecode |
|
@pombredanne the test suite at https://github.com/sassoftware/python-rpm-vercmp is failing sassoftware/python-rpm-vercmp#2 |
|
Fixed via #421 and we have new utility https://github.com/sbs2001/univers. We don't do normalization atm, because all version comparision we do is between packages with same name, type etc(not same version though :) ) |
There are many different versions styles and ways to express version ranges we need to deal with:
We need to craft a utility module to deal with these consistently. For starters in #101 npm avdisory use their own syntax to define these.
We may also need to deal with stored version ranges later.
This is based on this chat:
@NavonilDas wrote:
@pombredanne wrote:
@NavonilDas wrote:
@pombredanne wrote:
And here are some pointers:
The text was updated successfully, but these errors were encountered: