Please update pinned versions

[edrozenberg]

AWS CLI is unique on my system in pinning many things to specific versions, and pinning them to versions that are very old and no longer in common usage by "most people". I tested just now with the latest aws cli versions available.

And in particular from a security standpoint, pinning versions for security packages like cryptography and rsa opens security holes if they cannot be upgraded to always be current.

Would be great to see AWS CLI remove the pins because they create ongoing, irresolvable conflicts.

awscli==2.1.26

 - colorama [required: >=0.2.5,<0.4.4, installed: 0.4.4]
 - docutils [required: >=0.10,<0.16, installed: 0.16]
 - cryptography [required: >=2.8.0,<=2.9.0, installed: 3.4.4]
 - ruamel.yaml [required: >=0.15.0,<0.16.0, installed: 0.16.12]
 - wcwidth [required: <0.2.0, installed: 0.2.5]
 - prompt-toolkit [required: >=2.0.0,<3.0.0, installed: 3.0.16]

awscli==1.19.7

 - docutils [required: >=0.10,<0.16, installed: 0.16]
 - PyYAML [required: >=3.10,<5.4, installed: 5.4.1]
 - colorama [required: >=0.2.5,<0.4.4, installed: 0.4.4]
 - rsa [required: >=3.1.2,<=4.5.0, installed: 4.7]

[kdaily]

Hi @edrozenberg,

Thanks for your comment. We will continue to be conservative with version ranges and won't be removing the ceiling by default. This is to be sure that we maintain backwards compatibility as much as possible, and we have the chance to review any interface changes to the dependencies.

However, we can improve the process for raising the ceiling when new versions become available. It would be a nice feature to get notified on new dependency versions and automatically run interface testing on them so that we can respond to these changes more quickly. For the time being, if you have specific packages that you feel that need to be bumped up, please file an issue for them and describe the conflicts that you're getting with them.

[edrozenberg]

@kdaily thanks, maybe the common usage is to dedicate a machine or VM to be the "aws cli" machine, because the pinned versions of the aws cli reqs can prevent running other packages that require newer versions on the same machine.

[weddige]

I would really appreciate, if you could update colorama and cryptography. If you don't want to remove the ceiling, you maybe could unpin the minor versions.

[kdaily]

@weddige, we're looking into updating cryptography, but the introduction of a Rust dependency has made this more involved. There is an open PR for bumping colorama, but it still needs review as well.

[dconathan]

👍 for me on this issue... right now I can't install the latest schemathesis and awscli in the same environment...

ERROR: Could not find a version that matches colorama<0.4.4,<0.5.0,>=0.2.5,>=0.4.4
...
colorama<0.4.4,>=0.2.5 (from awscli==1.20.36)
colorama<0.5.0,>=0.4.4 (from schemathesis==3.9.7)

[Stranger6667]

@dconathan
On the Schemathesis side, 0.4.4 is not a hard requirement - I can surely relax it for the next release

[alex]

@kdaily I'm one of the maintainers of pyca/cryptography and we're interested in seeing what can be done to get the version cap bumped here. Since our first release with Rust we've made a number of improvements that should help users out: We ship wheels on more platforms (notably musllinux, arm64+universal2 for macOS), lowered our MSRV, and improved the output when compilation fails. Hopefully all of this makes it more tractable to increase the version cap. Thanks!

[nateprewitt]

Thanks for checking in @alex! We had originally paused this waiting for the Rust migration to play out and I think it slipped off the radar. I'll bring this up with the team and see if we can start getting this prioritized.

[alex]

Awesome! If there's more we can be doing, let us know

[nanonyme]

Any chance also updating the docutils dependency? Docutils versions supported by awscli no longer build with setuptools 60.

[kdaily]

@nanonyme,

There's an open PR (#6011) to do this, but it's blocked by some required work for the current documentation pages. I don't have a timeline on when it will get resolved right now.

[nateprewitt]

Providing a quick update on cryptography. #6636 is currently blocked on dropping manylinux1 support. Once that's done we'll be able to update.

[alex]

Is there a seperate place to track the manylinux1 effort?

[mkamioner]

Can pyyaml be upgraded to 6.0?

[kyleknap]

@alex I opened an issue to track dropping manylinux1 support: #6742. We also merged the PR from @nateprewitt: #6636 to loosen the version range of cryptography. This will make sure that we are using 36.0.1 in all of the official v2 artifacts except the Linux x86_64 installer.

In addition to dropping support for manylinux1, I'd like to see us launch official support for installing the AWS CLI v2 from source: #6352. This will allow more granular control over which versions of a dependency are used for your v2 installation and also provide some path forward to using v2 even if the official pre-built installers do not support your environment (whether the installer dropped support for the environment or never supported it in the first place).

[jwhite007]

I concur with this issue. Some dependencies are becoming so outdated that they are beginning to interfere with other packages which require newer versions. See below to see how behind the dependencies are from the current release of awscli. Why is it that when a new version of awscli is released, it's not tested against the latest versions of its dependencies? Please keep up to date on dependencies.

awscli 1.22.92 released on 20220408

outdated deps:

colorama 0.4.3 released on 20191206
current 0.4.4 released on 20211012

docutils 0.15.2 released on 20190730
current 0.18.1 released on 20211123

PyYAML 5.4 released on 20210119
current 6.0 released on 20211013

rsa 4.7.2 released on 20210224
current 4.8 released on 20211124

[yan12125]

From #5943 (comment),

It would be a nice feature to get notified on new dependency versions and automatically run interface testing on them so that we can respond to these changes more quickly.

I noticed dependabot is configured for a few dependencies in aws-cli-v2: https://github.com/aws/aws-cli/blob/develop/.github/dependabot.yml. How about extending it for more dependencies?

[yan12125]

Could awscrt be updated as well? Currently aws-cli v2 requires [1] awscrt>=0.12.4,<=0.14.0. I'm looking into installing awscrt from sources on Python 3.11, and apparently 3.11 is supported since awscrt >= 0.15.0 [2].

See: https://bugs.archlinux.org/task/76618

[1] https://github.com/aws/aws-cli/blob/2.9.0/pyproject.toml#L37
[2] awslabs/aws-crt-python#402

[4x0v7]

The cryptography package is pinned at 38.0.1 but I'm seeing a security advisory recommending to update to 38.0.3 or later.
Using Trivy to scan, here's the result:

{
  "Target": "Python",
  "Class": "lang-pkgs",
  "Type": "python-pkg",
  "Vulnerabilities": [{
    "VulnerabilityID": "GHSA-39hc-v87j-747x",
    "PkgName": "cryptography",
    "PkgPath": "usr/local/aws-cli/v2/2.9.6/dist/cryptography-38.0.1.dist-info/METADATA",
    "InstalledVersion": "38.0.1",
    "FixedVersion": "38.0.3",
    "Layer": {
      "DiffID": "sha256:433fd60ff46245940844818b911c7b163bb4991020d438661d3ad7281eecdc56"
    },
    "SeveritySource": "ghsa",
    "PrimaryURL": "https://github.com/advisories/GHSA-39hc-v87j-747x",
    "DataSource": {
      "ID": "ghsa",
      "Name": "GitHub Security Advisory Pip",
      "URL": "https://github.com/advisories?query=type%3Areviewed+ecosystem%3Apip"
    },
    "Title": "Vulnerable OpenSSL included in cryptography wheels",
    "Description": "pyca/cryptography's wheels include a statically linked copy of OpenSSL. The versions of OpenSSL included in cryptography 37.0.0-38.0.3 are vulnerable to a number of security issues. More details about the vulnerabilities themselves can be found in https://www.openssl.org/news/secadv/20221101.txt.\n\nIf you are building cryptography source (\"sdist\") then you are responsible for upgrading your copy of OpenSSL. Only users installing from wheels built by the cryptography project (i.e., those distributed on PyPI) need to update their cryptography versions.",
    "Severity": "MEDIUM",
    "References": [
      "https://github.com/advisories/GHSA-39hc-v87j-747x",
      "https://github.com/pyca/cryptography/commit/382e759bcded5773330eeed748c86b213ec618c5",
      "https://github.com/pyca/cryptography/commit/cf2ada625d1188d6cd46396f301b98095da577f7",
      "https://github.com/pyca/cryptography/security/advisories/GHSA-39hc-v87j-747x"
    ]
  }]
}

[tim-finnigan]

Thanks @4x0v7 - cryptography was just added to dependabot and ceiling was raised for v2 in this PR: #7518.

[mgzenitech]

Please update PyYAML to v6. It was released year ago.

[proutyio]

Please update PyYAML to v6. It was released year ago.

Looks like they have an open PR. Over a year since it was opened.
#6648

Hopefully it will be merged soon.

[kchoudhu]

Any update on when the update to PyYAML is going to go through? This is actively blocking updates to Python 3.11.

[doxie-king]

When can this get updated?

[sadikkuzu]

Any update?

[Andarius]

The latest version seems to fix this issue: https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst

[nat-418]

Can docutils be updated to 0.19 ?

[yan12125]

A few weeks ago I submitted a pull request to upgrade ruamel-yaml: #8072. Can someone review it?

[rganesan-gsngames]

Any update on this?

[amardeep2006]

Is this possible to bump up cryptography as suggested by dependabot.
#8266 and #8030

[chkdmin]

Now pyyaml support 6.0 in #8037

[yan12125]

Thanks for the dependabot integration. I noticed that tests in some dependabot pull requests fail with Python dependencies not met (ex: #8570). That issue be resolved by invoking https://github.com/aws/aws-cli/blob/v2/.github/workflows/update-lockfiles.yml automatically for dependabot branches.

[BwL1289]

Would also be nice for awscliv2 to update urllib to v2

[yan12125]

botocore bundled in aws-cli-v2 does not support urllib3 2.x yet. At least the following changes should be backported:

boto/botocore#2922
boto/botocore#2924
boto/botocore#2990

Several other changes from botocore are also needed (ex: #8342 (comment)). It takes me too much time to manually backport those changes. Hopefully the bundled botocore can catch up upstream botocore in a more manageable way. For example, with a script and/or a CI bot.

PS. support for urllib3 was once discussed in #8011 as well

[BwL1289]

It takes me too much time to manually backport those changes. Hopefully the bundled botocore can catch up upstream botocore in a more manageable way. For example, with a script and/or a CI bot.

thanks @yan12125. agree that this is the way to go with packaged botocore.