publish v2 to PyPI #4947
Comments
|
Not publishing CLI v2 to PyPI has been a conscious decision and for the time being we have no plans to publish to PyPI. In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature. |
joguSD
added
the
closing-soon
|
@joguSD Could you please elaborate on the reasoning behind this conscious decision? Perhaps link us to some dialog around this discussion? |
|
I think that time may prove if that decision was more towards conscious or in-conscious one. I am really curious to see how this will evolve, I only hope I have enough popcorn 🍿. On the other hand, other wheels coming from the same source did not get their notoriety without any reasons. boto, boto3 anyone? At least pypi was 3rd party that guaranteed that a specific version is not altered once uploaded. I guess now we will have to keep saving checksums . |
|
So |
no-response
bot
removed
the
closing-soon
|
Sure, we all know that is impossible to force push code to github, not. |
|
Or use a tarballs link for greater speed https://adamj.eu/tech/2019/03/11/pip-install-from-a-git-repository/ |
|
Just as a reminder, the AWS CLI V1 will continue to receive regular updates Here are some reasons why we dropped publishing to PyPI for AWS CLI V2, and are
Hopefully that helps to shed some light on why we decided not to publish the Lastly, we're looking to continue to improve the installation process for AWS |
|
Those sound as decent and understandable reasons — except perhaps the #1. I'd avoid going into language comparison contests here — but it's hard to refute that Python is among the few must know languages for DevOps folks and the like. Likely, I'm being ignorant in this judgement; but one can be forgiven for not "really knowing" even Bash — but not Python. It's the language of humanity's first ever photo of a [shadow of] black hole, remember?
Edit: sorry, disregard this. Lastly: What's the route to go to install aws-cli v2 from source? |
|
Not only For example, on a vanilla A1(ARM-base) Amazon Linux 2 instance, you can install AWS-CLI v2 as follows: |
|
What about providing an AppImage for Linux vs the zip archive with the installer? I created one using the contents from the dist dir and it seems to be working fine. Just throwing it out there as it may be worthwhile way to distribute the v2 release for Linux. I can share steps I used if there's any interest. |
|
That like is like gold Translates to "We made a release using unreleased dependencies." and "We didn't test if the release artifact can be installed". Nobody mentioned is that binary code downloaded needs to signed on MacOS, so which means that the so called Let's face it, v2 was released even without a tag being made of github, this should say a lot. AFAIK it could also be made from another repository, another license. The v2 was not presented as a preview and was directly published on https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html as a recommended release when the code repository, didn't had any tag, prerelease or not. I would think twice before arguing that so many users are not able to use pip, or compile code, when basic software development practices like tagging are not followed. |
I agree. Doing things in a domain requires domain knowledge.
This is more domain knowledge, sure.
It seems to me that this is your crux. Supporting PIP costs time and money and it sounds like it is generally painful. I suspect this is because, of the people that need support, they don't have the domain knowledge you'd mentioned. There is a difference between "supporting PIP as the method of install" vs "uploading your package to PyPi". You can keep on delivering your
You may not have control over it, but you can check for it and make it dependent on a specific version.
Additional binaries... sounds like dependencies, which could be checked for during the build process. The take-away I'm trying to make here is that supplying your packages via PIP isn't antithetical to your desire to limit support to your official installer. Would you please reconsider your aversion to pypi? Many developers would appreciate it. |
Don't you think that this is quite intentional?.. To a user of your package, that's an atrocious idea: that installing an API client tool gives control over openssl on my system to the tool author. I'm getting shivers just from thinking of it! Constructively, there're two things you can do:
Why, what's wrong with Thousands of developers do include arbitrary files in their pip installs. I see no problem here. Oh wait, actually I do. Binaries don't belong in source installs. If I'd want to rewrite aws-cli in golang or something — I'd make it separate project, in a separate repo, and maintain its interface as "CLI compatible" rather than vaguely "backwards-compatible". Don't ship binaries where source is expected. No-no. No no no no. |
|
@joguSD Would it make sense to have separate issue for "install with a package manager" or "install with package manager X". Currently It's hard to automate the installation with the bundled installer (because you may not have a directory in your PATH that's suited for not-packaged binaries). And it could be a good place to consolidate all the related issues like this one, #4926 #4842 #4660 #3553 #3627 #2501 and #727. With v1 the suggestion was (probably rightly so) to use pip when needing a packaged/automated installation, with v2 this isn't an option anymore |
|
@joguSD what about platforms you don't directly support but language packaging mechanisms (like PyPi) do? For instance, Cygwin? |
|
Hi! I do understand that AWS is trying to make it easier to install the AWS CLI for users who are not familiar with Python. It is regrettable that it is done at the expense of users who are familiar with Python, in a way that prevents them from integrating the AWS CLI within many environments. For instance, many platforms will let you specify Python dependencies in a This is the case on Heroku, and it seems to be the case on AWS Lambda too. If I want to run a script using the AWS CLI on Lambda, it looks like I could do it with AWS CLI v1 (since the Lambda documentation explains how to package Python dependencies) but I don't know if it'd be possible with AWS CLI v2. There are also many platforms that won't allow root access. A package published on PyPI can be installed as an unprivileged user, while the AWS CLI v2 installer requires sudo. There are ways to have your cake and eat it too: check out Docker Compose installation instructions. Compose is available as a standalone executable for most platforms, and can also be installed with I understand the desire to make Python an "implementation detail". But every software is written in some language, and that language has best practices around installation procedures. For instance, C code can leverage e.g. autoconf and automake, and every Linux distribution will have helpers to facilitate packaging software that uses autoconf and automake. Likewise, software written in Python can leverage PyPI. Finally, if CLI installation is truly a huge point of friction, there are more drastic solutions as well. For instance, it could be ported to Go, to allow producing a standalone binary for every major platform. In 2012, I was working at Docker Inc. when we made the decision to rewrite our container engine from Python to Go, in order to facilitate its adoption by the largest number of people. I do understand that the AWS CLI is a complex piece of software that needs to interface with hundreds of services. But if a 15-person start-up (the headcount of Docker back then) can rewrite a container engine in Go, perhaps a company like AWS can manage to port a CLI as well? ;-) If porting the CLI to Go seems to big a burden, I'd suggest to at least offer a PyPI installation process. It's fine if it's "not officially supported". At least don't make our job harder. Thanks, |
|
The installation instructions look like a alpha release note. However, the documentation suggests that v2 is a prod release. The only installation method that seems to be prod ready is Windows MSI (I have not tested). Please provide options for installing using package managers such as pip etc ASAP. |
|
Unfortunately a new core AWS feature (SSO) is a non-starter for us for the moment, until
|
|
Another problem with the current install process is that while there are instructions on how to update the package, there doesn't appear to be any documented way to programmatically check if an update is available. |
|
I'm having to use the new |
|
For my own future reference 😄 |
|
Another disadvantage with not publishing to pip, you making me have a second python interpreter installed, which might be fine for a dev machine, but in a docker image isnt good. This is related to the issues installing on alpine linux in #4685 - if it just used the system python (as in v1) then would be no problem. |
The reason for installing version one aws/aws-cli#4947
Not sure I get your point here. Installing with pip does not mean pip has to be the only way to install aws cli.
Again, I do not get the point you are making here. Nobody is forcing AWS CLI to be distributed as a source python package. You can ship pre-built versions of AWS CLI as wheels to pypi, thus have no build-time requirement during installation.
What are you implying here?
Of course you can, multiple libraries on pypi do ship with prebuilt dependencies embedded. To be honest, this move of getting out of pypi is really getting on my nerve. |
This is especially bad, since there is not a great way to automate upgrading the AWS CLI now. |
Of course you are right, my comment was meant tongue in cheek. |
|
If you want to sell your services you better bend over backwards to make those tools readily available in a professional manner. Meaning packages in all Linux distros ... skip pypi if you want but things like https://bugs.gentoo.org/767268#c1 are shocking results of a really poor deployment channel for that essential component. |
|
Removing the path of least resistance (installing via pip) just makes me not want to upgrade. I'm sure a lot of other people will ignore v2 until v1 is EOL and we're forced to install a python cli tool in a non native way. |
I'm sorry if this is bitter feedback for some, but having learned that AWS staff has difficulties with basic packaging and versioning so much that they want to make Python an implementation detail of aws-cli — I'm reluctant to use even v1. Who knows what other nonsense they decided to put into there; I don't care enough to review the 70kSLoC myself. |
$ wget https://awscli.amazonaws.com/AWSCLIV2-2.1.26.pkg
$ open AWSCLIV2-2.1.26.pkg
$ aws --version
aws-cli/2.1.26 Python/3.7.4 Darwin/20.3.0 exe/x86_64 prompt/offMeanwhile, a pip-installed CLIv1 runs fine on Python 3.9. |
|
I'm clearly late to this conversation... but just to highlight a major issue again: package managers are there for a reason. Throwing away package management altogether is such an incredible antipattern. There is so much that package managers do in the ops space. Even in the environment of the dev workstation, where there's a human to control everything, what used to be
is now
That's the recommended "happy path" with no edge cases. And then repeat that all again when there's an update... if you remember to check for one. This is not the right method for less-skilled admins, plus rather than "doing the work once" by the developers, you're making every admin duplicate the work. If not pip because you want to go language agnostic, then why not rpm for Amazon Linux and maybe deb because it's the other super-popular package format? Both of these are designed for dependency management and deal with upgrades fluidly, and you don't have to hunt for a link in a webpage for packages in them. I don't have the venv or docker problems that other users have but I'll keep using v1 anyway, because updating to get new features is a one-liner, rather than a manual zipfile install method from last century (no joke there - rpm's from '97, apt's from '98, cpan '95...) |
|
The reason this is an anti-pattern for development is that a different version of botocore may be installed for other reasons, and so we Python developers who use virtual environments simply cannot use awscli v2 without using the workarounds above, because it would conflict with the packages used by other tools. This works against proper development practices. Some community work-arounds, such as pypi package awscliv2, work for me only at home because within my federal agency, we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation. |
|
@danizen i would recommend keeping working with venvs, because they're so widely used, and so more manageable than complete system environments, just a note. |
Also, I dont want to install and run docker just for the awscli. It's not that I cant install it, it's just that I dont use docker, and having to go that route just for a cli tool ... yeah, that's a NOPE |
|
Here you have a very well done paper on why the route you've choose with awscli v2 is wrong -- https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/ |
|
Unbelievable. And the worst part is - it's Amazons official. I wouldn't be surprised if it was Alibaba cloud or something, but it's Amazon. Really ? Wow, just wow. |
|
This issue isn't going to progress anything, it's not getting closed nor resolved, but kept open to blackhole discussion, nothing will come from this, and nobody can force amazon to do anything about it. So i suggest finding other ways to forward this issue, other than leaving behind a comment here saying your support. I've been watching this issue thread for a while now, and I haven't seen a member respond to the latest concerns and calls for python best-practice here, other than suggesting an entirely un-ergonomic way of deployment, not even willing to meet the developers half-way, dismissing releasing on pypi in favour of a vendored appimage blob. Earlier here it's already been said that amazon considers using python in general as an implementation detail to how the aws cli is run, fine, fair, but then it acts as if it's above any choice it makes in that regard, e.g. not answering to what the community of python expects of their standardised way of publishing a package for a 30-year-old programming language, on a 20-year-old publication site, for a cloud service that has then only existed for 15 years, and then act as if their way is the highway. Well then, @aws, fucking commit, make your own https://pypi.aws instance, dont just sit around and dabble with this absolute bullshit. You love developers so much? You love your customers so much? Prove it then, your platform has been a hot piece of overpriced garbage for a decade and a half now, so you cant change my mind, but you might change others. |
|
Just boycott awscli v2. Don't use it. If anyone ever wonders why — explain. Hopefully you can be understood. |
|
I do see the problem with installing in a virtual environment from AWS perspective - there can be as many installations of AWS CLI as there are virtual environments, and together with user knowledge and the issue of including other capabilities maybe not written in Python and controlling the version of SSL (LibreSSL probably on their horizon), it makes sense. However, I have programs that use boto3 to do things similar to aws sso login, and they rely on boto3 and therefore botocore. Bottom-lines:
If these three requirements can be met by AWS, I see no problem with the lack of PyPI. Making an image available addresses the first two requirements, but the 3rd requirement is not met because many Windows users still are not allowed to use WSL. |
This is this biggest thing for me. I'd actually prefer having it in an apt repo to PyPi, but the current only official deployment mechanism is... not good. I also don't love that an entire python runtime is bundled with it. Or that if you upgrade the aws-cli, it leaves the old version lying around. |
|
Well, we are at 2021 and rely on a zip file and bash script with couple cps and lns to install python libraries, how cool is that ? |
|
It's so easy to upgrade awscli v1.
Done. No root privileges required. It's what all the paying AWS customers want. Instead we get this 20 step process which requires a dedicated installation page to explain (instead of a couple lines of the README file) which discourages upgrading. This probably creates more of a risk than anything else honestly. I too am probably going to want to write a Salt state to manage this, but how many AWS customers are going to bother with that? Shouldn't AWS be taking on the inconveniences themselves instead of trying to push them onto paying customers?
Then continue to provide the already existing installation methods too.
If you're already supplying another installation method, what does it matter? It's the user's decision.
Just say you won't support anyone who isn't using a dedicated virtualenv environment for awscli. That would prevent you from dealing with all of those issues.
Yes but it's not that hard to check and fail installation if your requirements are not met, with the appropriate error message. I think most users wouldn't have a problem, unless you decide to rely on some bleeding edge library feature (which wouldn't be a great idea anyway).
Then have awscli initiate a check for whatever dependencies there might be (if ever that becomes a thing) at runtime where required. It's not a big deal. But even if it was, this isn't your typical free software project where people can only hope, beg and perhaps offer a donation to help persuade a change in how things are done. In this case, you are getting paid to write this library because we are all paying for the AWS service. Hence I would expect AWS developers to take on some of the inconvenience to satisfy customers. Well that's my thoughts on the matter. I'm a bit frustrated because I went to use |
|
@joguSD any response yet? Or are y'all explicitly ignoring this issue, even when it's attracting a lot of attention (even today) because you're ignoring one of the oldest language's best practices? Do you all really think you're so above that to not bother with these "silly practices"? I re-read your first message, if you're not going to use python for awscli, commit, don't fuck around and say "oh but we might not do that", it's been a year, people are still annoyed at this not being on pypi, put it on pypi. |
|
Ironically I can see a few of the Amazon leadership principles having been neglected and even broken with how they treat this issue. For one I don't see "Customer Obsession" being respected with this approach... |
|
Hi everyone. First off, thank you all for the frank feedback about this decision, particularly specific feedback from some on this issue about why this matters to you. While @joguSD has already detailed why the team decided to not support However, I would like to ask that we try to keep this conversation civil and constructive. Just as a friendly reminder, contributions to this repository, including issue comments, are governed by the Amazon Open Source Code of Conduct. For our part, I can promise that we're reconsidering this decision internally based on the feedback we've received here. We want to consider the implications of a such a change (see @joguSD's comment linked above), so that will take some time. Expect an update from the team on that front by the end of next week. |
|
The Even if I clone this project and check out |
I am guessing that is by design and intentional? They aren't tagging v2.x.x releases in botocore. To find the "2.0.0dev97 release" (ahem) of botocore you must read the commit log messages from the v2 branch. Then check out out the commit. Then build a wheel like |
|
Hi all - I’m the manager of the AWS CLI team, and I wanted to address the conversation that’s been going on here, as well as some broader issues. I appreciate everyone who’s taken the time to comment. Every comment on every issue in this repository is read by multiple members of the team, and we value every anecdote and data point about how our product works or doesn’t work for you. One of our goals for the rest of 2021 is to do a better job of making that communication bi-directional and to more directly engage with the community on longer-term plans and roadmaps. I’ll discuss some of our plans for that, but first I want to address the issue at hand. We do not plan to publish the AWS CLI project to PyPI at this time. I appreciate that this will not be a popular decision among people following this issue. In Q2, we plan to begin work on a source bundle that is well documented, easily ingestible by package managers, and that can be used to build the CLI for platforms other than those we support out of the box. This source distribution will include botocore along with the CLI. Our hope is that providing an officially-supported source distribution will alleviate many of the frustrations that you have experienced with our current approach. We understand that we need to make our product easily available for those who can’t or don’t want to use one of our pre-packaged binaries or container images. Once we’ve released and validated the source bundle, we’ll open an issue here to prioritize distribution channels that will provide the most value and ensure that maintainers have what they need from us. Starting in April, we’re going to begin posting monthly roadmap updates as issues on our GitHub repository. These will lay out what we’re actively working on and what’s on the horizon. Our backlog includes a mix of priorities, some driven by the CLI team and the community, some which align with larger AWS goals. Our focus will be on sharing the priorities emerging from our team, which will include Github issues we are planning to work on. We will iterate on the exact format and approach for these updates as we determine what works best, but they will be at least monthly for the rest of 2021. I’m going to lock this issue now, and we will be closing any new iterations of it as duplicates. I would ask that everyone please keep in mind the Code of Conduct https://aws.github.io/code-of-conduct when using GitHub to communicate with the team. |
and others
It's great that there's a first-class supported installation mechanism that does not rely on environment specifics, but this is still a normal Python package, and for those customers who do already have an existing compatible environment the installation mechanism is a lot heavier-weight than using the existing Python distribution mechanisms.
Could we get v2 publishing to PyPI so that those customers who have those prerequisites can have a simpler and easier to maintain installation/updating/etc story?
The text was updated successfully, but these errors were encountered: