UPM is the Universal Package Manager. It allows you to manage packages for any (supported) programming language through the same interface following the principle of least astonishment. At Repl.it, we use UPM to provide deep package manager integration for many different programming languages using the same infrastructure.
UPM does not implement package management itself. Instead, it runs a package manager for you. The value added is:
- you don't have to figure out whether to use Pip or Pipenv or Poetry to manage your Python packages or wade into the Cabal-versus-Stack holy war in Haskell-land
- you don't have to puzzle out why
pip search flaskdoesn't return Flask in the search results - you don't have to debug Bundler silently dropping your command-line options if you don't specify them in the right (undocumented) order
- you don't have to think about why the developers of NPM and Yarn
decided to implement two completely different and mutually
incompatible behaviors for
list --depth=0, neither of which is exactly what you want - you don't have to investigate what format the Yarn lockfile is in (turns out: almost YAML, but not quite)
- et cetera (I could go on all day)
In other words, UPM eliminates the need to remember a huge collection of language-specific package manager quirks and weirdness, and adds a few nifty extra features like dependency guessing and machine-parseable specfile and lockfile listing.
- Core:
upm add,upm remove,upm lock,upm install,upm list - Index:
upm search,upm info - Guess:
upm guess
| core | index | guess | |
|---|---|---|---|
| python-python3-poetry | yes | yes | yes |
| python-python2-poetry | yes | yes | yes |
| nodejs-yarn | yes | yes | yes |
| nodejs-npm | yes | yes | yes |
| ruby-bundler | yes | yes | |
| elisp-cask | yes | yes | yes |
| dart-pub.dev | yes | yes |
You have many options. UPM is a single binary with no dependencies, so you can install it anywhere. Tarballs are available on the releases page. Read on for instructions on installing via a package manager.
Available on Homebrew in a custom tap.
$ brew install replit/tap/upm
.deb packages are available on the releases page.
.rpm packages are available on the releases page.
Soon to be available on the Arch User
Repository. Right now, you can clone this
repository and install with makepkg using the PKGBUILD in
packaging/aur.
Available on Scoop in a custom bucket.
$ scoop bucket add replit https://github.com/replit/scoop-bucket.git
$ scoop install upm
Soon to be available on the Snap Store. Right now, .snap packages are available on the releases page.
You can try out UPM right away in a Docker image based on Ubuntu that has all the supported languages and package managers already installed.
$ docker run -it --rm replco/upm
Additional tags are also available. replco/upm:full is the same as
the above, while replco/upm:light just has the UPM binary installed
to /usr/local/bin and none of the languages or package managers
installed. If you want to run a specific tagged release, rather than
the latest development snapshot, use e.g. replco/upm:1.0,
replco/upm:1.0-full, or replco/upm:1.0-light.
Let's create a new Python project:
$ mkdir ~/python
$ cd ~/python
We'll start by adding Flask as a dependency. UPM will handle setting up the project for us:
$ upm -l python add flask
--> python3 -m poetry init --no-interaction
This command will guide you through creating your pyproject.toml config.
--> python3 -m poetry add flask
Creating virtualenv python-py3.7 in /root/.cache/pypoetry/virtualenvs
Using version ^1.1 for flask
Updating dependencies
Resolving dependencies... (0.6s)
Writing lock file
Package operations: 6 installs, 0 updates, 0 removals
- Installing markupsafe (1.1.1)
- Installing click (7.0)
- Installing itsdangerous (1.1.0)
- Installing jinja2 (2.10.1)
- Installing werkzeug (0.15.4)
- Installing flask (1.1.1)
UPM operates on a specfile and lockfile for each project. The
specfile says what your project's dependencies are in a human-readable
format, while the lockfile specifies exact versions for everything,
including transitive dependencies. For Python, the specfile is
pyproject.toml and the lockfile is poetry.lock:
$ ls
poetry.lock pyproject.toml
We don't have to read them ourselves, because UPM can handle that. Notice that UPM is now aware that our project uses Python, because of the files that were created:
$ upm list
name spec
----- ----
flask ^1.1
$ upm list -a
name version
------------ -------
click 7.0
flask 1.1.1
itsdangerous 1.1.0
jinja2 2.10.1
markupsafe 1.1.1
werkzeug 0.15.4
Let's search for another dependency to add:
$ upm search nose
--> python3 -c '<secret sauce>' nose
Name Description Version
----------------- ---------------------------------------------------------------------- -------
nose nose extends unittest to make testing easier 1.3.7
nose-detecthttp A nose plugin to detect tests making http calls. 1.1.0
nose-picker nose plugin that picks a subset of your unit tests 0.5.5
nose-progressive A testrunner with a progress bar and smarter tracebacks 1.5.2
nose-unittest UNKNOWN 0.1.1
nose-blockage Raise errors when communicating outside of tests 0.1.2
nose-watcher A nose plugin to watch for changes within the local directory. 0.1.3
nose-bisect A Nose plugin which allows bisection of test failures. 0.1.0
nose-printlog Print log to console in nose tests 0.2.0
nose-json A JSON report plugin for Nose. 0.2.4
nose-faulthandler Nose plugin. Activates faulthandler module for test runs. 0.1
nose-knows 0.2
nose-pagerduty PagerDuty alert plugin for nose 0.2.0
nose-logpertest Logging nose plugin to create log per test 0.0.1
nose-bleed A progressive coverage plugin for Nose. 0.5.1
nose-numpyseterr Nose plugin to set how floating-point errors are handled by numpy 0.1
nose-skipreq nose plugin that will skip Google API RequestError exceptions. 2.0
nose-selecttests Specify whitelist of keywords for tests to be run by nose 0.5
nose-pacman A testrunner with a pacman progress bar 0.1.0
nose-switch Add special switches in code, based on options set when running tests. 0.1.5
We can get more information about a package like this:
$ upm info nose
--> python3 -c '<secret sauce>' nose
Name: nose
Description: nose extends unittest to make testing easier
Version: 1.3.7
Homepage: http://readthedocs.org/docs/nose/
Author: Jason Pellerin <[email protected]>
License: GNU LGPL
For piping into other programs, the search and info commands can
also output JSON:
$ upm info nose --format=json | jq
--> python3 -c '<secret sauce>' nose
{
"name": "nose",
"description": "nose extends unittest to make testing easier",
"version": "1.3.7",
"homepageURL": "http://readthedocs.org/docs/nose/",
"author": "Jason Pellerin <[email protected]>",
"license": "GNU LGPL"
}
UPM can also look at your project's source code and guess what packages need to be installed. We use this on Repl.it to help developers get started faster. To see it in action, we'll need some source code:
$ git clone https://github.com/replit/play.git ~/play
$ cd ~/play
$ upm add --guess
--> python3 -c '<secret sauce>' '<secret sauce>'
--> python3 -m poetry init --no-interaction
This command will guide you through creating your pyproject.toml config.
--> python3 -m poetry add pygame pymunk setuptools
Creating virtualenv play-py3.7 in /root/.cache/pypoetry/virtualenvs
Using version ^1.9 for pygame
Using version ^5.5 for pymunk
Using version ^41.0 for setuptools
Updating dependencies
Resolving dependencies... (1.4s)
Writing lock file
Package operations: 4 installs, 0 updates, 0 removals
- Installing pycparser (2.19)
- Installing cffi (1.12.3)
- Installing pygame (1.9.6)
- Installing pymunk (5.5.0)
You can also just get the list of guessed dependencies, if you want.
The -a flag lists all guessed dependencies, even the ones already
added to the specfile:
$ upm guess -a
pygame
pymunk
setuptools
All of this might seem a bit too simple to justify a new tool, but the real power of UPM is that it works exactly the same for every programming language:
$ upm -l nodejs info express
Name: express
Description: Fast, unopinionated, minimalist web framework
Version: 4.17.1
Homepage: http://expressjs.com/
Source code: git+https://github.com/expressjs/express.git
Bug tracker: https://github.com/expressjs/express/issues
Author: TJ Holowaychuk <[email protected]>
License: MIT
$ upm -l ruby info jekyll
--> ruby -e '<secret sauce>' jekyll
Name: jekyll
Description: Jekyll is a simple, blog aware, static site generator.
Version: 3.8.6
Homepage: https://github.com/jekyll/jekyll
Documentation: http://jekyllrb.com
Source code: https://github.com/jekyll/jekyll
Bug tracker: https://github.com/jekyll/jekyll/issues
Author: Tom Preston-Werner
License: MIT
Dependencies: addressable, colorator, em-websocket, i18n, jekyll-sass-converter, jekyll-watch, kramdown, liquid, mercenary, pathutil, rouge, safe_yaml
$ upm -l elisp info elnode
--> emacs -Q --batch --eval '<secret sauce>' /tmp/elpa552971126 info elnode
Name: elnode
Description: The Emacs webserver.
Version: 20190702.1509
Dependencies: web, dash, noflet, s, creole, fakir, db, kv
That includes adding and removing packages, listing the specfile and lockfile, searching package indices, and guessing project dependencies. UPM knows all the best practices for each language so that you don't have to!
Explore the command-line interface at your leisure:
$ upm --help
Usage:
upm [command]
Available Commands:
which-language Query language autodetection
list-languages List supported languages
search Search for packages online
info Show package information from online registry
add Add packages to the specfile
remove Remove packages from the specfile
lock Generate the lockfile from the specfile
install Install packages from the lockfile
list List packages from the specfile (or lockfile)
guess Guess what packages are needed by your project
show-specfile Print the filename of the specfile
show-lockfile Print the filename of the lockfile
show-package-dir Print the directory where packages are installed
help Help about any command
Flags:
-h, --help display command-line usage
--ignored-packages strings packages to ignore when guessing (comma-separated)
-l, --lang string specify project language(s) manually
-q, --quiet don't show what commands are being run
-v, --version display command version
Use "upm [command] --help" for more information about a command.
Here are useful things to know that aren't obvious:
- Language detection: Your project's language is autodetected by
the files in the current directory. This can be overridden either
partially or completely by specifying a value for the
-loption. You can see the available languages by runningupm list-languages. In addition to a full language (e.g.python-python3-poetry), you can specify something simpler (e.g.python,python3,python2,poetry,python-poetry). In that case, UPM will examine all of the matching languages and pick whichever one it thinks is best. You can experiment with this logic by providing the-loption toupm which-language. - Information flow: Conceptually, information about packages flows
one way in UPM: add/remove -> specfile -> lockfile -> installed
packages. You run
upm addandupm remove, which modifies the specfile, and then the lockfile is automatically generated from the specfile, and then packages are automatically installed or uninstalled according to the lockfile. Just runningupm addorupm removewill automatically perform all of these steps. Skipping steps is unfortunately not supported, because few package managers support that. You can however run only later steps in the pipeline by means of theupm lockandupm installcommands. - Caching: UPM maintains a simple JSON cache in the
.upmsubdirectory of your project, in order to improve performance. This is used to (1) skip generating the lockfile from the specfile if the specfile hasn't changed since last time; (2) skip reinstalling packages from the lockfile if the lockfile hasn't changed since last time; and (3) skip doing a full analysis of your code onupm guessif your imports haven't actually changed since last time (according to a quick regexp search). To reset the cache, you can delete that directory. However, this shouldn't be necessary very often, because you can use the--force-lockand--force-installoptions toupm add,upm remove,upm lock, andupm install(it is just--forceforupm installdue to lack of ambiguity) in order to ignore the cache for cases (1) and (2).
UPM_PROJECT: path to top-level directory containing project files. UPM uses this as its working directory. Defaults to the first parent directory containing a directory entry named.upm(like Git searches for.git), or the current directory if.upmis not found.UPM_PYTHON2: if nonempty, use instead ofpython2when invoking Python 2.UPM_PYTHON3: if nonempty, use instead ofpython3when invoking Python 3.UPM_SILENCE_SUBROUTINES: if nonempty, then enable-qwhen running commands that are not directly related to the operation the user requested (e.g. if runningupm add, enable-qwhen reading the specfile to check which packages are already added).UPM_STORE: path of file used to store the JSON cache file, relative or absolute. Defaults to.upm/store.json.
UPM itself has no dependencies. It is a single statically-linked binary. However, if you wish to actually use it to manage packages for a language, then the relevant language package manager needs to be installed, as follows:
python-python3-poetry/python-python2-poetry- Python 2/3
- Pip for appropriate version(s) of Python
- Poetry for appropriate version(s) of Python
nodejs-yarnruby-bundlerelisp-cask
All of these dependencies are already installed in the
replco/upm:full Docker image.
$ make help
usage:
make upm Build the UPM binary
make dev Run a shell with UPM source code and all package managers inside Docker
make light Build a Docker image with just the UPM binary
make full Build a Docker image with the UPM binary and all package managers
make doc Open Godoc in web browser
make deploy Publish UPM snapshot Docker images to Docker Hub
make pkgbuild Update and test PKGBUILD
make clean Remove build artifacts
make help Show this message
To build UPM, run make upm (or just make). This requires an
installation of Go. Then add the directory
./cmd/upm to your $PATH so that you can run the binary. To remove
build artifacts, run make clean.
You can use Docker to avoid needing to
install the package managers that UPM drives. To do this, run make dev. This will build an image and launch a shell inside the container
with the UPM source directory on your computer synced with the
filesystem inside the container. The same Makefile targets are
available, and UPM is added to the $PATH automatically. You only
need to restart the shell if you edit the Dockerfile or the scripts
used by the Dockerfile. Aliases available inside the shell:
l:ls -lAhFmt: create temporary directory and cd to it (convenient for switching to a new "project" context)u: build UPM binary if source code has been modified, then run with given argumentsub: same asu, but force rebuilding binary (may be useful if you previously built outside the Docker container)
To build a Docker image which has only the UPM binary, for embedding
in other images, run make light. The image will be tagged as
upm:light. Alternatively, to build a Docker image which has the
binary and all the package managers, but not the UPM source code, run
make full. The image will be tagged as upm:full. These two images
are automatically built and deployed to Docker
Hub when a commit is merged to
master.
UPM does not currently have any tests; however, we plan to fix this.
Whenever a commit is merged to master, snapshot Docker images are
built and pushed to Docker Hub by CircleCI. Whenever a tagged release
(e.g. v1.0) is pushed to GitHub, the following happens:
- Release Docker images are tagged and pushed to Docker Hub.
- The changelog is parsed and published as a GitHub
Release with the following assets:
- source code (.zip and .tar.gz)
- binary (.tar.gz; darwin, freebsd, linux, and windows; 386 and amd64)
- Debian package (.deb; 386 and amd64)
- RPM package (.rpm; 386 and amd64)
- Snappy package (.snap; 386 and amd64)
- checksums (.txt)
- The Repl.it Homebrew tap is updated.
- The Repl.it Scoop bucket is updated.
Once you push a release and it passes CI, the following must be done manually:
- Edit
packaging/aur/PKGBUILDwith the new version and runmake pkgbuildto update and test the AUR package. Then push a new commit. (Once we are allowed to publish to AUR, you should also pushpackaging/aurthere.)