Package manager & build system

[CyberTailor]

What are your thoughts on how Cyo software and libraries should be configured -> built -> tested -> installed?

• Should it be compatible with the nimble spec?
• If not, should it be integrated with Meson or CMake?
• Central package registry or decentralized module system?
• What about dependency resolution?

It's better to discuss this with both developers and package maintainers.

Just to be clear, it's not about building Cyo itself.

[haxscramper]

I've been collecting some thoughts on this matter, and for now the answers are:

• no if you mean package.nimble format -- it has many issues detailed in Nimble improvements nim-lang/RFCs#398 and should be replaced by simple declarative format
• technically no, but only because package manager should only set up the environment nim.cfg with --path statements, so it will effectively make it possible to run cyo in Meson transparently and with correct package information. So technically there should be no need for direct support
• Decentralized module system based on GitHub
• tl;dr: MVS for dep resolution, for more see below

There are many counterpoints to the minimum version selection package managers, most of which talk about how things are "usually" done and that it "does not make sense" or "nobody would manually upgrade the packages". Most of them effectively boil down to the fact that MVS is not the best tool for every single workflow, which is perfectly understandable.

There are also several considerations, both of the technical and ideological nature, that are involved in decision about how package management is done.

• First and foremost – proper implementation of the maximum version selection is a complicated task because the problem itself is a NP-complete. There have been discussions about using a better algorithm for the resolution, but it is still pretty sophisticated, especially if you want to get a UX.

  Minimum version selection, on the other hand, is a trivial and very predictable algorithm that can be written in a hundred lines of code or so.

• Second – usage of the MVS would require package authors to be honest with what their requirements are. If you actually need the 0.4.0 and not the 0.3.0 you should no longer rely on the implicit assumption that PM will "do the right thing".

• Thirdly, package manager won't suddenly pull a different version of the dependency unless you explicitly specify it can do so.

• Extra – it is possible to transform an ecosystem that was bout around minimum version selection into one that favors maximum version selection because it will effectively require relaxing the version ranges. Making transition the other way might be too hard.

• Extra – MVS requirements are less vulnerable to bitrot. If you write the package that uses specific versions, leave it for five years and come back, it will pull the same versions.

Based on above points, it is safe to assume that MVS is a better solution at the start.

Package resolution algorithm is pretty simple:

• Package directory path and git URL of the package to install is supplied to the package manager.
• First package is cloned to the directory path. If a specific version of the package is required, it is checked out.
• Each package contains specification of requirement ranges requirements.toml (name/format are provisional) which is looked over to determine which packages to download next.
• For each newfound package:
  • If a new package is required and is missing, it is downloaded and checked out to a minimum possible version.
  • If a new package is required and had already been downloaded it is checked to a new minimum version, which is computed as max(min_first, min_second)
• New packages are processed recursively and cloned into the package directory as needed.
• After installation process is finished, package manager generates nim.cfg file which contains multiple --path=, pointing to the new package locations.

After installation, any tool can read nim.cfg and get information about
packages.

[haxscramper]

Package manager is most likely to be implemented based on (or using) https://github.com/disruptek/nimph with aforementioned alternations to package dependency description and resolution.

[haxscramper]

Additional bits that I haven't figured out completely yet:

• List of requirements can potentially be replaced to git submodules, but a lot of people find them confusing, they are not as approachable as simple config.toml file.
• Dependency resolution algorithm transparently treats both local and remote packages. If you need to develop a package locally you can use symbolic link and it will be processed as "required and had already been downloaded".
• You can roll any particular dependency to a specified version as needed - either independently or updating versions for all packages. This is very handy for upgrading dependencies.
• Lock files can be trivially implemented by writing dependency hash trees into a separate file.

[CyberTailor]

but only because package manager should only set up the environment nim.cfg with --path statements

Meson and CMake has their own routines for finding dependencies. How to find them: by name (Python-like) or by some scheme, e.g. github.com/foo/[email protected] (Go-like)?

Decentralized module system based on GitHub

Like Go modules? If so:

1. It should support other forges too
2. It should be able to work offline (preferably without bundling dependencies)

MVS for dep resolution, for more see below

Pinned dependencies are usually frowned upon (buggy/vulnerable deps are not ok). Downstream often patches out such restrictions.

package manager generates nim.cfg file

Maybe it should be named differently to avoid writing in a file under VCS control.

[haxscramper]

• Using url + version format, similarly to how it is done now
• 1. Yes, similar to go modules. I have no idea what "forge" refers to, but from the package manager perspective it only needs to provide the code from URL. It can be a self-hosted GitLab, enterprise GitHub, your local directory or anything. Quality-of-life improvements (search by name, upgrade to a fork, fork my dependency to work on this and then push it) are unlikely to be implemented for a wide range of code hosting sites, partially because they might not provide the required API (for real-world example of what I'm talking about you can look at supported nimph features)
• 2. If the environment hasn't changed the package manager wouldn't have to run, so if you installed everything and then don't touch required versions then there would be no need for package manager to run.
  • If you packaged your directory into a zip and copied it somewhere else along with nim.cfg then PM does not need to run.
  • The environment is your nim.cfg (automatically generated) plus dependencies requirement. The former is generated from the latter using MVS algorithm. If there is no need to fetch anything (package has already been downloaded, can be git checkout-ed to required tag) it wouldn't be done

Pinned dependencies are usually frowned upon (buggy/vulnerable deps are not ok). Downstream often patches out such restrictions.

You can increase required version after testing it doesn't break the code and is safe to use.

Maybe it should be named differently to avoid writing in a file under VCS control.

Maybe. Although you can have more than one nim.cfg file and have a directory structure like this

deps/
nim.cfg
project1/
project2/
projcet3/

to avoid re-downloading things for all of your projects (in this case nim.cfg is out of VCS)

or

project/
  nim.cfg
  deps/

in which case nim.cfg is perfectly well-contained in the repository, and its modification does not cause any notable issues because there are pretty benign entries such as

--path="$confg/deps/some_dependency

which are

1. Rarely modified (version of the dependency is not written out to nim.cfg, it is a simple path to the source, so compiler could get its module search path updated)
2. New entries are only added and removed when you add or remove a dependency, which doesn't happen too often and at most would match the edits in deps list -- one-two lines in most cases).
3. It might be possible to add dedicated anything.cfg processing, but it is not clear how to order them priority-wise (what overrides what) -- the current implementation at least can rely on the directory structure.

[haxscramper]

Sorry for the late reply, I hope it answers your questions