Loading: look for vendored packages in $project/packages

[StefanKarpinski]

No description provided.

[DilumAluthge]

I like it.

Can you add "closes #35195"?

[fredrikekre]

Suggested change

	if project !== nothing
	if project !== nothing && project in load_path()

[DilumAluthge]

Could you explain this change?

[fredrikekre]

Seems weird to look for stuff in a path that is not on the LOAD_PATH.

[DilumAluthge]

Sure. I think this will be true in 99% of cases anyway. The only way that the active project wouldn’t be the first entry in the load path would be if someone manually popped it off.

[fredrikekre]

Yes, so my suggestion fixes that.

[StefanKarpinski]

The situation where this can happen is if @ is removed from the LOAD_PATH. I have to think a little about whether it makes sense for that to interact with this or not...

Then this can be simplified to this:

Suggested change

	if project !== nothing
	if project in load_path()

That's because load_path already filters out nothing values and only returns strings.

[KristofferC]

How would you get the packages into packages? You need to set DEPOT_PATH[1] to the package directory, remove the other depots and instantiate, then remove the registry and artifact directories and then the leftover packages directory works as an implicit package depot for the specific project?

What is the use case for this? I feel this PR lacks some kind of description / justification of what the goal is to enable and why the current methods are not satisfactory.

[DilumAluthge]

How would you get the packages into packages?

Simple approach:

1. cp ~/.julia/packages -Rf /path_to_my_project/packages

Advanced approach:

1. Decide which packages you want to "bundle" in your "standalone bundle."
2. For each such package, do cp ~/.julia/packages/packagename/packageslug -Rf /path_to_my_project/packages/packagename/packageslug

The plan is that someone will develop an external tool to automate that process. We don’t need to add the functionality to Julia for creating the "self-contained standalone bundles." That can go in an external tool. We only need to add support for reading from the standalone bundles.

[DilumAluthge]

The use case is to be able to make a single directory that contains your project, as well as all of the packages and artifacts that your project uses. You can then make a single tarball from that directory and:

1. Upload the tarball e.g. Zenodo
2. Send the tarball to e.g. journal reviewers
3. Upload the tarball to a form that e.g. is submitted to an administrator that reviews all source code before it can be transferred onto a classified computer system.
4. Copy the tarball onto write-once media (e.g. DVD-ROM), and then use the write-once media to transfer the tarball to e.g. an airgapped computer system with no Internet access.

I can imagine other use cases as well. I’ve started calling these "self-contained bundles" or "standalone bundles".

[KristofferC]

Simple approach:

Yeah but that will bring you all of the packages and all versions of them.

The use case is to be able to make a single directory that contains your project, as well as all of the packages and artifacts that your project uses.

Yeah, you would do this now by adding

Package/.julia/packages/...
Package/.julia/artifacts/...

and doing push!(DEPOT_PATH, "Package/.julia"). Is all of this to avoid having to do that push!?

[KristofferC]

Regarding #35222 (comment), I agree with that a specific .julia directory or similar would be good since people will want to add more things artifacts, in the future perhaps sysimages etc. Best to keep all that in one directory to prevent polluting the top level package too much.

[tkf]

and doing push!(DEPOT_PATH, "Package/.julia"). Is all of this to avoid having to do that push!?

I think it is important to have a canonical name for path like this. Here is a possible source of inspiration: There is something called zipapp (see also PEP 441) in Python stdlib that can be used to turn a zipped Python package into an executable application. I think similar approach is possible if we have a canonical name for the project-local depot(-like) directory path. We need a specific name for this since it has to be understood by tools (including julia binary itself). Of course, I'm not suggesting to do this here/now. But the approach like this PR seems to be the right direction to get there. (ref #34759)

[StefanKarpinski]

Just to clarify the thinking here: what we want is the ability download a tarball of an application and run it with the guarantee that it will not need to download any resources (packages or artifacts) in order to run. We may want to go a bit further and say that the app will use only resources within the tarball, not any system-installed resources. This illuminates a few things:

• We do want to allow projects to provide content-addressed resources (packages, artifacts) because that's what's required to satisfy the goal.

• We don't want to allow projects to affect the behavior of the package manager in other ways—that's both undesirable (potentially dangerous, confusing) and not a goal.

These dos and don'ts are why treating the project as a full-blown depot is not desirable. Looking for packages and artifacts in the project is not arbitrary or a slippery slope: these are the only content-addressed resources that exist in Julia/Pkg; together they are necessary and sufficient to allow a project to be self-contained in terms of dependencies.

The idea that we want to enable shipping apps in such a way that they run only using the resources in the project is why it's desirable to look at a project's packages and artifacts directories first: if the the project ships with all the necessary resources, then it's guaranteed that when running the project you won't hit the filesystem outside of the project at all. While content addressing should guarantee that it doesn't matter where we load content-addressed resources from, I think it still makes sense to make projects fully self-contained in the sense that we don't even look outside of the project for resources.

The sysimg point is interesting: that's something that is not saved in depots, so it would be specific to projects only. @KristofferC, can you elaborate on how you imagine that working?

[tkf]

Re sysimage, here is an RFC/POC: #35794