Introduce user-defined wrapdb mirror

[klokik]

This consolidates all queries to the wrapdb to go through the open_wrapdburl() function. The function handles a domain-specific URL scheme, wrapdb. When encountered, it substitutes the scheme with https and an authority(net loc) with either the upstream wrapdb address or a user-defined one stored in the subprojects/wrap-mirrors.json file.

The user should use meson wrap set-db-address <mirror-address> command to create the file. It looks a bit more secure to me than using an environment variable.

Justification:

• wrapdb servers may not always be available
• internal organization policies prohibit using external sources during the build process
• wraps for non-public proprietary subprojects

[jpakkane]

user-defined one from the MESON_WRAPDB_MIRROR environment variable.

This has the potential to be a really nasty way of hijacking user queries for malware injection. Not to mention all the other ways in which environment variables are terrible for storing persistent configuration.

[klokik]

@jpakkane you mean adding more malware, once attacker already gained control over the users shell environment? Or getting the first handful of malware by user getting source code from the untrustworthy mirror?

I'd be glad to hear for alternative more secure options. Providing the mirror address as a meson wrap command argument on each invocation would be acceptable for me, but less convenient in day-to-day use.
Other than that meson does not seem to use any persistent configuration. This does not belong to the machine files either.
How about a meson wrap set-mirror (address) that will be stored in the subprojects dir, along the local version of db?

Getting code from untrusted sources IMO is a user problem. I honestly don't expect anyone to use random third party mirror found on the search engine.

[klokik]

Ok, I can see this being abused on a build machine, as a stepping ladder of a more complex attack.What's worse - it will affect users that are not aware of this variable.

[klokik]

@jpakkane, I changed it to use an explicit user-provided address, unaffected by environment variables, limited to a single workspace.

It looks tempting to store the address inside subprojects/wrap.json along with the revisions it provides, but this will require breaking file layout and annoyance for the end-users.

[dnicolodi]

I am not sure I like the UI for this feature. I think that the use case you have in mind for it is very different from what I have in mind. Why isn't a command line argument to meson wrap enough? If this is a functionality intended mostly for corporate environment where access to the "upstream" wrap database is somehow forbidden, doesn't a network level redirect provide a transparent solution for the same problem?

[klokik]

Command-line argument will have to be set for each of the update and install sub-commands, and transparent fallback can not happen unless the address is stored somewhere.

May I propose an optional positional argument to the update-db command that will create the "persistent" file?

Actually, you caught my mental confusion on what what it is for. I'm calling it a mirror, but it is not what I want it for - I want a different wrap-db-source.

This is indeed for a corporate environment, but a simple redirect is not acceptable. All sources should be stored on the company-controlled machines, wraps and source updates have to be audited to an extent. Wraps for the subprojects developed by other teams, that have no interest in using meson yet are to be served from this source as well.

[dnicolodi]

This is indeed for a corporate environment, but a simple redirect is not acceptable. All sources should be stored on the company-controlled machines, wraps and source updates have to be audited to an extent. Wraps for the subprojects developed by other teams, that have no interest in using meson yet are to be served from this source as well.

I don't understand why a redirect is not acceptable. If that is your policy, most likely you have measures in place that forbid access to https://wrapdb.mesonbuild.com. You just need to change those measures to redirect access to that resource to https://wrapdb.localnet or whatever you like. Then you just serve your own wrap-db from that URL. Your wrap-db may contain your hand-picked selection of wraps. Why would this not work?

[dnicolodi]

May I propose an optional positional argument to the update-db command that will create the "persistent" file?

AFAIK running meson wrap update-db is not required to use other meson wrap commands. Therefore, making the configuration of the wrap database URL dependent on running it seems at least counterintuitive.

One thing I haven't understood yet is whether you want a setting that is per system, per user, per project, or per specific wrap. Depending on what is the use case the configuration would need to be stored in a different location.

I am also not convinced that there is the need of a meson command to set the WrapDB URL instead that a simple configuration file format.

[klokik]

You just need to change those measures to redirect access to that resource to https://wrapdb.localnet or whatever you like. Then you just serve your own wrap-db from that URL. Your wrap-db may contain your hand-picked selection of wraps. Why would this not work?

1. It will break TLS
2. Let's say that policy is only enforced on the CI machines
3. I have no control over this redirect
4. meson is overly attached to this specific address hardcoded in a few places

[klokik]

One thing I haven't understood yet is whether you want a setting that is per system, per user, per project, or per specific wrap. Depending on what is the use case the configuration would need to be stored in a different location.

Ideally setting wrapdb source per project with all of its subprojects. + A single instance of wrap server on the network.

AFAIK running meson wrap update-db is not required to use other meson wrap commands. Therefore, making the configuration of the wrap database URL dependent on running it seems at least counterintuitive.

Certainly, it will not require update-db with the changes proposed in this MR. Only commands that query wrapdb need the update command whether with the source set or not. I don't think that one is supposed to have wrapdb.json file checked-in, and without it status/info/search/list won't work. I see what you mean now: other commands do query release data directly from the server bypassing local wrapdb.json file. I don't like it.

[klokik]

May I propose an optional positional argument to the update-db command that will create the "persistent" file?

I did it, but I don't like it. Will revert to a separate wrap subcommand.

[dnicolodi]

I really don't like that this does not allow to set a protocol. For now the wrap database speaks only https, but I can imagine a future where a different protocol is implemented.

[klokik]

Like ftp, sftp, git? This will require parsing urls differently. An ssh/git user might want to pass a key somehow. Doable, but not sure if I want to open this can of worms yet.

Having a file: or plain http: protocol will be convenient for local testing, but not really useful otherwise.

[dnicolodi]

I don't understand what you mean with "opining this can of worms". You are making the location of the wrap database configurable. What I am asking is to do not assume that the protocol is https and make the protocol part of the URL part of the configuration string instead of hard-coding it to be https. Where are the worms in this request?

[klokik]

Wish granted. There is enforcement of https in the code with the assumption that the only other alternative is http. I didn't get it right yet. WIP.