Make it easier to run Openverse from an offline media

[dhruvkb]

Fixes

Fixes #4478 by @dhruvkb

Description

This PR makes Openverse easier to install from an offline media. It does so by providing recipes that can

• clean all env files from the repo to make sure that are no more secrets: just unenv
• dump all Docker images needed for Openverse into .tar.gz files: just docker/dump-3p && just docker/dump-lint && just docker/dump-ov
• load all Docker images from the images directory: just docker/load.

Testing Instructions

0. Dump all Docker images using the command above.
1. Clear your Docker images using docker image prune -a.
2. Load the images back using the command above.
3. Run Openverse, it should not need to download anything from the Internet.

Checklist

• My pull request has a descriptive title (not a vague title likeUpdate index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.
• I ran the DAG documentation generator (just catalog/generate-docs for catalog
  PRs) or the media properties generator (just catalog/generate-docs media-props
  for the catalog or just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[krysal]

@dhruvkb Is this paused/blocked into something? What is it missing to be ready?

[dhruvkb]

@krysal it's moving pretty slowly. I'd like to keep it open for a couple more weeks, if that's okay.

[dhruvkb]

I have marked this as ready for review but one of the issues with this is I cannot test it between two computer. There isn't a USB file format that'll natively support both Mac and Linux, which are the two computers I have.

If anyone has two Macs or two Linux computers, please create a USB drive (with APFS or ext4 file system respectively) and try it out.

[sarayourfriend]

There isn't a USB file format that'll natively support both Mac and Linux, which are the two computers I have.

exFAT should be the one to use here, it's the portable filesystem to use these days, what prevents it from being used?

[zackkrida]

Yes, I personally have two usb drives formatted exFAT that I use to transfer files between multiple machines. I'll test this!

[dhruvkb]

exFAT does not support UNIX-like permissions so I've seen in the past that copying files over it can cause permission issues. So it's probably better to use a format that's native to the OS.

[sarayourfriend]

exFAT does not support UNIX-like permissions so I've seen in the past that copying files over it can cause permission issues

This seems like an error in using exFAT? Are there file permissions that need to be maintained from the machine that generated the images onto the new machine? That seems highly unlikely considering these are images meant to be shared?

KDE and GNOME both mount exFAT devices with the user and group options set to the logged in user, so I'm not sure when this would be relevant unless mounting the device via mount directly, in which cause surely we can assume the correct options will be known by the person mounting the device?

Are you suggesting that we would need to have at least two USBs, potentially three (if we allow for someone arriving relying on WSL), all with different filesystems native to the OS being used?

[dhruvkb]

We have determined that this is not a complete solution because of compatibility issues affecting Docker images, compiled Python and Node packages and USB drive file systems. A comprehensive solution would be overengineered, unnecessary and unused except for 2-3 days a year.

[sarayourfriend]

We also talked about looking into github.dev again, and seeing if we can make it work, perhaps building on the foundation of ov to create a Dockerised development environment.