Mechanism for ensuring datasette/datasette-app-support/etc are at required version

[simonw]

The new CSV menu item in #29 only works if datasette-app-support [.2 ol higher is installed - but the user may have an older ~/.datasette-app/venv virtual environment running.

So there needs to be some kind of mechanism to notice that the plugin is an unsupported version and upgrade it.

[simonw]

As I add more default plugins in #81 the scope of this issue has expanded to ensuring all of the default dependencies are installed, even if the virtual environment already exists.

Relevant code:

datasette-app/main.js

Lines 186 to 207 in 79fd738

	async ensureDatasetteInstalled() {
	const datasette_app_dir = path.join(process.env.HOME, ".datasette-app");
	const venv_dir = path.join(datasette_app_dir, "venv");
	const datasette_binary = path.join(venv_dir, "bin", "datasette");
	if (fs.existsSync(datasette_binary)) {
	return datasette_binary;
	}
	if (!fs.existsSync(datasette_app_dir)) {
	await mkdir(datasette_app_dir);
	}
	if (!fs.existsSync(venv_dir)) {
	await execFile(findPython(), ["-m", "venv", venv_dir]);
	}
	const pip_path = path.join(venv_dir, "bin", "pip");
	await execFile(pip_path, [
	"install",
	"datasette==0.59a2",
	"datasette-app-support>=0.5",
	]);
	await new Promise((resolve) => setTimeout(resolve, 500));
	return datasette_binary;
	}

[simonw]

~/.datasette-app/venv/bin/pip list --format json

Returns the list of currently installed packages as JSON:

[
  {
    "name": "datasette",
    "version": "0.59a2"
  },
  {
    "name": "datasette-app-support",
    "version": "0.6"
  },
  {
    "name": "datasette-cluster-map",
    "version": "0.17.1"
  }
]

[simonw]

I can compare version numbers correctly using from packaging.version import parse as parse_version.

Or if I do the comparisons in JavaScript I can use https://www.npmjs.com/package/semver

[simonw]

The plan:

• Every time the app is started, use pip list --format json to obtain a list of currently installed packages in that virtual environment
• The Electron app carries a list of required dependencies and their minimum versions (which gets updated when I release a new version of the app shell) - if any of the versions in the virtual environment are below that number, they get upgraded before the server starts
• One edge-case: if the user has uninstalled (Ability to uninstall a plugin #72) one of the default plugins (from Bundle a sensible collection of default plugins #81) I will need to remember that they uninstalled it and NOT re-install it on startup. This means I'll need to persist the list of deliberately uninstalled plugins somewhere. I'll handle this as part of issue Ability to uninstall a plugin #72.

[simonw]

Annoyingly, semver said "TypeError: Invalid Version: 0.59a2" about one of my version numbers. Apparently I've not been using valid semver!

semver.coerce(item.version) is meant to be able to coerce broken versions, but 0.59a2 comes out as 0.59.0 using that, which isn't right.

[simonw]

I used this code to explore the behaviour of semver with various options:

    const versionsProcess = await execFile(pip_path, [
      "list",
      "--format",
      "json",
    ]);
    const versions = {};
    for (const item of JSON.parse(versionsProcess.stdout)) {
      versions[item.name] = {
        original: item.version,
        clean: semver.clean(item.version),
        clean_loose: semver.clean(item.version, { loose: true, includePrerelease: true }),
        coerce: semver.clean(item.version),
        coerce_loose: semver.clean(item.version, { loose: true, includePrerelease: true }),
      };
    }

Partial output:

  aiofiles: {
    original: '0.7.0',
    clean: '0.7.0',
    clean_loose: '0.7.0',
    coerce: '0.7.0',
    coerce_loose: '0.7.0'
  },
  aniso8601: {
    original: '7.0.0',
    clean: '7.0.0',
    clean_loose: '7.0.0',
    coerce: '7.0.0',
    coerce_loose: '7.0.0'
  },
  anyio: {
    original: '3.3.0',
    clean: '3.3.0',
    clean_loose: '3.3.0',
    coerce: '3.3.0',
    coerce_loose: '3.3.0'
  },
  'asgi-csrf': {
    original: '0.9',
    clean: null,
    clean_loose: null,
    coerce: null,
    coerce_loose: null
  },
  asgiref: {
    original: '3.4.1',
    clean: '3.4.1',
    clean_loose: '3.4.1',
    coerce: '3.4.1',
    coerce_loose: '3.4.1'
  },
  certifi: {
    original: '2021.5.30',
    clean: '2021.5.30',
    clean_loose: '2021.5.30',
    coerce: '2021.5.30',
    coerce_loose: '2021.5.30'
  },
  'charset-normalizer': {
    original: '2.0.4',
    clean: '2.0.4',
    clean_loose: '2.0.4',
    coerce: '2.0.4',
    coerce_loose: '2.0.4'
  },
  click: {
    original: '8.0.1',
    clean: '8.0.1',
    clean_loose: '8.0.1',
    coerce: '8.0.1',
    coerce_loose: '8.0.1'
  },
  'click-default-group': {
    original: '1.2.2',
    clean: '1.2.2',
    clean_loose: '1.2.2',
    coerce: '1.2.2',
    coerce_loose: '1.2.2'
  },
  datasette: {
    original: '0.59a2',
    clean: null,
    clean_loose: null,
    coerce: null,
    coerce_loose: null
  },
  'datasette-app-support': {
    original: '0.6',
    clean: null,
    clean_loose: null,
    coerce: null,
    coerce_loose: null
  },
  'datasette-cluster-map': {
    original: '0.17.1',
    clean: '0.17.1',
    clean_loose: '0.17.1',
    coerce: '0.17.1',
    coerce_loose: '0.17.1'
  }

It's returning null for anything that doesn't have at least a.b.c.

[simonw]

I'm using version numbers (hopefully) that are valid https://www.python.org/dev/peps/pep-0440/ - which it turns out isn't quite compatible with the semver library: semver/semver#483

[simonw]

I just realized I'm re-inventing what pip install datasette>=0.59a2 datasette-app-support>=0.6 is already doing for me - I turned off WiFi to confirm that pip doesn't make any network requests if the required minimum versions are already available.

So it's safe for me to run pip install ... on startup! It won't make a network request if the packages are there already unless I include -U.