tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell #31216

mkoeppe · 2021-01-09T19:19:23Z

We add the following to the local-... environments:

an environment variable that can be passed to tox to skip system package installs, directly reusing a previously set up system

SKIP_SYSTEM_PKG_INSTALL=yes tox -e local-homebrew-macos-standard -- config.status
SKIP_SYSTEM_PKG_INSTALL=yes tox -e local-homebrew-macos-standard -- config.status

This can save time and also give developers more control for experiments with system packages.

a target that gives an interactive shell in the tox environment:

tox -e local-homebrew-macos-standard -- bash
tox -e local-direct -- bash
SKIP_SYSTEM_PKG_INSTALL=yes SKIP_BOOTSTRAP=1 SKIP_CONFIGURE=1 tox -e local-homebrew-macos-standard -- bash

Depends on #30944

CC: @tobiasdiez @kliem @dimpase

Component: porting

Author: Matthias Koeppe

Branch/Commit: 2d84b5a

Reviewer: Tobias Diez

Issue created by migration from https://trac.sagemath.org/ticket/31216

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-01-09T19:52:17Z

Dependencies: #30944

mkoeppe · 2021-01-09T21:15:15Z

Branch: u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell

mkoeppe · 2021-01-09T21:17:51Z

Commit: 13f58fc

mkoeppe · 2021-01-09T21:17:51Z

Last 10 new commits:

`e5fe752`	`Merge tag '9.3.beta4' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available`
`78ff9d5`	`src/sage/misc/package.py: Add one more # optional - build`
`a44042f`	`Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap`
`64bde5f`	`Merge tag '9.3.beta5' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available`
`e9a7572`	`src/sage/misc/package.py: Improve source formatting`
`c7bcda9`	`Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap`
`9988c5f`	`ci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq`
`ab19133`	`Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard`
`c6ad06e`	`Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31216/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell`
`13f58fc`	`tox.ini (local): Run an interactive shell if posargs are bash`

sagetrac-git · 2021-01-10T00:46:52Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ace4ef6`	`tox.ini (local): Add option SKIP_SYSTEM_PKG_INSTALL=1`

sagetrac-git · 2021-01-10T00:46:52Z

Changed commit from 13f58fc to ace4ef6

mkoeppe · 2021-01-10T00:47:17Z

Author: Matthias Koeppe

tobiasdiez · 2021-01-10T12:57:48Z

comment:8

On a first glance this looks good. Thanks! Could you please also add documentation for these tox commands / switches. Also I was wondering if one could add a tox -e local-homebrew-macos-standard -- install-system-packages that only installs the system packages?

sagetrac-git · 2021-01-10T21:07:51Z

Changed commit from ace4ef6 to ddc4f0b

sagetrac-git · 2021-01-10T21:07:51Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ddc4f0b`	`tox.ini (local): Handle environment variables SKIP_BOOTSTRAP, SKIP_CONFIGURE`

sagetrac-git · 2021-01-11T04:36:37Z

Branch pushed to git repo; I updated commit sha1. New commits:

`bb6aec4`	`src/doc/en/developer/portability_testing.rst: Updates`
`6253bd8`	`src/doc/en/developer/portability_testing.rst: New section on SKIP_... environment variables`

sagetrac-git · 2021-01-11T04:36:37Z

Changed commit from ddc4f0b to 6253bd8

mkoeppe · 2021-01-11T04:38:12Z

comment:12

Replying to @tobiasdiez:

Could you please also add documentation for these tox commands / switches.

Done

Also I was wondering if one could add a tox -e local-homebrew-macos-standard -- install-system-packages that only installs the system packages?

I have instead added more SKIP_... options.

tobiasdiez · 2021-01-11T15:23:21Z

comment:13

Replying to @mkoeppe:

I have instead added more SKIP_... options.

Thanks! Would it also make sense to add one to skip make? (I remember you once told me how to do this, I think, it was via config.status or something - but maybe the version using skip is closer in spirit anyway).

I'll review this once #30944 is merged in develop.

mkoeppe · 2021-01-11T19:10:53Z

comment:14

Replying to @tobiasdiez:

Would it also make sense to add one to skip make?

You can just pass a make target that does nothing - such as Makefile

tobiasdiez · 2021-01-12T12:56:13Z

comment:15

local-sudo-standard run-test: commands[1] | bash -c 'case "" in 1|y*|Y*);; *) eval $(build/bin/sage-print-system-package-command $(build/bin/sage-guess-package-system) --sudo update) ;;'/usr/bin/bash: -c: line 1: syntax error: unexpected end of file

I think there is a "esac" missing at the end of these two lines:


+    local-{root,sudo}:    bash -c 'case "{env:SKIP_SYSTEM_PKG_INSTALL:}" in 1|y*|Y*);; *) eval $(build/bin/sage-print-system-package-command {env:SYSTEM} {env:__SUDO:} update) ;;'
+    local-{root,sudo}:    bash -c 'case "{env:SKIP_SYSTEM_PKG_INSTALL:}" in 1|y*|Y*);; *) PACKAGES=$(build/bin/sage-get-system-packages {env:SYSTEM} $(PATH=build/bin:$PATH build/bin/sage-package list {env:SAGE_PACKAGE_LIST_ARGS}) _bootstrap); eval $(build/bin/sage-print-system-package-command {env:SYSTEM} {env:__SUDO:} --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" ;;'

tobiasdiez · 2021-01-12T13:48:49Z

comment:16

Replying to @mkoeppe:

Replying to @tobiasdiez:

Would it also make sense to add one to skip make?

You can just pass a make target that does nothing - such as Makefile

tox -e local-sudo-standard Makefile with SKIP_BOOTSTRAP: yes and SKIP_CONFIGURE: yes still runs bootstrap and base-toolchain in the make command, which then fails since configure was not called before.

mkoeppe · 2021-01-12T18:15:36Z

comment:17

Ah, of course. Make that configure.ac instead of Makefile...

tobiasdiez · 2021-01-18T17:05:16Z

comment:18

That worked, thanks! Apart from the missing "esac" mentioned above, this looks good to me.

sagetrac-git · 2021-01-18T19:33:13Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2d84b5a`	`tox.ini: Add missing esac`

sagetrac-git · 2021-01-18T19:33:13Z

Changed commit from 6253bd8 to 2d84b5a

tobiasdiez · 2021-01-19T15:44:28Z

comment:21

Is the prefix [mkoeppe@sage sage]$ really needed / what's its purpose?
sagemath/sagetrac-mirror@u/mkoeppe/tox__improve_local_sudo_ubuntu_standard...u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell#diff-c13a55474e037a35696cfba4768b81295c0383f1940052c5fcbef0737387f165R610

Otherwise the changes look good to me.

mkoeppe · 2021-01-19T19:06:41Z

comment:22

Replying to @tobiasdiez:

Is the prefix [mkoeppe@sage sage]$ really needed / what's its purpose?

It's a typical shell prompt on posix systems. The second sage indicates the current directory. This is consistent with other parts of the Sage developer guide, for example https://doc.sagemath.org/html/en/developer/doctesting.html

tobiasdiez · 2021-01-19T21:18:01Z

comment:23

Thanks, then it looks good to me!

mkoeppe · 2021-01-19T21:18:57Z

comment:24

Thanks!

mkoeppe · 2021-01-19T21:18:57Z

Reviewer: Tobias Diez

vbraun · 2021-01-31T20:53:33Z

Changed branch from u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell to 2d84b5a

mkoeppe added this to the sage-9.3 milestone Jan 9, 2021

mkoeppe added c: porting labels Jan 9, 2021

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Jan 10, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell~~ tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell Jan 10, 2021

tobiasdiez added s: needs work and removed s: needs review labels Jan 12, 2021

mkoeppe added s: needs review and removed s: needs work labels Jan 18, 2021

tobiasdiez added s: positive review and removed s: needs review labels Jan 19, 2021

vbraun removed the s: positive review label Jan 31, 2021

vbraun closed this as completed in 6a5ad3f Jan 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell #31216

tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell #31216

mkoeppe commented Jan 9, 2021

mkoeppe commented Jan 9, 2021

mkoeppe commented Jan 9, 2021

mkoeppe commented Jan 9, 2021

This comment has been minimized.

mkoeppe commented Jan 9, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jan 10, 2021

sagetrac-git mannequin commented Jan 10, 2021

mkoeppe commented Jan 10, 2021

This comment has been minimized.

tobiasdiez commented Jan 10, 2021

sagetrac-git mannequin commented Jan 10, 2021

sagetrac-git mannequin commented Jan 10, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jan 11, 2021

sagetrac-git mannequin commented Jan 11, 2021

mkoeppe commented Jan 11, 2021

tobiasdiez commented Jan 11, 2021

mkoeppe commented Jan 11, 2021

tobiasdiez commented Jan 12, 2021

tobiasdiez commented Jan 12, 2021

mkoeppe commented Jan 12, 2021

tobiasdiez commented Jan 18, 2021

sagetrac-git mannequin commented Jan 18, 2021

sagetrac-git mannequin commented Jan 18, 2021

tobiasdiez commented Jan 19, 2021

mkoeppe commented Jan 19, 2021

tobiasdiez commented Jan 19, 2021

mkoeppe commented Jan 19, 2021

mkoeppe commented Jan 19, 2021

vbraun commented Jan 31, 2021

tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell #31216

tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell #31216

Comments

mkoeppe commented Jan 9, 2021

mkoeppe commented Jan 9, 2021

mkoeppe commented Jan 9, 2021

mkoeppe commented Jan 9, 2021

This comment has been minimized.

mkoeppe commented Jan 9, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jan 10, 2021

sagetrac-git mannequin commented Jan 10, 2021

mkoeppe commented Jan 10, 2021

This comment has been minimized.

tobiasdiez commented Jan 10, 2021

sagetrac-git mannequin commented Jan 10, 2021

sagetrac-git mannequin commented Jan 10, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jan 11, 2021

sagetrac-git mannequin commented Jan 11, 2021

mkoeppe commented Jan 11, 2021

tobiasdiez commented Jan 11, 2021

mkoeppe commented Jan 11, 2021

tobiasdiez commented Jan 12, 2021

tobiasdiez commented Jan 12, 2021

mkoeppe commented Jan 12, 2021

tobiasdiez commented Jan 18, 2021

sagetrac-git mannequin commented Jan 18, 2021

sagetrac-git mannequin commented Jan 18, 2021

tobiasdiez commented Jan 19, 2021

mkoeppe commented Jan 19, 2021

tobiasdiez commented Jan 19, 2021

mkoeppe commented Jan 19, 2021

mkoeppe commented Jan 19, 2021

vbraun commented Jan 31, 2021