Persist package installation across reboots

[peterfeiner]

Rather than installing stuff in /tmp/pkb, which gets wiped out on
reboot, use /opt/pkb. Keeping packages around after a reboot is
essential for re-using the perfkitbenchmarker.linux_packages.* code
for image preparation.

[peterfeiner]

@tedsta another one for you - please take a look.

[cwilkes]

I appreciate making the change look like the previous item but can we use the same posixpath.join(x,y) as used elsewhere?

[peterfeiner]

The code seems to be split about 50/50. You'll notice that most of the places where I changed vm_util.VM_TMP_DIR to INSTALL_DIR use format strings.

I don't mind changing everything to posixpath.join, but I'd prefer to do it as a follow-on commit. I'll change everything, including the stuff that still uses VM_TMP_DIR. I'm also willing to change everything to use format strings -- that'd be my preference since posixpath.join() is just as portable as '%s/%s', but I'm not picky.

[cwilkes]

Looks good

[cwilkes]

Maybe just say installs to INSTALL_DIR so that we won't have to change it again

[peterfeiner]

Done.

[cwilkes]

Is this meant to change all the installation directories in all benchmarks? git grep vm_util.VM_TMP shows what could be a lot of other places where it could be an installation directory.

[tedsta]

Unfortunately I don't think github allows me to comment on code that's not in the vicinity of changes, but:

In linux_virtual_machine.py in the PackageCleanup, SnapshotPackages, and RestorePackages methods of BaseLinuxMixin, vm_util.VM_TMP_DIR is still used. I don't think it breaks anything but it'd be nice to remain consistent.

I noticed PackageCleanup is only ever used for static VMs. Did you leave out deleting /opt/pkb in there for a reason?

[peterfeiner]

@tedsta wrote:

In linux_virtual_machine.py in the CleanupPackages, SnapshotPackages, and RestorePackages methods of BaseLinuxMixin, vm_util.VM_TMP_DIR is still used. I don't think it breaks anything but it'd be nice to remain consistent.

Good catch. I've fixed these.

I noticed CleanupPackages is only used for static VMs. Did you leave out deleting /opt/pkb in there for a reason?

I'm not sure where you mean. There isn't any function called CleanupPackages anywhere.

[peterfeiner]

@cwilkes wrote:

Is this meant to change all the installation directories in all benchmarks? git grep vm_util.VM_TMP shows what could be a lot of other places where it could be an installation directory.

I had looked through with git grep 'temp\|tmp\|TempDir\|temp_dir\|TMP_DIR' perfkitbenchmarker/linux_packages, but, as your command shows, there are quite a few uses of vm_util.VM_TMP outside of linux_packages. I'll go through all of the matches I get from git grep '\<temp\>\|\<tmp\>\|TempDir\|temp_dir\|TMP_DIR'

[tedsta]

Sorry, meant PackageCleanup not CleanupPackages. Edited above comment.

[peterfeiner]

@tedsta I did modify PackageCleanup to delete /opt/pkb instead of /tmp/pkb since none of the packages install stuff to /tmp/pkb anymore.

[peterfeiner]

@cwilkes There were two more places where vm_util.VM_TMP was being used as an installation directory: sysbench05plus and scimark2. I've fixed these. Please take another look.

[tedsta]

This LGTM, but maybe we should wait for @cwilkes to give his approval too

[cwilkes]

Looks good to me