Consolidate installer setup steps

[MCGallaspy]

All the steps we take before releasing an installer should be as consolidated as possible, so that we don't duplicate effort across OSes. This issue attempts to document the current state and make a roadmap to consolidation.

@benjaoming made this document, which at the time of this posting is a bit out of date.

The steps I take to create the Windows installer are documented here: https://github.com/learningequality/installers/blob/master/windows/README.md#instructions-to-build-kalitesetupexe
This issue is related: #48

In particular tasks like including documentation, running node build.js, including assessment items, and running kalite manage collectstatic are platform-agnostic. If I understand it correctly, then we can make these and other pre-release tasks the responsibility of setup.py, so that they can be run with a command like python setup.py [your cool command name]. @aronasorman this is the main point of our discussion from 9/4 dev meeting.

Roadmap

• Update description of platform-independent prerelease tasks in this document
• Implement above tasks in setup.py
• Use setuptools for Windows installer (see Windows installer should use setup.py (setuptools) #48)
• Use setuptools for Ubuntu/Debian and OSX (I think we already do, right?)

Milestone for this is up for discussion -- if these changes for each individual task are safe, then we should aim to include them in 0.15. If it will require more extensive work/testing then we should consider it for 0.16.

[MCGallaspy]

@cpauya your input appreciated here too!

[jamalex]

Another example of something we want to do during this "pre-build" stage is: #80

[jamalex]

If I understand it correctly, then we can make these and other pre-release tasks the responsibility of setup.py, so that they can be run with a command like python setup.py [your cool command name]

We should look (in a thorough way) at what the best tool/stage in the pipeline is for doing this "pre-build" step. Another option to examine, for example, would be to use node.js (since we need it for the build process anyway) to coordinate the early stages of the build process.

The overall goal is to build a standard output format (e.g. a zip file, or target "built" directory), with all of the common cross-platform build steps completed, that can then serve as input to the various installers.

[jamalex]

Update description of platform-independent prerelease tasks in this document

This document currently seems focused on the steps to run for prepping things and building/uploading/releasing particular installers, but there's no list of the "pre-build" tasks that we want to automate. @MCGallaspy listed some in the issue description above -- I'll add a section in the doc where we can list this out more fully.

[benjaoming]

The overall goal is to build a standard output format (e.g. a zip file, or target "built" directory), with all of the common cross-platform build steps completed, that can then serve as input to the various installers.

This is the source distribution or in short, sdist :)

It's here, .zip available, too: https://pypi.python.org/pypi/ka-lite-static

[jamalex]

Yes, the sdist zip was what I had in mind as a possible output format here (or at least, one component of it), but I don't know enough about it to know how well it will handle everything we need -- e.g. "bundling in an empty database that gets unpacked to the user directory, but only if there's no DB in the user directory yet".

I added a preliminary list of things we want to do when building from source -> distributable bundle (to use as input to installers). @rtibbles @aronasorman @MCGallaspy @benjaoming, please add things to there as you see fit, so we can assess and discuss:
https://docs.google.com/document/d/1Z06ING-sgMoq5EqZPVK2Zu1cUIpF9SKJunG17Q8Fcr8/edit#heading=h.q1tnequspos4

[benjaoming]

"bundling in an empty database that gets unpacked to the user directory, but only if there's no DB in the user directory yet"

It would have to be a data file that gets installed unconditionally, and then it would have to be the application (kalite manage setup) that makes the assessment of whether or not to bootstrap the user .kalite directory with an empty database. But this could happen several times within the same installation.

One possible gotcha: Downgrading KA Lite. But I think this is a problem already, anyways. So no need to deal with it.

[cpauya]

The OSX installer has a setup.sh script which does everything to build the .app and generate the .dmg (installer).

Just a note that the OSX .app uses Pyrun and not virtualenv yet. But we already have an issue to drop pyrun and use virtualenv which needs to be tested if feasible for OSX.

The script downloads an archive of a ka-lite branch and will run python setup.py install on it.

These are the pre-release steps that the setup script must do so particular packages/data are included in the bundled ka-lite or in the Pyrun's site-packages folder:

1. Download the assessment.zip. -- This is extracted by the .app on the installed machine upon first run.
2. Run pip install requirements.txt.
3. Build the docs
   1. Run pip install -r requirements_sphinx.txt.
   2. Run sphinx-build inside the docs directory.
4. Compile yml to json
   1. Run pip install pyyaml==3.11
   2. Run kalite manage compileymltojson
   3. Run pip uninstall pyyaml

When successful, these are copied into the .appsResources` folder and the .app and it's .dmg are then built:

1. assessment.zip file
2. the Pyrun directory (which includes ka-lite and other modules in it'ssite-packages`)

I'm sure the above can be streamlined, so I'd also definitely like these steps to be consolidated so that the install/build script for the .app will only do the OS-specific tasks.

I have to ask though how is the assessment.zip included in the other OS installers? Is it also as an .zip archive and later extracted on the installed machine? If so, then we can consolidate that into the pre-release steps but I don't know how setup.py can do that or for any other non-package inclusions? Perhaps the same data location if we also include a pre-migrated empty database?

[benjaoming]

@cpauya please don't build the docs, we can include them in the sdist

All these steps can be made redundant if you use the sdist:

# 4. Download KA-Lite zip based on develop branch.
# 5. Extract KA-Lite and move into `ka-lite` folder.
# 6. Make ka-lite repo production-ready (delete .KALITE_SOURCE_DIR, etc).
# 7. Run pyrun-2.7/bin/pip install -r ka-lite/requirements.txt
# 8. Run `bin/kalite manage compileymltojson`, needs `pyrun/pip install pyyaml==3.11

[jamalex]

@cpauya please don't build the docs, we can include them in the sdist

This is the point of the list we're building -- once we know what the necessary steps are, and make sure they're handled by a particular part of the cross-OS build process, we can trim back the work being done by each installer.

[aronasorman]

@benjaoming since 0.15 is nigh it would be good to have atleast the docs in there with the current way of building, but I agree that this is redundant once we're using the sdist. Perhaps we should make that a priority for 0.16.

[jamalex]

I have to ask though how is the assessment.zip included in the other OS installers? Is it also as an .zip archive and later extracted on the installed machine? If so, then we can consolidate that into the pre-release steps but I don't know how setup.py can do that or for any other non-package inclusions?

Not sure whether the Deb and Windows installers include a zip and then unpack it, or have it pre-unzipped and installed directly -- perhaps @benjaoming and @MCGallaspy can chime in on that.

However, ideally, we want to avoid having the assessment items get installed "twice" (once in a fixed location and once by being installed into the user data directory), since it's quite large. Perhaps @benjaoming can describe here how he handles this in the Deb? (which has two modes -- with and without the bundled assessment items)

[MCGallaspy]

In Windows, this is the case:

have it pre-unzipped and installed directly

[jamalex]

It's would have to be a data file that gets installed unconditionally, and then it would have to be the application (kalite manage setup) that makes the assessment of whether or not to bootstrap the user .kalite directory with an empty database. But this could happen several times within the same installation.

Yes, that sounds like the best approach. Could you create an issue for that?

(In principle, this could be an approach for assessment items as well, except that they're so big, so not actually a great plan)

One possible gotcha: Downgrading KA Lite. But I think this is a problem already, anyways. So no need to deal with it.

Since it would only be copying a DB in when no DB exists yet anyway, it shouldn't affect downgrading. But yeah, downgrading is problematic regardless.

[benjaoming]

@jamalex

Yes, that sounds like the best approach. Could you create an issue for that?

Done! learningequality/ka-lite#4395

[MCGallaspy]

Sent email to interested parties with build process tree, as we discussed.

[MCGallaspy]

Okay, based on this graphic this is my understanding of what we would like the build process to be.

Build process (what we do on our machines)

1. Get source from github.
2. Prepare some resources in repo for distribution
   1. compileymltojson
   2. node build.js
   3. Build documentation
   4. (Optional) unpack assessment items. Could result in included/non-included distributables.
3. Create source distributable bundle. python setup.py sdist --static From here, we can upload to PyPI so people can run from source, or continue in the build process.
4. Create platform specific installers. Essentially the only responsibility of the installers will be python setup.py install [[sdist bundle]]. On Windows, we will also, e.g. register an uninstall key, handle upgrade cases by copying user data, create icons, and do other stuff which is not directly handled by the application itself.
   1. (Optional) In the case of e.g. Raspberry Pi, where it's important to use certain settings, we the installer might include a settings module for distribution as well.

Installation process (what user does on his/her machine)

1. Get platform-specific installer, and run it. Will unpack a "basic" KA Lite source distribution as prepared above. This step unpacks the source to application data space (i.e. it's shared by the system). Subsequent steps unpack data in user data space, and so will have different results if run as different users.
   1. User feedback during installation might be used to create a custom settings module from a template. For most machines, this would probably be kalite.project.settings.base. For Raspberry Pi, it might be e.g. kalite.project.settings.rpi.
   2. At this point, user can run the app and download content, like videos or language packs.
2. (Optional) If we wish to include downloadable content with the distributable, we should create content packs. The user will configure their settings appropriately, then run an import command (to be developed) in order to unpack e.g. language packs and videos in the appropriate settings-defined locations. For Raspberry Pi, this could also include static files, since they live in user data space. The idea here is that eventually a user could also create an installation and export a content pack for quick deployment.

[benjaoming]

Yes, this is a good summary, can't really think of much more. Will use it as a recipe for the Makefile.

Minor detail: It has to be python setup.py sdist --static for creating the source dist, if it's supposed to include bundled dependencies.

[MCGallaspy]

Okay, before you get too deep into that, let's confirm everyone is on the same page.

[cpauya]

I'm good with this. 👍

[aronasorman]

Summary looks good, and what I had in mind.

[benjaoming]

All this is now also in a Gdrive document and should ideally just be maintained there.. and in the Bamboo scripts!

In the meantime, I don't see that the original summary by Mike has changed, it's still how we do things, although Bamboo has now taken over most tasks.

Closing this, go team, great work!!