Skip to content

docker newbie1.5 instructions first build of webwork

Jump to bottom Edit New page
Nathan Wallach edited this page Jun 10, 2022 · 3 revisions

Download and install a WeBWorK container:

Github and git activity

  • The two Git config settings included below -c core.autocrlf=false -c core.symlinks=true should really only be needed on Windows (as these are the default values on Linux and MacOS X), however forcing them should not interfere with anything on Linux and MacOS X.
  • It is not necessary to install the pg repository below. You can skip that git clone instruction unless you plan to make changes to the core pg code, in which case you will also need to edit webwork2/docker-compose.yml to include your local pg tree at run-time. ( See https://github.com/openwebwork/webwor1k2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises) -- do this on the second time you build WW :-) )
cd webwork_docker
ls  # it should be empty -- running the demo above left no trace in this directory

git clone https://github.com/openwebwork/webwork2 -c core.autocrlf=false -c core.symlinks=true 
  # grab the webwork2 repo

#optional -- not needed until you start modifying pg
# git clone https://github.com/openwebwork/pg -c core.autocrlf=false -c core.symlinks=true 
# grab and install the pg repo

cd webwork2

git branch 
 * main               
# should show  "* main"  highlighted.

# Change to the branch needed.
#   If you are doing advanced work you may want to switch to the develop branch or some other special branch.
#   If you intend to build a docker image using a special branch of WW/PG you will need to modify `docker-compose.yml` to use it.
  • Depending on how fast and how accurately you type these git commands will take 5 minutes or less.
    • Typically a docker image will be able to download the OPL metadata from GitHub and avoid needed to locally run OPL-update.
    • If there is a need to locally create the OPL database tables, the process takes quite some time (20+ minutes).

Preparations before you build the Docker image

Before proceeding, shut everything down in Docker and remove unwanted old images. docker ps -a should be clear.

We are still in the webwork2 directory:

ls

If everything is successful the ls listing will show the usual webwork files but also the files
Dockerfile, DockerfileStage1, DockerfileStage2, docker-compose.yml. Additional files are in the docker-config subdirectory includingdocker-entrypoint.sh.

Before building the images and starting up the server for the first time, you probably should replace the default values of the database passwords. These passwords are easily changed only when the database is first started, by the first run of docker-compose up below.

  • The "regular" (webworkWrite) DB password used by WeBWorK to communicate with the database is set in the hidden .env file on the line which begins WEBWORK_DB_PASSWORD=. The password for webworkWrite starts immediately after the = sign.
  • Also in .env edit the WEBWORK_MYSQL_ROOT_PASSWORD= line to set an SQL root password for the mariaDB database.

These notes are for those planning to build WeBWorK-2.16 (or WeBWorK-2.17) to do local work. Those interested in creating a production server should look at: https://github.com/openwebwork/webwork2/wiki/Running-a-production-WeBWorK-server-using-Docker.

  • sanity checks:

    • git branch should return main (or the branch you selected to use)
    • git status should show that the file webwork2/.env has been modified
    • git diff .env should show only the passwords you changed

Preparations for using a more recent version of WeBWorK:

  • If you want to build a Docker image for WeBWorK 2.17 before its formal release, you would need to:
    1. git checkout WeBWorK-2.17 (in the webwork2 directory)
    2. Edit docker-compose.yml to set WEBWORK2_BRANCH=WeBWorK-2.17 and PG_BRANCH=PG-2.17 instead of using the main for these.
    3. Use --tag webwork-base:forWW217 below instead of --tag webwork-base:forWW216

Similar changes can be used to make use of the develop branch or any other custom branch which has the Docker files suitable set up. The correct tag to use appears in the second FROM line of DockerfileStage2 (the one which is in the webwork2 directory you currently are in and which should be checked out the the relevant branch).

Building the Docker image (current release, WeBWorK 2.16 at the time this was written)

Stage 1 (OS layer):

Now while still in the webwork2 directory type

docker build --tag webwork-base:forWW216 -f DockerfileStage1 .

Note: For WeBWorK 2.17 the tag above should be webwork-base:forWW217. For a different branch check what value appears in the second FROM line in DockerfileStage2.

The docker build command takes quite some time. The terminal window will scroll through lots of lines to let you know that something is running. You can see that it is installing a lot of things for the ubuntu version of the Linux operating system. Eventually it installs several additional CPAN modules.

After building the underlying operating system layer, we build an additional later with WeBWorK in stage 2.

Stage 2 (webwork):

Run:

docker-compose build

This phase builds the WeBWorK code on top of the operating system. It should not take long.

The two stage process makes it faster and easier to update the "webwork" part of the Docker image.

Once the build process is complete you should see a line like

Successfully tagged webwork:latest

Starting up the WeBWorK container

docker-compose up

This ties everything together and again will take some time, 5 to 10 minutes as the installation lines fly by. Toward the end there are lines about creating the mysql (database) application db_1 and finally the webwork application app_1. Then the admin course is created and then it sets up the OPL and related databases. Setting up the OPL also takes some time, be patient.

Towards the end you should see

app_1  | WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
app_1  | WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc 
app_1  | WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
app_1  | webwork.apache2-config:  WeBWorK server is starting
app_1  | webwork.apache2-config:  WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
app_1  | webwork.apache2-config:  The following locations and urls are set in webwork2/conf/site.conf
app_1  | webwork.apache2-config:  PG root directory set to /opt/webwork/pg
app_1  | webwork.apache2-config:  WeBWorK server userID is www-data
app_1  | webwork.apache2-config:  WeBWorK server groupID is www-data
app_1  | webwork.apache2-config:  The webwork url on this site is http://localhost/webwork2
app_1  | webwork.apache2-config:  The webwork smtp server address is localhost
app_1  | webwork.apache2-config:     The webwork smtp server port is 
app_1  | webwork.apache2-config:     The webwork smtp server protocol is 'not ssl'

IMPORTANT: Don't close the terminal window since that will stop the container running the website.

On my iMac the whole process took about 35 minutes from typing docker-compose build . ; docker-compose up; to the final result

DO NOT CLOSE THIS WINDOW!!!! The window must be kept open while running the webserver. If you close the window it will shut down the webwork server.

Next time you will probably prefer to run

docker-compose up -d   # detaches the terminal so you don't see all the output
                     # and so that you can quit the terminal without stopping the container.

Testing out the locally running WeBWorK container

Once the lines have stopped scrolling you can type http://localhost:8080/webwork2 into your browser and you should have a webwork site running.

The only course is "course administration" and it has login "admin" with password "admin".

You can login and add yourself as another admin in the admin course. (First add yourself as a student, then promote yourself to admin.)

You can check the library browser to make sure that it works and that the OPL database has been updated properly.

Create a new course 'test_course', add some guest "students" by importing from demoCourse.lst and add a few homework problems to the course using the LibraryBrowser.

What could possibly go wrong?

  • I forgot to select a target set from the course in the library browser. Usually the blank for the target set would read "Select a Set from this Course" but because no homework sets had been set up at all this field was set to "no sets in this course yet". This lead to an "instructorXMLRPC: 403 Forbidden" error.

    • Usually "instructorXMLRPC: 403 Forbidden" error is triggered by some glitch in setting up the site URL in the site.conf file so that it works properly with AJAX, but not in this case. Making sure that I had created a new set in the course and then choosing that set as a target was enough to clear the error.

    • I found a message in the error log at /var/log/apache2/error.log indicating that no homework set had been created for the course (see more information below on reading this log) but that message was not transmitted to the screen. It's likely that this particular error can only occur with a brand new, empty course.

  • On some builds I got an "app_1 exited with error 1" at the end of the "docker-compose build ." command or the "docker-compose up". This is obviously not good. In some cases I had to go back to the beginning and start the reinstall from the beginning. This involved removing all prebuilt docker packages so docker container ps -a and docker images should all be empty. If they are not use docker container rm <id> and docker rmi <id> to remove them. You only need to type the first couple of numbers/letters of the SHA identifier -- not the whole thing.

    • The docker set up is still fragile. We hope to break it into pieces so that webwork2, pg, mysql and OPL are all handled separately. This way if one download fails you don't have to start the entire process over. In particular you can update changes to webwork2 and pg without necessarily rebuilding the OPL database. With WeBWorK-2.16 (which is also now the 'main' download) the docker build process has been split into pieces so that the operating system layer (Stage 1) doesn't need to be rebuilt each time you make changes.

Building updated Docker images:

  • If you want to force a rebuild of the base OS image (the one which installs lots of Ubuntu packages, etc.) to get updates to the base OS you would:
    1. cd into webwork2 directory (as usual)
    2. Run: docker pull ubuntu:20.04 (gets an update version of the Ubuntu base image)
    3. Run: docker pull alpine/git
    4. Run: docker build --no-cache --tag webwork-base:forWW216 -f DockerfileStage1 .
    5. (If WW 2.17 is in use the tag should be webwork-base:forWW217 in the prior line).
  • If you want to rebuild the webwork layer of the image (to get updated webwork code from GitHub) you would:
    1. cd into webwork2 directory (as usual)
    2. Run: docker-compose build --no-cache
  • Using the --no-cache setting to forces Docker not to use cached layers from the prior builds, so it will rebuild everything using the latest versions.
  • You can get an updated version of MariaDB 10.4 using *. docker pull mariadb:10.4

Next step

In a new terminal window go to webwork_docker/webwork2. Don't close the first terminal window.

Where is the data being stored? You won't be able to see much of the data that you usually expect to see stored in webwork on the disk, at least not from "outside" the docker environment. One thing you can see is the courses directory.

A new directory has been created to hold persistent webwork2 data: (We assume you are still in your webwork_docker/webwork2)

cd ..
ls ww-docker-data
   courses
ls ww-docker-data/courses
  admin/			adminClasslist.lst	defaultClasslist.lst	modelCourse/		test_course/

You'll see courses, which is where persistent data for courses run in the Docker container will be saved.

IMPORTANT: You can add (upload) archived courses to the courses directory and then unarchive them using the admin course in the browser. This is the quickest way to populate your docker webwork with courses you have been using on your production site. Even if you don't have command line access to your production site you can archive and download your own courses using the commands on the FileManager page of your course.

With older versions of the Docker config files, there will also be a db directory which is the location for the persistent storage for the database. More recent versions of the Docker config files put the databases into a named Docker volume. It's wise not to touch the db directory unless you are a MariaDB expert.

Read more about the persistent and transient storage at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises). It's one of the things that takes a while to get used to. The exact balance of which data is persistent and which is transient will continue to be tweaked as we work on creating the best experience of using docker with WeBWorK.

In your second terminal window (in directory webwork_docker/webwork2) -- you may have to cd to webwork_docker/webwork2 -- type

docker-compose down
   Stopping webwork2_app_1 ... done
   Stopping webwork2_db_1  ... done
   Stopping webwork2_r_1   ... done
   Removing webwork2_app_1 ... done
   Removing webwork2_db_1  ... done
  Removing webwork2_r_1   ... done

If you go back to your first window you will see that the build process has exited and your browser can no longer access the webwork site.

congratulations you've completed your first use of webwork on your laptop using docker.

Next try -- things will be quicker now:

In the first window type

docker-compose up -d
    Creating network webwork2_default with the default driver
    Creating webwork2_db_1 ... done
    Creating webwork2_r_1  ... done
    Creating webwork2_app_1 ... done

Notice that things are much quicker this time as the images are already built. And the webserver has started up right where you left off.

Close the first terminal window. And check that the website is still running. (This is what the -d accomplished.)

In the second terminal (webwork_docker/webwork2 directory) type

docker container ls  
# (or docker ps will also work)

Observe the container id for the webwork container. Some commands will require at least the first 4 or 5 characters of that WeBWorK_id to identify the container. Sometimes the name webwork2_app_1 alone will work. You will need this in the command below.

type

docker container exec -it WeBWorK_id bash
     root@WeBWorK_id:/opt/webwork#

A terminal window has just been opened to the inside of the "remote" machine at the directory /opt/webwork/webwork2.

This command is worth creating an alias for. In my .bashrc account I have added

alias dockerbash="docker container exec -it webwork2_app_1 bash"

so that I don't have to remember the whole string. This command lets you "login via a terminal window" to the webwork2 site running in the docker container. What you see should be similar to what you see when you login via a command line to your production server or a development server. One difference is that you also have an "outside" view via the desktop of the permanent files. Changes made on the outside, e.g. copying a course from your desktop into the ww-docker-data/courses folder can be seen (after a brief pause) via the terminal at /opt/webwork/courses

Anything that you would do by logging in to your local WeBWorK server on the command line you can do by "logging" in to the container using the approach above.

Warning: Recall that changes made in the command line of the Docker container will only be persistent if they are making changes to files on the persistent storage. See https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises). Courses are on permanent storage, changes to macro files in pg or in the OPL are not. Changes to a macro file in a course template/macros file will be preserved.

When you are done exploring type exit and you will close the terminal window "connection" to the container (the remote machine)

Installing an archived course:

If you have a copy of an archived course that you've downloaded from your usual webwork site (a .gz file) we can place that in the container as follows.

  1. In webwork_docker/webwork2
docker-compose down
cp ../path/to/course/archiveFile.tgz ww-docker-data/courses   
docker-compose up
  1. Check the web browser and the admin course. You should be able to "unarchive" your course using the "unarchive" tab.
  2. If it is an old course you may have to "update courses" to fix the database on the course you have imported.
  3. In the web browser open the library browser and make sure that it works.
  4. You can now add more problems to that course. Changes will be preserved.

Misc:

The text below is not up to date as the current Docker setup does not keep git information as part of the pg tree, so you cannot run the git commands listed. If you want to use a git tracker version of pg mount it from outside the image to the container by modifying the relevant lines of docker-compose.yml and restart the Docker container.

Double check that you are running the versions of webwork and pg that you want. At the bash prompt (which you opened inside the Docker container) type:

cd pg
git fetch origin   # will refresh you copy of the index 
                   # of the branches available at origin (aka github.com/openwebwork)
git branch -a          
git checkout -b develop origin/develop  # if you want to test the develop branch
apachectl graceful # restarts the web server.

You do not need to rebuild the entire docker container to change from one branch of of webwork to another. (whew!)

Adding a new course

  • Now try adding a new course from the browser using the Course Administration web page.
  • You should see that you can build it using modelCourse as a template.

Restarting just the WeBWorK container and keeping the database container running:

  • docker-compose restart app

Shutting down (and restarting) the Docker container:

  • When you are done using your Docker container type docker-compose down and docker will stop using CPU's on your computer.
  • It will still take up some disk memory space.
  • It can be restarted with docker-compose up -d .

Final comments:

  • see Docker for newbies4 for adding daemon_course (still under development)

That's enough for now. Play with the WeBWorK site you have created and go back to skimming Docker's getting started docs to try to figure out what you have just done. We've only used the first two sections of that document -- there is more that you can do to create a "production site" that can handle many students.

Warning With what we have so far you DO NOT want to use this as a production WeBWorK site -- this is a local installation on your machine for your use only. Most likely the databases are not using secure passwords for example, so there would not be much security if you start letting people do homework on a network connected to your computer.

You can read some suggestions for using WeBWorK inside Docker on a production server at https://github.com/openwebwork/webwork2/wiki/Running-a-production-WeBWorK-server-using-Docker

To repeat, the purpose of using Docker (so far) is to allow nearly everyone who has worked with WeBWorK using the command line to download and test new patches or pull requests to WeBWorK. Faster testing will allow us to proceed more quickly with new WeBWorK features. You can for example create homework questions that use the latest features in the develop branch without endangering the behavior of your school's production WeBWorK site.

The UBC people are eventually planning to use docker on their production machines so that it is super simple to install WeBWorK.

Also watch out, this installation is temporary. If you create a WeBWorK course in your local site it will stay around if you do docker-compose down, which just makes the container dormant. It will also stay around if you quit the Docker app. If you remove the container however (a command with a "rm" or "rmi" in it) the site will disappear. You could first save the course using the "archive" button in the FileManager in WeBWorK and download the archived file. (You could also do this for multiple courses from Course Administration course.)

Cleaning up

All the persistent files (database and course files) are stored in webwork_docker/ww-docker-data directory. In order to do a completely clean uninstall of this "webwork in a box", remove the ww-docker-data directory or webwork_docker directory (and the named mysql storage volume).

Add a custom footer
Add a custom sidebar
Clone this wiki locally