Refactor topic_tools to use app-based settings and remove unused ASSE…

docs/usermanual/userman_admin.rst

@@ -645,14 +645,23 @@ You can also run the ``kalite`` with a completely different Python settings modu
 Changing base settings
 ^^^^^^^^^^^^^^^^^^^^^^
 
-By default, ``~/.kalite/settings.py`` will load ``kalite.project.settings.base`` which are the basic settings. But you can also load Raspberry Pi settings by changing the file to read something like:
+By default, ``~/.kalite/settings.py`` will load ``kalite.project.settings.base`` which are the basic settings. But you can also load Raspberry Pi settings by changing the file to read something like::
 
-  .. ::
-    from kalite.project.settings.raspberry_pi import *
-    # Put your settings here, e.g.
-    # MY_SETTING_VAR = 123
+  from kalite.project.settings.raspberry_pi import *
+  # Put your settings here, e.g.
+  # MY_SETTING_VAR = 123
 
 
+.. NOTE::
+    When changing ``CONTENT_ROOT``, you should also change your ``DATABASES`` **if you
+    have downloaded your own assessment items** and you want to keep the
+    read-only assessment_items database (~50 MB) together
+    with your other media contents (for portability). Example::
+
+      from kalite.project.settings.base import *
+      CONTENT_ROOT = '/example'
+      DATABASES['assessment_items']['NAME'] = os.path.join(CONTENT_ROOT, 'assessmentitems.sqlite')
+
 
 Available settings
 ------------------
@@ -661,37 +670,45 @@ Available settings
 Most common settings
 ^^^^^^^^^^^^^^^^^^^^
 
-* DEBUG = <True or False> (default = False) -- Enables debug mode. In case you run into technical issues, enable this setting before troubleshooting / reporting.
-* CONTENT_ROOT = "<path to desired content folder>" (default=ka-lite/content)
-  This is the path that KA Lite will use to look for KA Lite video files to play.  Change the path to another local directory to get video files from that directory. NB! Directory has to be writable for the user running the server in order to download videos.
-* TIME_ZONE = <desired time zone>  (default = "America/Los_Angeles")
+* ``DEBUG = <True or False> (default = False)``
+  Enables debug mode. In case you run into technical issues, enable this setting before troubleshooting / reporting.
+* ``CONTENT_ROOT = "<path to desired content folder>" (default=~/.kalite/content)``
+  This is the path that KA Lite will use to look for KA Lite video files to play.
+  Change the path to another local directory to get video files from that directory.
+  NB! Directory has to be writable for the user running the server in order to download videos.
+* ``ASSESSMENT_ITEMS_ZIP_URL = "scheme://path/to/assessmentitems.zip"``
+  ``(default=https://learningequality.org/downloads/ka-lite/0.14/content/assessment.zip)``
+  This is useful if you need an auto-deployment to fetch assessment items (exercises) from a local source. You can use
+  ``kalite manage setup --dl-assessment-items`` to automatically download assessment items.
+* ``TIME_ZONE = <desired time zone>  (default = "America/Los_Angeles")``
   You can set this to be the local time zone for your installation. Choices can be found here.
-* LANGUAGE_CODE = "<desired ISO 639-1 Language Code>" (default = "en-us")
+* ``LANGUAGE_CODE = "<desired ISO 639-1 Language Code>" (default = "en-us")``
   You can set this to the desired language code for this installation (All choices can be found here).  If there are translations available, our web server will show them in KA Lite. Soon, we hope to provide support for internationalized content inside the KA Lite interface.
-* USE_I18N = <True or False> (default = True)
+* ``USE_I18N = <True or False> (default = True)``
   If you set this to False, our web server will make some optimizations so as to avoid loading internationalization tools. Things might run a little faster, but you won't have support for translated content.
-* USE_L10N = <True or False> (default = False)
+* ``USE_L10N = <True or False> (default = False)``
   By default, this is set to False. If you set this to True, Django will format dates, numbers and calendars according to the current locale. For example, January 5, 2000 would be 1/5/2000 if locale = "en-us" and 5/1/2000 if locale = "en-gb"
 
 
 User restrictions
 ^^^^^^^^^^^^^^^^^
 
-* LOCKDOWN = <True or False> (default = False)
+* ``LOCKDOWN = <True or False> (default = False)``
   With this setting, users must be logged in order to access videos & exercises
-* DISABLE_SELF_ADMIN = <True or False> (default = False). Disables user sign ups.
+* ``DISABLE_SELF_ADMIN = <True or False> (default = False)``
+  Disables user sign ups.
 
 
 Online Synchronization
 ^^^^^^^^^^^^^^^^^^^^^^
 
-* USER_LOG_MAX_RECORDS = <desired maxium for user log records> (default = 0)
+* ``USER_LOG_MAX_RECORDS = <desired maxium for user log records> (default = 0)``
   When this is set to any non-zero number, we will record (and sync for online tracking) user login activity, summarized for every month (which is configurable, see below).  Default is set to 0, for efficiency purposes--but if you want to record this, setting to 1 is enough!  The # of records kept are not "summary" records, but raw records of every login.  These "raw" data are not synced, but are kept on your local machine only--there's too many of them.  Currently, we have no specific report to view these data (though we may have for v0.10.1)
-* USER_LOG_SUMMARY_FREQUENCY = <desired frequency in the following format (number, amount of time)> (default = (1, "months")
+* ``USER_LOG_SUMMARY_FREQUENCY = <desired frequency in the following format (number, amount of time)> (default = (1, "months")``
   This determines the granularity of how we summarize and store user log data.  One database row is kept for each student, on each KA Lite installation, for the defined time period.  Acceptable values are:
   (1, "months"), (2, "months"), (3, "months"), (6, "months") - separate logged data for every month, 2 months, 3 months, or 6 months, respectively
   (1, "weeks") - separate logged data for every week ** NOTE THIS MAY PRODUCE A LOT OF DATA **
-* SYNC_SESSIONS_MAX_RECORDS = <desired max records of sync sessions> (default = 10)
+* ``SYNC_SESSIONS_MAX_RECORDS = <desired max records of sync sessions> (default = 10)``
   Every time your installation syncs data, we record the time of the sync, the # of successful logs that were uploaded and downloaded, and any failures.
   This setting is how many such records we keep on your local server, for display.
   When you log in to our online server, you will see a *full* history of these records.
@@ -700,22 +717,22 @@ Online Synchronization
 Optimization of storage and system load
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-* CRONSERVER_FREQUENCY = <desired frequency of cronserver to run in seconds> (default = 10 minutes)
+* ``CRONSERVER_FREQUENCY = <desired frequency of cronserver to run in seconds> (default = 10 minutes)``
   This is how frequently KA Lite tries to synchronize user data with other Devices on your Zone.  This can be changed to sync data more often (use a smaller #), or if you're never online (can be set to a large number)
-* CACHE_TIME = <desired length of cache time, in seconds> (default = 5*365*24*60*60) (that's 5 years!)
+* ``CACHE_TIME = <desired length of cache time, in seconds> (default = 5*365*24*60*60) (that's 5 years!)``
   Our basic topic pages, video pages, and exercise pages rarely change--only when you download new videos (changes made by user logins are made in a different way).  Therefore, we can "cache" copies of these pages, to avoid constantly regenerating them, and speed up KA Lite.  We have logic to delete the cached copies, and therefore generate new copies, if you download new videos or delete old videos through our interface.
   If you would like to disable caching, set CACHE_TIME = 0 .
   Read a little more about caching on Wikipedia.
-* CACHE_LOCATION = '<path to cache directory>' (default= dir named kalite_web_cache in the OS temporary dir)
+* ``CACHE_LOCATION = '<path to cache directory>' (default= dir named kalite_web_cache in the OS temporary dir)``
   Some operating systems will clear the temporary directories when the system is rebooted.  To retain the cache between reboots, an alternative location can be specified.  (for example on Linix, "/var/tmp/kalite_web_cache")
-* CHERRYPY_THREAD_COUNT = <number of threads> (default=50)
+* ``CHERRYPY_THREAD_COUNT = <number of threads> (default=50)``
   The CherryPy Webserver can handle multiple page requests simultaneously.  The default is 50, but for slow or single CPU servers, performance will be improved if the number of threads is reduced.  Minimum number of threads is 10, optimum setting for Raspberry Pi is 18.
 
 
 Raspberry Pi
 ^^^^^^^^^^^^
 
-* USE_MPLAYER = <True or False> (default = False)
+* ``USE_MPLAYER = <True or False> (default = False)``
   With this setting, if the browser is run from the same computer as the KA Lite server, then instead of playing the video in the browser, the video will be launched outside of the browser and played in mplayer - a light-weight video player that is included with the KA Lite software.
   This is intended for use only on the Raspberry Pi, where no other video player is available.
 

kalite/contentload/management/commands/generate_assessment_zips.py

@@ -8,17 +8,18 @@
 from multiprocessing.dummy import Pool as ThreadPool
 from threading import Lock
 
-from django.conf import settings
+from django.conf import settings as django_settings
 from django.core.management import call_command
 from django.core.management.base import NoArgsCommand
 
 import kalite.version as version
 
 from kalite.topic_tools import get_content_cache, get_exercise_cache
+from kalite.contentload import settings
 
-logging = settings.LOG
+logging = django_settings.LOG
 
-ZIP_FILE_PATH = os.path.join(settings.USER_DATA_ROOT, "assessment.zip")
+ZIP_FILE_PATH = os.path.join(django_settings.USER_DATA_ROOT, "assessment.zip")
 
 IMAGE_URL_REGEX = re.compile('https?://[\w\.\-\/]+\/(?P<filename>[\w\.\-\%]+\.(png|gif|jpg|jpeg|svg))', flags=re.IGNORECASE)
 

kalite/contentload/management/commands/init_assessment_items.py

@@ -1,4 +1,3 @@
-import copy
 import os
 
 from django.core.management import call_command
@@ -7,11 +6,12 @@
 from fle_utils.general import softload_json
 from optparse import make_option
 
-from kalite import settings; logging = settings.LOG
+from django.conf import settings as django_settings
+logging = django_settings.LOG
+
 from kalite.topic_tools.models import AssessmentItem
 
-# From topic_tools/__init__.py
-ASSESSMENT_ITEMS_FILEPATH = os.path.join(settings.CHANNEL_DATA_PATH, "assessmentitems.json")
+from kalite.contentload import settings
 
 
 class Command(BaseCommand):
@@ -20,7 +20,7 @@ class Command(BaseCommand):
         make_option("-a", "--assessment-items-filepath",
                     action="store",
                     dest="assessment_items_filepath",
-                    default=ASSESSMENT_ITEMS_FILEPATH,
+                    default=settings.KHAN_ASSESSMENT_ITEM_JSON_PATH,
                     help="Override the JSON data source to import assessment items from"),
         make_option("-d", "--database-path",
                     action="store",

kalite/contentload/management/commands/unpack_assessment_zip.py

@@ -8,14 +8,14 @@
 from fle_utils.general import ensure_dir
 from optparse import make_option
 
-from django.conf import settings
+from django.conf import settings as django_settings
 from django.core.management.base import BaseCommand, CommandError
 
-logging = settings.LOG
+logging = django_settings.LOG
 
 from kalite import version
+from kalite.contentload import settings
 
-logging = settings.LOG
 
 class Command(BaseCommand):
 
@@ -79,9 +79,24 @@ def should_upgrade_assessment_items():
 
 
 def unpack_zipfile_to_khan_content(zf):
-    folder = settings.KHAN_CONTENT_PATH
+    folder = settings.KHAN_ASSESSMENT_ITEM_ROOT
     ensure_dir(folder)
     zf.extractall(folder)
+    # Ensure that special files are in their configured locations
+    os.rename(
+        os.path.join(settings.KHAN_ASSESSMENT_ITEM_ROOT, 'assessmentitems.version'),
+        settings.KHAN_ASSESSMENT_ITEM_VERSION_PATH
+    )
+    os.rename(
+        os.path.join(settings.KHAN_ASSESSMENT_ITEM_ROOT, 'assessmentitems.sqlite'),
+        settings.KHAN_ASSESSMENT_ITEM_DATABASE_PATH
+    )
+    # JSON file is apparrently not required (not in the test at least)
+    if os.path.isfile(os.path.join(settings.KHAN_ASSESSMENT_ITEM_ROOT, 'assessmentitems.json')):
+        os.rename(
+            os.path.join(settings.KHAN_ASSESSMENT_ITEM_ROOT, 'assessmentitems.json'),
+            settings.KHAN_ASSESSMENT_ITEM_JSON_PATH
+        )
 
 
 def is_valid_url(url):

kalite/contentload/settings.py

@@ -0,0 +1,45 @@
+"""
+
+New settings pattern
+
+See:
+https://github.com/learningequality/ka-lite/issues/4054
+https://github.com/learningequality/ka-lite/issues/3757
+
+All settings for the topic_tools app should be defined here, they can
+only on django.conf.settings
+"""
+import os
+from django.conf import settings
+
+from kalite import ROOT_DATA_PATH
+
+# OUTDATED - Special setting for Khan Academy content, was only used for
+# assessment item unpacking. All assessment items have been moved from this
+# location in distributed.management.commands.setup
+OLD_ASSESSMENT_ITEMS_LOCATION = os.path.join(settings.CONTENT_ROOT, "khan")
+
+
+# Are assessment items distributed in the system-wide data directory?
+if settings.ASSESSMENT_ITEMS_SYSTEM_WIDE:
+    ASSESSMENT_ITEM_ROOT = os.path.join(ROOT_DATA_PATH, 'assessment')
+    KHAN_ASSESSMENT_ITEM_ROOT = os.path.join(ASSESSMENT_ITEM_ROOT, 'khan')
+else:
+    # Where assessment items are stored in general (but no additional channels
+    # exist yet so it's all hard-coded for KHAN_ASSESSMENT_ITEM_ROOT below)
+    ASSESSMENT_ITEM_ROOT = os.path.join(settings.CONTENT_ROOT, 'assessment')
+
+    if not os.path.exists(ASSESSMENT_ITEM_ROOT):
+        os.mkdir(ASSESSMENT_ITEM_ROOT)
+
+    KHAN_ASSESSMENT_ITEM_ROOT = os.path.join(ASSESSMENT_ITEM_ROOT, 'khan')
+    if not os.path.exists(KHAN_ASSESSMENT_ITEM_ROOT):
+        os.mkdir(KHAN_ASSESSMENT_ITEM_ROOT)
+
+# This one should always the settings because it is part of settings.DATABASES
+KHAN_ASSESSMENT_ITEM_DATABASE_PATH = settings.DATABASES['assessment_items']['NAME']
+
+# Default locations of specific elements from the assessment items bundle.
+# Files will be forced into this location when running unpack_assessment_zip
+KHAN_ASSESSMENT_ITEM_VERSION_PATH = os.path.join(KHAN_ASSESSMENT_ITEM_ROOT, 'assessmentitems.version')
+KHAN_ASSESSMENT_ITEM_JSON_PATH = os.path.join(KHAN_ASSESSMENT_ITEM_ROOT, 'assessmentitems.json')

kalite/contentload/tests/import_tests.py

@@ -12,6 +12,7 @@
 from kalite.contentload.management.commands.channels import import_channel, base
 
 from fle_utils.general import ensure_dir
+import unittest
 
 
 class TestContentImportTopicTree(MainTestCase):
@@ -95,6 +96,7 @@ def recursively_find_nodes(self, node, key, value, cache):
             for child in node["children"]:
                 self.recursively_find_nodes(child, key, value, cache)
 
+    @unittest.skip("Skipping until kaa package is replaced")
     def test_create_topic_tree(self):
         slug = os.path.basename(self.tempdir)
         topic_tree = {