-
Notifications
You must be signed in to change notification settings - Fork 713
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #4 from indirectlylit/channel-switching
sync with master
- Loading branch information
Showing
83 changed files
with
2,242 additions
and
393 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
Content | ||
====================================== | ||
|
||
This is a core module found in ``kolibri/Content``. | ||
|
||
.. toctree:: | ||
:maxdepth: 1 | ||
|
||
content/concepts_and_definitions | ||
content/implementation | ||
content/api_methods | ||
content/api_endpoints | ||
|
||
|
||
Models | ||
------ | ||
|
||
.. automodule:: kolibri.content.models | ||
:members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
API endpoints | ||
------------- | ||
|
||
request specific content: | ||
|
||
>>> localhost:8000/api/content/<channel_id>/contentnode/<content_id> | ||
|
||
search content: | ||
|
||
>>> localhost:8000/api/content/<channel_id>/contentnode/?search=<search words> | ||
|
||
request specific content with specified fields: | ||
|
||
>>> localhost:8000/api/content/<channel_id>/contentnode/<content_id>/?fields=pk,title,kind | ||
|
||
request paginated contents | ||
|
||
>>> localhost:8000/api/content/<channel_id>/contentnode/?page=6&page_size=10 | ||
|
||
request combines different usages | ||
|
||
>>> localhost:8000/api/content/<channel_id>/contentnode/?fields=pk,title,kind,instance_id,description,files&page=6&page_size=10&search=wh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
API Methods | ||
----------- | ||
|
||
.. automodule:: kolibri.content.api | ||
:members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
Concepts and Definitions | ||
======================== | ||
|
||
ContentNode | ||
----------- | ||
|
||
High level abstraction for prepresenting different content kinds, such as Topic, Video, Audio, Exercise, Document, and can be easily extended to support new content kinds. With multiple ContentNode objects, it supports grouping, arranging them in tree structure, and symmetric and asymmetric relationship between two ContentNode objects. | ||
|
||
File | ||
---- | ||
|
||
A Django model that is used to store details about the source file, such as what language it supports, how big is the size, which format the file is and where to find the source file. | ||
|
||
ContentDB Diagram | ||
----------------- | ||
.. image:: ../img/content_distributed_db.png | ||
.. Source: https://www.draw.io/#G0B5xDzmtBJIQlNlEybldiODJqUHM | ||
**PK = Primary Key | ||
**FK = Foreign Key | ||
**M2M = ManyToManyField | ||
ContentTag | ||
---------- | ||
|
||
This model is used to establish a filtering system for all ContentNode objects. | ||
|
||
|
||
ChannelMetadata | ||
--------------- | ||
|
||
A Django model in each content database that stores the database readable names, description and author for each channel. | ||
|
||
ChannelMetadataCache | ||
-------------------- | ||
This class stores the channel metadata cached/denormed into the default database. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,115 @@ | ||
Implementation Details and Workflows | ||
==================================== | ||
|
||
To achieve using separate databases for each channel and being able to switch channels dynamically, the following data structure and utility functions have been implemented. | ||
|
||
ContentDBRoutingMiddleware | ||
-------------------------- | ||
|
||
This middleware will be applied to every request, and will dynamically select a database based on the channel_id. | ||
If a channel ID was included in the URL, it will ensure the appropriate content DB is used for the duration of the request. (Note: `set_active_content_database` is thread-local, so this shouldn't interfere with other parallel requests.) | ||
|
||
For example, this is how the client side dynamically requests data from a specific channel: | ||
|
||
>>> localhost:8000/api/content/<channel_1_id>/contentnode | ||
|
||
this will respond with all the contentnode data stored in database <channel_1_id>.sqlite3 | ||
|
||
>>> localhost:8000/api/content/<channel_2_id>/contentnode | ||
|
||
this will respond with all the contentnode data stored in database <channel_2_id>.sqlite3 | ||
|
||
get_active_content_database | ||
--------------------------- | ||
|
||
A utility function to retrieve the temporary thread-local variable that `using_content_database` sets | ||
|
||
set_active_content_database | ||
--------------------------- | ||
|
||
A utility function to set the temporary thread-local variable | ||
|
||
using_content_database | ||
---------------------- | ||
|
||
A decorator and context manager to do queries on a specific content DB. | ||
|
||
Usage as a context manager: | ||
|
||
.. code-block:: python | ||
from models import ContentNode | ||
with using_content_database("nalanda"): | ||
objects = ContentNode.objects.all() | ||
return objects.count() | ||
Usage as a decorator: | ||
|
||
.. code-block:: python | ||
from models import ContentNode | ||
@using_content_database('nalanda') | ||
def delete_all_the_nalanda_content(): | ||
ContentNode.objects.all().delete() | ||
ContentDBRouter | ||
--------------- | ||
|
||
A router that decides what content database to read from based on a thread-local variable. | ||
|
||
ContentNode | ||
----------- | ||
|
||
``ContentNode`` is implemented as a Django model that inherits from two abstract classes, MPTTModel and ContentDatabaseModel. | ||
`django-mptt's MPTTModel <http://django-mptt.github.io/django-mptt/>`_, which | ||
allows for efficient traversal and querying of the ContentNode tree. | ||
``ContentDatabaseModel`` is used as a marker so that the content_db_router knows to query against the content database only if the model inherits from ContentDatabaseModel. | ||
|
||
The tree structure is established by the ``parent`` field that is a foreign key pointing to another ContentNode object. You can also create a symmetric relationship using the ``related`` field, or an asymmetric field using the ``is_prerequisite`` field. | ||
|
||
File | ||
---- | ||
|
||
The ``File`` model also inherits from ``ContentDatabaseModel``. | ||
|
||
To find where the source file is located, the class method ``get_url`` uses the ``checksum`` field and ``settings.CONTENT_STORAGE_DIR`` to calculate the file path. Every source file is named based on its MD5 hash value (this value is also stored in the ``checksum`` field) and stored in a namespaced folder under the directory specified in ``settings.CONTENT_STORAGE_DIR``. Because it's likely to have thousands of content files, and some filesystems cannot handle a flat folder with a large number of files very well, we create namespaced subfolders to improve the performance. So the eventual file path would look something like: | ||
|
||
``/home/user/.kolibri/content/storage/9/8/9808fa7c560b9801acccf0f6cf74c3ea.mp4`` | ||
|
||
As you can see, it is fine to store your content files outside of the kolibri project folder as long as you set the ``settings.CONTENT_STORAGE_DIR`` accordingly. | ||
|
||
The front-end will then use the ``extension`` field to decide which content player should be used. When the ``supplementary`` field's value is ``True``, that means this File object isn't necessary and can display the content without it. For example, we will mark caption (subtitle) file as supplementary. | ||
|
||
Content Constants | ||
----------------- | ||
|
||
A Python module that stores constants for the ``kind`` field in ContentNode model and the ``preset`` field and ``extension`` field in File model. | ||
|
||
.. automodule:: kolibri.content.constants.content_kinds | ||
.. automodule:: kolibri.content.constants.extensions | ||
.. automodule:: kolibri.content.constants.presets | ||
|
||
Workflows | ||
--------- | ||
|
||
There are two workflows we currently designed to handle content UI rendering and content playback rendering | ||
|
||
- Content UI Rendering | ||
|
||
1. Start with a ContentNode object. | ||
2. Get the associated File object that has the ``thumbnail`` field being True. | ||
3. Get the thumbnail image by calling this File's ``get_url`` method. | ||
4. Determine the template using the ``kind`` field of this ContentNode object. | ||
5. Renders the template with the thumbnail image. | ||
|
||
|
||
- Content Playback Rendering | ||
|
||
1. Start with a ContentNode object. | ||
2. Retrieve a queryset of associated File objects that are filtered by the preset. | ||
3. Use the ``thumbnail`` field as a filter on this queryset to get the File object and call this File object's ``get_url`` method to get the source file (the thumbnail image) | ||
4. Use the ``supplementary`` field as a filter on this queryset to get the "supplementary" File objects, such as caption (subtitle), and call these File objects' ``get_url`` method to get the source files. | ||
5. Use the ``supplementary`` field as a filter on this queryset to get the essential File object. Call its ``get_url`` method to get the source file and use its ``extension`` field to choose the content player. | ||
6. Play the content. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,6 +6,7 @@ Getting Started | |
|
||
user/index | ||
dev/index | ||
dev/content | ||
cli | ||
changelog | ||
contributing | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.