Dataset Management

Excerpted from original GitLab merge 46 and merge 58 with additional notes

See also #84, #135, #136

Overview

Dataset management is a little complicated for historical reasons as we had different settings that we needed accomodate:

• In early development, there was no UI for creating and editing the different database tables being used, so we would dynamically populate a Loki database from *.db.json files.
• Later in development, we needed a way to reliably spin up specific database states for testing, so with each npm run dev, the database was re-initialized.
• For production and piloting with students we needed a database that would save student values and persist between runs. Yet we also needed a fallback to create a new DB if one didn't already exist, so for production scripts, if meme.loki did not exist, the app would automatically generate one from the *.db.json files in /templates/meme/ or if not found, /templates/_blank/.

These different approaches are reflected in the different startup methods (for clarity, we are assuming / is the root repo directory, e.g. /templates/ refers to /<repodir>/templates/):

for	command	built from	database file
Development	npm run dev	templates/_blank/*.db.json (NOTE: no changes are saved)	runtime/test.loki*
Production	npm start	meme.loki or if not exist, system/datasets/meme/*.db.json	runtime/meme.loki*
Electron Debug from Command Line -- does not rebuild	npm run electron	meme.loki or if not exist, system/datasets/meme/*.db.json	runtime/meme.loki*
Production from Command Line	npm run app	meme.loki or if not exist, system/datasets/meme/*.db.json	runtime/meme.loki*
Production from Electron App	Double-click meme.app	meme.loki or if not exist, system/datasets/meme/*.db.json	runtime/meme.loki*

• NOTE historically the runtime directory was in repo_directory/runtime. With the Electron code signing release (as of April 2023), the runtime directory for the built Electron app is in ~/Documents/MEME/db.

Running MEME

For development, we often want to reinitialize the database for each run so we can reliably reproduce errors. The following commands are run from the Terminal at the root of the repository:

1. npm run dev - uses the system/datasets/test dataset, reinitializing the runtime/test.loki database file on each run.
2. npm run electron - uses the system/datasets/meme dataset, initializing runtime/meme.loki ONCE only if meme.loki is not present. After that, the meme.loki file is retained.

For student trials, we want to retain data between runs.

3. npm run app - uses the virtual filesystem inside the meme.app bundle, located in meme.app/Contents/Resources/runtime/meme.loki. As with (2), the .loki file is created if one doesn't exist, using the dataset from meme.app/Contents/Resources/app/system/datasets/meme. The meme "app bundle" is created in `repo_directory/dist/meme-darwin-x64/ when you use npm run package.
4. "double-click electron app" - Same as (3), but without debug console information. You can find the directory of the app by opening a Terminal window and dragging the app icon onto it.

Locating Data

The INITIAL DATASETS are located in different places depending on which run mode you are using:

• If you are using run dev or run electron, the data directories are in <repodir>/templates/_blank.
• If you are running the packaged app through run app or double-clicking, the data directory is located in <repodir>/dist/MEME macOS/templates/_blank.

The RUNTIME DATABASE and LOG FILES are located as follows:

• If you are using run dev or run electron, the runtime directory is in <repodir>/data/. You'll find the appropriate .loki files here (test.loki or meme.loki) and logs folder here.
• If you are running the packaged app, the runtime directory is in <repodir>/dist/MEME macOS/data

Backing-up Data

You can make copies of the data directory and data/db/meme.loki file to back-up a given dataset if you are using the npm run electron. If you're using npm run app or double-clicking the meme.app icon method of running MEME, you can make copies of the <repodir>/dist/MEME macOS/data folder, or if you've moved the distribution files to another computer or location, just copy the data folder in the MEME macOS folder.

---

Appendices

Which Datasets are Active When?

For run dev, the test dataset is used from system/datasets/test. For all other current run modes, the meme dataset is used.

Overriding the Default Dataset

It is now possible to reset the database using different named datasets.

The default datasets are:

• system/datasets/test used in the dev run mode to reset the database to a known state every time the server is run.
• system/datasets/meme is used in all other cases to create the database if the runtime/meme.loki doesn't exist. After that, the database is loaded as-is on every server run.

For typical use, to enable the persistent database you will use ANY RUNMODE except for 'npm run dev' which always erases the database with the contents of the test dataset.

• If you prefer the convenience of runtime files in the repo, use npm run electron
• If you are using the packaged MEME app, then you'll either use npm run app or double-click the meme.app application icon in Finder. To extract the data, you have to dig into the virtual filesystem (see

NOTE You do NOT need to use npm run package to use npm run electron. This does not run the packaged app, but instead uses a precompiled generic electron binary for debugging.

The datasets are used ONLY for initialization of the LokiJS database files located in the runtime/ directory. They are named based by dataset (e.g. 'runtime/test.loki' is initialized with the contents from 'system/datasets/test).

Additionally, you can override the dataset by specifying the DATASET environment variable on the command line. For example, to load the 'meme' dataset in developer mode you would type the following on the command line.

DATASET='meme' npm run dev

When you clean the system using npm run clean, this removes the contents of the runtime directory.

• In developer mode: The database will be loaded with the contents in the test dataset directory, as it normally does, every time the server is started.
• In electron debug mode: After npm run clean, MEME will detect the missing database collections and initialize them from the meme dataset directory one time. After that, the database will retain information normally in the runtime/meme.loki file
• You can make duplicates of the dataset directories using the Finder if you want to have several versions for testing.
• You can also rename the meme dataset directory to something else so you can swap-in another one.

IMPORTANT!

The runtime directory stores the live database and logs, while the datasets directory is only used to pre-populate the database when needed (testing).

Don't run npm run clean if you have data you want to keep! It will ERASE it all with no way to get it back!

Technically how does this work?

The server-database.js module initializes the appropriate LokiJS file for the run mode. This file is autoupdated 3 seconds after every database change, and is stored in the appropriate runtime directory.

The files in a typical dataset directory have the extension .db.js and export their content as a CommonJS module using module.exports. Currently each file corresponds to one of the ADMDATA or PMCDATA properties as follows:

dataset/
  meme/
    classroomResources.db.js
    classrooms.db.js
    criteria.db.js
    groups.db.js
    models.db.js
    ratingsDefinitions.db.js
    resources.db.js
    sentenceStarters.db.js
    teachers.db.js

After the database instance is loaded by LokiJS, we determine whether to (1) reset the database from the corresponding dataset OR (2) use the database as-is.

---

Preparing an Electron Build

Overview

IMPORTANT: Read Reference Build Differences first to understand the different ways of building the app!!!

General Goals

• Start meme.app for the first time should start with a blank template if nothing has been defined.
• Researchers and teachers should be able to start meme.app with a default set of data (e.g. classroom/teacher/students/projects/resources etc) that can be defined easily via copying and pasting the whole MEME macOS distribution folder.
• Technical users should be able to define a starting project database by editing template *.db.js files.

Generally when preparing an Electron package to deliver to another site, you need to define three things:

I. Prepare Resources
II. Prepare Default Template
III. Prepare Distribution Files

Repo paths

Where to find files (for clarity, we are assuming / is the root repo directory, e.g. /src/ refers to /<repodir>/src/):

Folder ()	Source Code	Template Data	Local Runtime Data	Electron Runtime Data
/src/	X
/docs/	X
/node_modules/	from package.json
/templates/		X
/templates/_blank/		X
/built/			built from /src/
/data/			copied from /templates/
/resources/			references /resources/ directly
/dist/				built from /src/ via electron compile
/dist/MEME macOS/				X
/dist/MEME macOS/data/				copied from /data/
/dist/MEME macOS/resources/				copied from /templates/_blank/ and manual override
/dist/MEME macOS/templates/				copied from /templates/ and manual override

Where do I find my current running project database (loki) files?

The current running project database files are saved in different paths depending on how you start the server.

Run Mode	Command	Sources	Runtime Path*
Development	npm run dev	/meme/templates/test/*.db.js or if not exist, /meme/templates/_blank/*.db.js	/meme/data/db/test.loki
Electron Debug from Command Line -- with live reload (does not build dist)	npm run electron	meme.loki or if not exist, /meme/templates/meme/*.db.js or if not exist, /meme/templates/_blank_/*.db.js	/meme/data/db/meme.loki
"App" Mode -- Production from Command Line -- with console output	npm run app	same as electron	/meme/dist/MEME macOS/data/db/meme.loki
"App" Mode -- Production from Electron App	Double-click meme.app	same as electron	/meme/dist/MEME macOS/data/db/meme.loki

• The default project database name is always:

  • test.loki -- when running a Local Server via npm run dev, or
  • meme.loki -- when running an Electron app via npm run electron, npm run app, or double clicking the Electron meme.app.

• The two names (test.loki and meme.loki) are "burned in" and cannot be changed for loading. (But you can replace one meme.loki with another meme.lokifile in the same folder to swap out databases.

• Since the project database is just a single file, you can copy, archive, and replace test.loki and meme.loki files at will to manage versions over time.

• Generally, you only change the name (e.g. test.loki vs meme.loki) if you need to manually create a backup for archival purposes. e.g. you want to save your pilot study as meme_2023-0930.loki. If you want to load a different db file, you'd either use the Electron MKZIP export/import, or rename the db file from meme_2023-0930.loki to meme.loki.

• NOTE that the initial _blank template project is created from *.db.js files, that are then compiled and saved as *.loki when the app loads.

• The default runtime folder is <repo_folder>/data/test.loki, typically /meme/data/test.loki.

---

I. Prepare Resources

How do I add resources and their corresponding URLs in the Resource Editor?

Projects rely on references to "resources". Resources might be files (e.g. pdfs of articles, graphics), or simply links (e.g. url to a website, url to a simulation). File resources must be prepared for a project in advance. Link resources can be set up with the project.

Adding File Resources

There are two ways to add resources:

1. Use the "resources" defined in the main template to automatically copy resources for running during runtime, usually stored at /<repodir>/templates/_blank/resources/*.*, or
2. Add resources as you develop materials for the project directly to the corresponding runtime path, usually stored at /<repodir>/resources/*.* for local dev or at /<repodir>/dist/MEME macOS/resources/*.* when using Electron.

Resources are saved either to the local server or to the Electron app, depending on how you start the app.

Local Server Resources

For Local Servers, during compile the resources are copied from the /<repodir>/templates/_blank/resources to the /<repodir>/resources folder. You can then add additional resources as needed.

Electron App Resources

For the Electron App, during compile the resources are copied from the /<repodir>/templates/_blank/resources to the /<repodir>/dist/MEME macOS/resources folder. You can then add additional resources as needed.

Here's a table summarizing the sources and paths of resources

Run Mode	Command	Sources	Runtime Path*
Development	npm run dev	/<repodir>/templates/_blank/resources/*.*	/<repodir>/resources/*.*
Electron Debug from Command Line -- with live reload (does not build dist)	npm run electron	/<repodir>/templates/_blank/resources/*.*	/<repodir>/resources/*.*
"App" Mode -- Production from Command Line -- with console output	npm run app	/<repodir>/templates/_blank/resources/*.*	/<repodir>/dist/MEME macOS/resources/*.*
"App" Mode -- Production from Electron App	Double-click meme.app	/<repodir>/templates/_blank/resources/*.*	/<repodir>/dist/MEME macOS/resources/*.*

*Runtime Path -- You can add additional non-templated resources in the Runtime path folder.

URL Reference

All references should use *.* as the URL for resources.

You can use subfolders as well, e.g. resources in the disk path /<repodir>/dist/MEME macOS/resources/subfolder/helloworld.pdf would use a URL of subfolder/helloworld.pdf.

(Prior versions of the app required specifying a static subfolder .../static/dlc/*.* for the url).

II. Prepare Default Template

New projects are always started from a template. By default, the _blank template is used. But existing projects (database files and resources) can be used to override the defaults.

How are projects created by default and where are they stored?

MEME tries to provide a graceful way to fall back to usable projects and templates. Here's the order in which MEME will attempt to create projects:

... for Local Development (npm run dev )

For local development, the test.loki database will always be generated from the *.db.js files with every run.

1. Using the files in...
   • <repodir>/templates/_blank/*.db.js and
   • <repodir>/templates/_blank/resources/*...
2. ...generate and load <repodir>/data/db/test.loki

...for Electron ( npm run electron, npm run app, or double-clicking meme.app)

For Electron Mode, we try these in order:

1. Use meme.loki

   • If meme.loki has been created...
   • ...copy <repodir>/data/db/meme.loki ...
   • ...to <repodir>/dist/MEME macOS/data/db/meme.loki

2. Use templates/meme/*.db.js

   • If meme.loki has not been created yet, but there is a <repodir>/templates/meme/ folder...
   • ...use <repodir>/templates/meme/*.db.js files to...
   • ...construct <repodir>/dist/MEME macOS/data/db/meme.loki

3. Fall back to /templates/_blank:

   • ...use <repodir>/templates/_blank/*.db.js to...

   • ...construct a new <repodir>/dist/MEME macOS/data/db/meme.loki

How do I create a template for new projects?

If you want to create a starting template for projects, e.g. you want to create water quality projects that share resources to use for a few different sites, you have two options:

A. Duplicate the dist meme.loki

The easiest method is to run the distributed electron version and update the files in the <repodir>/dist/MEME macOS/ folder:

1. Build, package, and run the electron app

npm run electron
npm run package
npm run appsign

2. Start Electron via npm run app or double clicking the meme.app to load and run the files in <repodir>/dist/MEME macOS/
3. Edit any resources, add any teachers, classrooms, and other settings
4. Build and run the electron app to make sure the data is correct

npm run package
npm run appsign

6. Once you've confirmed everything is to your liking
7. Duplicate the whole <repodir>/dist/MEME macOS/ folder

The copied app will start with the same project data.

This is somewhat fragile in that you'd need to lock down the meme.loki and associated files to make sure you don't inadvertently change something. The eaiset method is probably to simply zip the <repodir>/dist/MEME macOS/ folder.

B. Clone the meme.loki

If you've already been refining the running meme.loki file you can also just run the packager to build the distribution files.

1. npm run electron
2. Edit any resources, add any teachers, classrooms, and other settings
3. Build and run the electron app to make sure the data is correct

npm run package
npm run appsign

4. Duplicate the whole <repodir>/dist/MEME macOS/ folder

C. Edit <repodir>/templates/meme/*.db.js files

For advanced coders, you can edit the *.db.js files directly. The easiest way here is to:

1. Copy <repodir>/templates/_blank/*.db.js to <repodir>/templates/meme/*.db.js
2. Edit the files in <repodir>/templates/meme/*.db.js
3. Build and run the electron app to make sure the data is correct
4. Duplicate the whole<repodir>/dist/MEME macOS/ folder

The copied app will then start with fresh *.db.js files on the initial build.

NOTE that any changes you make to the Electron app will then be saved in the current meme.loki file, and you can just copy those. Any changes to meme.loki will NOT be reflected in the <repodir>/templates/meme/*.db.jsfiles though -- you'll have to duplicate those changes by hand editing the *.db.js files.

III. Prepare Distribution Files

Here's what you need to do to prepare the combined package of the Electron app and project data for sharing to other sites.

Overview

1. Decide on a template
2. Prepare resources
3. Customize starting project
4. Package and Sign
5. Fix Quarantine Flag
6. Test Other Computers
7. Zip, Copy, and Distribute

1. Decide on a Template

Generally you would start with the __blank template. If you want to use another template, see II. Prepare Default Template for more information.

2. Prepare Resources

If you start with the _blank template, any resources already defined in /<repodir>/templates/_blank/resources/*.* will be copied over.

If you want to add additional resources add them to /<repodir>/resources/*.*. During packaging, the resources will be copied over.

3. Customize Starting Project

Do an initial compile to build the meme.loki file to review the starting setup.

npm run electron
http://localhost/#/admin

Add any classrooms, teachers, student groups, and students you want to share across all copies.

Add any ratings definitions, criteria for a good model, sentence starters, and resource definitions.

If you want to add a starting demo project, login and create a model.

4. Package and Sign

npm run electron   # build the `meme.loki` file from `*.db.js` files and run Electron locally
npm run package    # build the the distribution files for `meme.app` Electron app
npm run appsign    # code sign the `meme.app`
npm run app        # or double click on `meme.app`

Run the app to make sure it looks right.

5. Fix Quarantine Flag (macOS)

In addition to codesigning, if you want to copy the Electron app to another machine, you need to run a install.sh script.

1. Build the electron app (npm run package) and copy the /dist/MEME macOS folder to your new computer/new location, e.g. to ~/Desktop/MEME macOS.
2. Open a terminal.
3. cd ~/Desktop/MEME macOS
4. ./install.sh
5. The quarantine flag should now be fixed.
6. Double click the meme.app Electron application to start the server to make sure it works.
7. If you see javascript error, then the quarantine flag was probably not removed successfully. Try typing xattr meme.app -- you should NOT see the line com.apple.quarantine If you copy the app to another computer you may have to run ./install.sh again.

6. Test Other Computers

Make sure everything works. Make sure you test:

• Intel Macs
• Apple Silicon Macs (M1, M2)
• Older macOS versions that you might be running on

7. Zip, Copy, and Distribute

Once everything is set you can zip the <repodir>/dist/MEME macoS/ folder and copy the zip file to another computer.

Unzip it to run it.