Design Data Management

2019-10-18 Id String vs Number Type Issues

The database uses numeric ids for indexing/search efficiency. We rely on the numeric ids to automatically assign new ids to created objects. However, the client side's libraries tend to use strings instead of numbers:

• The way we iterate over keys using Object.keys() will always return strings instead of numbers. Our fast lookup structures use strings as a result.
• The graphing library we use, Graphlib, also returns its list of node ids as strings even though we are writing them as numbers, and the keys themselves are numbers.
• The UI/HTML tends to sometimes return strings (when reading from a field) or was originally written with string ids in mind. Our original test graph uses text ids instead of numeric ids

We have the following convention currently:

• Client side should just assume IDs are strings, even if they are numbers.
• Database always uses IDs that are numbers, though it will happily accept an id in string format and then fail to find it in update and delete ops.
• NUMBER->STRING CONVERSION: The essential pmc-data.js routines InitializeModel() and BuildModel() are the only places where numeric ids from the server are converted to string ids. All code that touches data through PMCDATA can safely assume that the data structures are using string ids locally.
• STRING->NUMBER CONVERSION: Any communication with the database server-database.js via UR.DBQuery() needs to convert string ids to numbers. This is done using Number(id), where id is a string representation of a number.

2019-10-04 Client-Server DB Integration

The database is implemented using lokijs as a set of individual collections via the DATA module. Another module DATAMAP is used to validate the data objects that are sent to the server to update these collections. Hence, to add a new collection, you just need to update the DATAMAP.DBKEYS collection, and the server/client code will automatically know about them.

Writing to the Database

• Data is written through UR.DBQuery(cmd,data). The cmd parameter is either 'add', 'remove', or 'update', and data is an object that contains the database collections to be modified. DATAMAP maintains the list of associated messages in its DBCMDS structure, mapping the short command name with the longer URSYS message name. For example 'add' invokes a NetCall to 'NET:SRV_DBADD'), which is the main add handler.

• Data is not read directly. Instead, browser applications rely on the DATA module receiving 'NET:SYSTEM_DBSYNC' to synchronize changes to the underlying data structure. This mirrors how the React interface and SVG plotting modules work; in general, our dataflow model is to be responsive to change, instead of forcing change with fragile conditional logic. Subscribing to 'DATA_UPDATED' should be sufficient for receiving change notifications.

Each of the commands accept a different data input, and return the changed data.

• Use UR.DBQuery('add',data).then(results=>{}) to add data, optionally receive objects with added ids.
• Use UR.DBQuery('update',data).then(results=>{}) to update data by id, optionally receive changed data as confirmation.
• Use UR.DBQuery('remove',data).then(results=>{}) to remove data by id, optionally receive the deleted objects

The format of data are "collection keys" corresponding to DATAMAP.DBKEYS. Each key contains ONLY a value. The type of value depends on the command:

• 'add' - values are objects WITHOUT an id property. The returned objects will have assigned ids if they are needed at the time of creation as they are inserted into the designated collection.
• 'update' - values are objects WITH an id property that are used to update specific document objects in the designated collection.
• 'remove' - values are objects WITH an id prropery, which is used to delete the matching id in the designated collection.

If you pass a single value to a collection defined in the data parameter, UR.DBQuery will return a single value. EXAMPLE:

// add a value to the teachers collection
// receives rdata back from server
UR.DBQuery('add',{
  teachers: { name: 'Mrs. Smith' }
}).then(rdata=>{
  const teacherId = rdata.id; // added by db
  const name = rdata.name;
});

// FYI: collections are defined in DATAMAP.DBKEYS
// FYI: More examples in DATA's window.ur test methods

WARNING! While these methods all return the changed objects asynchronously, you should NOT force a database update in your optional then() handler. Let the DATA module do it for you, or handle it through the appropriate DBSYNC handler that eventually fires a DATA_UPDATED or BuildModel() followed by DATA_UPDATE. You can, however change viewmodel properties as a result of a data update that might affect the view (e.g. selected items).

Handling DBSync

Currently there is no handler for the DB updates that synch data, but this is coming next. The stub code is in data.js, which implements the 'NET:SYSTEM_DBSYNC' message handler for:

• sync add - calls MIR.SyncAddedData(data)
• sync update - calls MIR.SyncUpdatedData(data)
• sync remove - calls MIR.SyncRemovedData(data)

The data parameter is the 'NET:SYSTEM_DBSYNC' data packet received from the network, which can be converted into an array of syncdata objects through DATAMAP.ExtractSyncData(data) as follows:

const syncdata = DATAMA.ExtractSyncData(data);
syncdata.forEach(item=>{
  const { colkey, subkey, value } = item;
  ...
});

• colkey is the collection name that was updated, which can be used as a key to update the local data structure and trigger conditional logic. It is always the main collection name.
• subkey is the property name within the collection. This is used for collections of the form pmcData.entities, where pmcData refers to the pmcData collection, and entities refers to a prop within a pmcData record.
• value is the data used by the server to modify the database.

Value Format Examples

DBQuery Extract

Use DATAMAP.ExtractQueryData(data) on the server to extract params, which returns array of objects { colkey, subkey, value }

data.cmd	colkey	subkey	value
add	pmcData		{ ...props }
update	pmcData		{ id, ...props }
remove	pmcData		{ id }
add	pmcData	entities	{ id, entities:{ ...props } }
update	pmcData	entities	{ id, entities:{ id, ...props } }
remove	pmcData	entities	{ id, entities:{ id } }

DBSync Receive

You should rely on DBSYNC RECEIVE to handle data-driven updates, but a DBQuery() returns a Promise that also receives the updated data in DBSYNC format (see below). This may be useful in some cases where you need to make a local state update of a component, but you can not be sure if other changed data available yet.

DBSync Extract

Use DATAMAP.ExtractSyncData(data) to parse incoming sync data, which returns array of objects { colkey, subkey, value } as follows:

data.cmd	colkey	subkey	value
add	pmcData		{ id, ...props }
update	pmcData		{ id, ...props }
remove	pmcData		{ id, ...props }
add	pmcData	entities	{ id, ...props }
update	pmcData	entities	{ id, ...props }
remove	pmcData	entities	{ id, ...props }

NOTE: It is possible for a single data input to produce multiple syncdata object in the output array, as the database always sends sync in an array within the data packet.

2019-07-16 Classroom Admin DB Integration

ISSUES

1. adm-data.Load()

   • load loki tables at app initialization
   • ViewMain subscribes to load event
   • ViewMain et al render on load event
   • ViewMain et al update display via data publish
   • UR/Server handles everything (save, sync) behind the scenes
   • proper UID generator

2. Student login tokens

   • GroupsList needs to be able to generate tokens

3. student login/home page

   • students need a login
   • after they login they see a list of their models
   • what should the routes be? \

   => login + list of models (after login) \

   #/model/ => model URL?

4. model load

   • select model from home page
   • How do you return to home page?

5. Set resources for classroom via UI.

   • Save resource setting on checkbox click.

The Basic Loop

LOAD_DATA - UR system calls the LOAD_DATA hook defined in ViewMain (or wherever)

  UR.Lifecycle('LOAD_DATA',(data)=> {
    // this code will fire during the LOAD_DATA PHASE
    // which occurs BEFORE React even starts any rendering of anything
    // to ensure data is present

    // do initial mode setting that will affect LATER rendering styles
    // flags to set 
  });

  UR.OnDataBaseUpdate('ADMIN', (admdata)=> {
    // this fires when the ADMIN data set has new data!
    // at this point, you can use this to set React stuff
    // by parsing data and using setState() to make React
    // draw its elements

    // optimization: data might have some way of determing
    // what changed e.g
    if (admdata.teachers) {}
    if (admdata.classrooms) {}
    if (admdata.groups) {}
    // ...
    if (admdata.classroomResources) {}
  });

NAMING THOUGHTS - parts of things that go into a method name

• action-noun or noun-action, unambiguous which is the noun and which is the action
• state: pending, promise, or request
• code mechanism: listener, handler, pattern

How To Shim This In Until Ursys Is Working

adm-data.Load

const LoadData = (data) => {
    /* current hardcoded data stuff */
};

// FAKE THE CALL
LoadData(); // will become UR.LifeCycle('LOAD_DATA', LoadData);

ViewAdmin.jsx Load

in constructor:

Api Stuff

• what to name all the UR functions, partiular for URDATA SYNCHED LOADS

Implementation

ADMDATA.Load()

STUDENT LOGIN/HOME PAGE

MODEL LOAD

• copy/paste model URLs for sharing

To LISTEN/RECEIVE ADMIN DATA subscribe to the 'ADMIN:UPDATED' system event

// Admin Data Module
let db_admin = {}; 

UR.DB_Subscribe('ADMIN:UPDATED', localHandler);
// in class definition
localHandler(data) {
// save a local instance if you need to refer
// to it later
db_admin = data.db_admin;
    // UR.Pub('ADM_DATA_UPDATED');
}

ADMData.GetClassroomsByTeacher = teacherId => {
    return db_admin.classrooms.filter( 
        cls => cls.teacherId === teacherId 
    );
}

// Component
UR.DB_Subscribe('ADMIN:UPDATED', AdmDataUpdated);
//  UR.Sub('ADM_DATA_UPDATED', AdmDataUpdated);
// we're going to
AdmDataUpdated() { 
    classrooms = ADM.GetClassrooms();
    this.setState(classrooms: data.classrooms);
}

To USE ADMIN DATA in a component

• Call AdminData methods that are defined to return data structure. They are GUARANTEED to be up-to-date.
• NOTE: Components rerender data only on receive of 'ADMIN:UPDATED'

To CHANGE ADMIN DATA

• Use UR wrapped call UR.DB_GetObject('ADMIN',(data)=>{});
• when the function is executed by UR, it will then check the data object for changes and update itself consistently
• after the object is updated, UR will fire the related 'ADMIN:UPDATED' system event

To RENDER NEW ADMIN DATA

• subscribe to the 'ADMIN:UPDATED' system event
• setState() so React renders its view defined in render()

Traffic Cop Model

AdminData is the ONLY module that will use the UR.DATABASE calls and events AdminData will publish its own DERIVED event calls that drive UI changes when the database updates These derived event calls are defined based on the groups that have changed (e.g. the original groups object, maybe DERIVED group objects if there are any components that depend on them)

ADMINDATA EMITTED EVENT TYPE: BASE DATABASE OBJECT has changed ADMINDATA EMITTED EVENT TYPE: DERIVED DB OBJECT has changed (e.g. filtered list of classrooms has changed) ADMINDATA EMITTED EVENT TYPE: DERIVED COMPONENT DISPLAY DATA STRUCTURE has changed (e.g. teacher is selected)

Question: in the case of a teacher selected

• any related display elements in other components need to know to update
• PATTERN NOTE: the source component that receives the SELECT TEACHER events as part of local UI never sets its state directly IF that state impacts other components. In this case, it fires a change to ADMIN DATA (or a VIEWMODEL manager that is tied closely to admindata) and it updates state on RECEVING the appropriate event.ADMDATA

Eg. instead of code like this:

onclick = read a value, setState to update value in control, then pub event 'something happened'

...you write...

onclick = read a value, call VIEWMODEL changeValue(), and let the related EVENT come back to trigger the update of control.

Because this ensures (1) reduces change of async order of operation errors which also (2) reduces the need for component coupling in unsavory ways (props, call chains, event multiplication