New environment db

[JackUrb]

Overview

Following in the steps of #293 and as part of #291, this PR introduces the last piece of the replacement for the LIGHTDatabase. Importantly, this is built in a way that can be both deployed on the server and exported for local use. Further, it's built to contain and organize all of our current data forms. Some key functionality:

• Includes clean ways (DBElems) to manage the entities that back GraphNodes, and will allow us to create easier tools for directly building graphs from entities in the database.
  • DBElems have support for querying every relevant edge type (via the DBEdge and DBTextEdge) tables.
  • DBElems also now have support for arbitrary attributes (with name-key pairs)
  • All three DBElem tables and the DBEdge tables have some solid indexing, which should make queries a lot faster than the old LIGHTDatabase
  • The EnvDB comes out-of-box with the ability to preload everything into a _cache dict format, which may be more familiar.
• Edits and Flags:
  • The EnvDB is able to track both suggested edits to contents inside of the environment, as well as any time that a user flags content. We'll eventually be able to action directly against the DB from these constructs, but for now we can at least log them.
  • Note: One potential mis-match here is that flags for utterances/users are stored alongside flags for env contents. I don't think it makes sense to keep these in a separate table however.
• DBQuests are now part of the environment
  • These track different types and tiers of motivation (via a tree-style structure) based on the original collection format
  • The DBQuests have helpers for retrieving relevant quests up and down their chains
  • As of now, motivations are only allowed to have one parent, as this represents the direct annotation from one goal to being part of some larger motivation. If we end up wanting to cover a full set of links in the future, we may pursue adding a DBBaseEdge-style pairing system instead.
• We now have a place prepared to store user graphs in the long run, in the DBGraph's table. This is done by using the methods in the underlying BaseDB to cover appropriate file handling, and the EnvDB can act as a manifest for these graphs.

Note: This PR is pretty massive, I'm not expecting a full review here. If the above description makes sense, and you think the test coverage is enough to be confident about, then it's probably good to accept. If you're curious about SQLAlchemy setups though it might be an interesting pore-through! Once we have the proof of concept running E2E and are collecting data again, I'll loop back and do a documentation writeup.

Implementation

DBStatus

One key design consideration: all elements inserted into the DB by default are put into a REVIEW status. When pre-filling from our existing data, we can pre-approve, but once the game is live we won't want to make anything directly accessible. As such, all elements in the environment have a status field.

HasDBIDMixin

In order to have some kind of understanding of what the new arbitrary IDs are pointing to, all UUIDs entered into LIGHT have a 3 character prefix key. The HasDBIDMixin provides functionality for generating unique ids for a specific DB element, and for determining if a given ID belongs to a specific class.

DBElems

These cover the concepts of Rooms, Agents, and Objects that were in the original LIGHT dataset. In the new EnvDB these are built into DBElems, which each have corresponding DBNameKeys (for rough deduplication). All of the originally collected attributes as well as the other ones we've come up with have been built into the main table for these. Usage is as follows:

>> db = EnvDB(...)
>> room_1_id = db.create_room_entry(
..      name='Some room name',
..      base_name='base room',
..      description='It's a room',
..      backstory='A room used for example',
..  )
>> room_1 = db.get_room(room_1_id)

This code creates both the DBRoom as well as the DBRoomName if necessary.

Attributes

New to this DB, we introduce arbitrary attributes, which are saved as key-value pairs that can be attached to any node in the graph. From a DBElem we can query the ones directly tied to a specific element, though they can also be queried broadly from the table.

Edges

Both edge types, node_edges and text_edges, are available as loaded attributes of any DBElem. They link to lists of DBEdge and DBTextEdge respectively, and all of the details about those edges (including nodes they point to) can be directly referenced.

DBFlags and DBEdits

Not too much to share here, these are pretty standard storage right now, and the key functionality for executing them is yet to be implemented. We do want to start saving these in any live game though.

DBGraphs

The DBGraph table acts as a manifest for all currently saved and available worlds. For the production setting, these will be tied to creator IDs, though on export we'll likely drop these and replace the creator ID with the split that the graphs were released as a part of.

These only track the most-recently saved version of a user's graph.

In order to actually start tracking these, I've added db_id and title fields to the underlying OOGraph structure.

Testing

• mypy passes for the EnvDB
• New testing for the EnvDB. Coverage reporting below:

>> coverage report
Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
...
light/data_model/db/base.py                             81     12    85%
light/data_model/db/environment.py                     900     33    96%
light/data_model/db/episodes.py                        160      7    96%
light/data_model/db/users.py                           105      5    95%
...
------------------------------------------------------------------------

[klshuster]

taking your word for it, as you suggested

[klshuster]

just curious - are these mostly arbitrary?

[JackUrb]

Mostly, I went with reasonable lengths from what I've seen in the DB so far, then added ~50%.

Only ID_STRING_LENGTH is exact, as it's a 36 character uuid-4 with a 4 character KEY- attached.

[klshuster]

is this enforced?

[JackUrb]

It is by ID_STRING_LENGTH, but nothing else will cause a failure directly just from the class imports. I imagine it'd be possible to add a test that imports this class, finds all subclasses of HasDBIDMixin, and then asserts the length though... Let me see if that's doable in ~5m