Skip to content
This repository has been archived by the owner on Oct 11, 2023. It is now read-only.

LIGHT Refactor phase 2 - full deploy #310

Draft
wants to merge 1,407 commits into
base: main
Choose a base branch
from
Draft

Conversation

JackUrb
Copy link
Contributor

@JackUrb JackUrb commented Aug 22, 2022

Overview

Part 2 of the major LIGHT refactor includes migrating the codebase to a shared config setup, then using more well-contained servers for our deploy.

Steps

  • [LRP2] Hydra-ize local play and server deploys. #309 (hydrize the codebase, ensure configs work on live deploys). Live now runs on same code as local
  • Combine TornadoPlayerProvider and RegistryApplication into post-auth WorldServer
  • Create WorldServer registry table
  • Create LandingServer, inheriting the login components scrapped from the above, with new routing logic.
  • Commit terraform code for launching full server

jaseweston and others added 30 commits April 29, 2021 18:23
on_use events are totally broken
More backend and event fixes pre-launch
* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Addressing comments, clarifying code
* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou
* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix
* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented
* Some initial transitions over to model pool

* Moving initialization code out from where it occurred

* Wiring more of the system together

* Adding opt for reranked generative

* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented

* Upgrading OneRoomGraphBuilder to ModelPool

* Completing (almost) the rest of Modelpool references

* Works without loading models in play_map

* Model pool actually works

* Safety working as well

* removing prints

* Fixing some tests, skipping starspace

* Runs on server too

* Enums for model types
* Some initial transitions over to model pool

* Moving initialization code out from where it occurred

* Wiring more of the system together

* Adding opt for reranked generative

* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented

* Upgrading OneRoomGraphBuilder to ModelPool

* Completing (almost) the rest of Modelpool references

* Works without loading models in play_map

* Model pool actually works

* Safety working as well

* removing prints

* Fixing some tests, skipping starspace

* Runs on server too

* Creating LIGHT's ModelServer

* Undo server change

* tornado simplicity

* Handling for inline candidate models

* But regular models should also work without this

* Enums for model types
* Some initial transitions over to model pool

* Moving initialization code out from where it occurred

* Wiring more of the system together

* Adding opt for reranked generative

* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented

* Upgrading OneRoomGraphBuilder to ModelPool

* Completing (almost) the rest of Modelpool references

* Works without loading models in play_map

* Model pool actually works

* Safety working as well

* removing prints

* Fixing some tests, skipping starspace

* Runs on server too

* Creating LIGHT's ModelServer

* Undo server change

* tornado simplicity

* Handling for inline candidate models

* But regular models should also work without this

* Async... all of the things...

* Async the server too

* clearing up async server tests

* Correct async mock

* internalize init_world

* clean up tornado usage

* small GameInstance bug

* small GameInstance bug

* Some deploy fixes

* Moving safety model to async part

* Some safety fixes

* test fixes

* Enums for model types

* Checking for non-list to convert first
* Some initial transitions over to model pool

* Moving initialization code out from where it occurred

* Wiring more of the system together

* Adding opt for reranked generative

* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented

* Upgrading OneRoomGraphBuilder to ModelPool

* Completing (almost) the rest of Modelpool references

* Works without loading models in play_map

* Model pool actually works

* Safety working as well

* removing prints

* Fixing some tests, skipping starspace

* Runs on server too

* Creating LIGHT's ModelServer

* Undo server change

* tornado simplicity

* Handling for inline candidate models

* But regular models should also work without this

* Async... all of the things...

* Async the server too

* clearing up async server tests

* Correct async mock

* internalize init_world

* clean up tornado usage

* small GameInstance bug

* small GameInstance bug

* Some deploy fixes

* now using aws as a storage backend

* Moving safety model to async part

* Some safety fixes

* test fixes

* silly elif fix
* Some initial transitions over to model pool

* Moving initialization code out from where it occurred

* Wiring more of the system together

* Adding opt for reranked generative

* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented

* Upgrading OneRoomGraphBuilder to ModelPool

* Completing (almost) the rest of Modelpool references

* Works without loading models in play_map

* Model pool actually works

* Safety working as well

* removing prints

* Fixing some tests, skipping starspace

* Runs on server too

* Creating LIGHT's ModelServer

* Undo server change

* tornado simplicity

* Handling for inline candidate models

* But regular models should also work without this

* Async... all of the things...

* Async the server too

* clearing up async server tests

* Correct async mock

* internalize init_world

* clean up tornado usage

* small GameInstance bug

* small GameInstance bug

* Some deploy fixes

* now using aws as a storage backend

* Moving safety model to async part

* Some safety fixes

* test fixes

* silly elif fix

* Taking bug-fixes from stable server

* Model server changes too

* Skip another web test, works in prod, refactor incoming

* Enums for model types

* Checking for non-list to convert first

* Slightly more clarity
* Some initial transitions over to model pool

* Moving initialization code out from where it occurred

* Wiring more of the system together

* Adding opt for reranked generative

* Filling out UserDB

* Abstract enforce get first

* Clearer argument - num_turns

* Using enums in DB

* Initial episode data model

* Update content loggers to use episode formatting

* Updating tables to work with testing

* Fixing some test changes

* Fixing small warnings that were noise during tests

* Moving default log path

* Test fix

* Correcting math thanks to Kurt

* Updating env DB classes to SQLAlchemy

* Name keys and Elems coded

* Adding arbitrary node attributes

* First complete pass of EnvDB

* Mypy fixings

* Fixing agents

* Writing some tests

* Finishing tests for object and room creates and queries

* Edge testing

* Arbitrary attributes testing

* Quests and testing

* And finally, DBGraph tests

* fixing episode change

* TODO function

* final mypy fixes

* DBID testing

* a -> either a or an depending on aeiou

* adding WorldConfig to hold complex configuration vars

* Moving episode_db into relevant GraphBuilders

* Game launches, but not logging

* Local BaseDB, now saving episodes

* Missing files

* deleting miscommit

* test fix

* Migrating to UserDB

* No more LIGHTDatabase in TornadoServer

* Fixing tests

* Works after testing locally

* Updated messaging for unimplemented

* Upgrading OneRoomGraphBuilder to ModelPool

* Completing (almost) the rest of Modelpool references

* Works without loading models in play_map

* Model pool actually works

* Safety working as well

* removing prints

* Fixing some tests, skipping starspace

* Runs on server too

* Creating LIGHT's ModelServer

* Undo server change

* tornado simplicity

* Handling for inline candidate models

* But regular models should also work without this

* Async... all of the things...

* Async the server too

* clearing up async server tests

* Correct async mock

* internalize init_world

* clean up tornado usage

* small GameInstance bug

* small GameInstance bug

* Some deploy fixes

* now using aws as a storage backend

* Moving safety model to async part

* Some safety fixes

* test fixes

* silly elif fix

* Taking bug-fixes from stable server

* Model server changes too

* Skip another web test, works in prod, refactor incoming

* Ensure we're not using FB user data

* Methods for scrubbing the datasets

* Deleting player data and related graph info

* Fixing bugs, adding tests

* Environment exporting

* Export episode DB too

* 60 != 90

* Addressing comments
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 22, 2022
@JackUrb JackUrb changed the base branch from main to new-data-model August 22, 2022 18:42
Base automatically changed from new-data-model to main December 9, 2022 20:31
@JackUrb JackUrb changed the base branch from main to light-qa-merge March 15, 2023 21:12
@JackUrb JackUrb changed the base branch from light-qa-merge to main March 15, 2023 21:12
@facebook-github-bot
Copy link

Hi @JackUrb!

Thank you for your pull request.

We require contributors to sign our Contributor License Agreement, and yours needs attention.

You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants