Skip to content
This repository has been archived by the owner on May 6, 2024. It is now read-only.

Eric/dataset #318

Merged
merged 27 commits into from
May 16, 2022
Merged

Eric/dataset #318

merged 27 commits into from
May 16, 2022

Commits on May 9, 2022

  1. Add the db file, copied across from internal repo.

    Originally written by @HNR.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    50e4065 View commit details
    Browse the repository at this point in the history
  2. Convert back from uint16 to int16, since torch (when used) will only

    have int16 as a valid type.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    562e8ec View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a8df6e8 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8aa7d9a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8832261 View commit details
    Browse the repository at this point in the history
  6. Add a means of loading files from xlogfile for nle_data. Rename rowids

    to gameid, change fundamental sql request.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    0465a11 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    55b2184 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    2c20479 View commit details
    Browse the repository at this point in the history
  9. Rename functions in db.py

    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    c969559 View commit details
    Browse the repository at this point in the history
  10. Fixing up tests for converter, db and dataset.

    Add a new integration test testing the construction of the dataset from
    the altorg and nle_data directories. These altorg games have been checked to see if they are
    consecutive episodes, and the match up is correct.
    
    In this mock directory we include extra mock ttyrecs that should not be
    included in the dataset. Readmore about the mock dataset in altorg/about.txt
    
    Also: change dataset to default shuffle games.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    9afd765 View commit details
    Browse the repository at this point in the history
  11. Perform torch-isectomy.

    Remove all mention of torch, and instead just ship numpy arrays.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    ebf0630 View commit details
    Browse the repository at this point in the history
  12. Fixup test.

    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    9e71d1d View commit details
    Browse the repository at this point in the history
  13. Add blacklist logic to altorg mock data test and improve the

    documentation of populate db.
    
    We want to blacklist files that we have had trouble rendering in the
    past. In experience this is only a handful of ttyrecs in millions.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    25c6a77 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    caa3326 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    4ffaeb7 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    0ad3c12 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    34517f1 View commit details
    Browse the repository at this point in the history
  18. Add a ttyrec version to NLE.

    NLE does not use the standard ttyrec format, used by alt.org and others.
    It stores actions, as well as states, albeit in a different channel.
    This commit introduces a version to the ttyrec format - original
    ttyrecs are still 'ttyrec.bz2', whereas henceforth NLE generates
    'ttyrec2.bz2'.
    
    This will be more useful down the line, as readers can easily detect
    which way to read ttyrecs.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    06f016b View commit details
    Browse the repository at this point in the history
  19. Add ttyrec_version to the database schema.

    This allows the dataset to know which version of ttyrec is being read,
    which will allow the downstream Dataset object to chose the right way to read
    the arrays in future.
    
    This ttyrec version is always 1 for 'altorg' or old style ttyrecs, and
    currently 2 for any nle generated ttyrecs.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    2cbaa2d View commit details
    Browse the repository at this point in the history
  20. Add the ttyrec version to the dataset and converter objects.

    Now the converter objects are aware of which version of ttyrec they are
    decoding, have been passed it from the the dataset.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    3548aed View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    860dd08 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    3f633c7 View commit details
    Browse the repository at this point in the history
  23. Log in-game score in ttyrecs and bump ttyrec version to 3.

    We will now start storing the in-game score in ttyrecs, to provide
    greater information for offline learning methods. This is done by
    utilising a different 'channel' for score. As it stands the channels
    are:
     - 0 - terminal information
     - 1 - action input
     - 2 - in-game score (based on `botl_score()` via `blstats[NLE_BL_SCORE]`)
    
     Note the in-game score is recorded once, just before `nle_step` returns, at
     the same time that `obs->done` is set. This is done to avoid having to store
     the in-game score everytime the game flushes to screen, and to keep the final
     ttyrec size low.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    e67f15f View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    c453cb5 View commit details
    Browse the repository at this point in the history
  25. Add support for v3 ttyrecs in the dataset.

    Add support for v3 ttyrecs in the converter, and feed through to the
    dataset. Also add support to the read_tty script.  Note the converter
    now writes a score when the frame comes up, but only "flushes" the main
    frame when the action comes in.
    
    We test with ttyrec with an example of gaining core from picking up gold and
    killing monsters. We expect the action at index t to correspond to the
    state and reward just passed.
    cdmatters committed May 9, 2022
    Configuration menu
    Copy the full SHA
    4d5d932 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2022

  1. Configuration menu
    Copy the full SHA
    f6896d1 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2022

  1. Configuration menu
    Copy the full SHA
    19954e4 View commit details
    Browse the repository at this point in the history