This repository has been archived by the owner on May 6, 2024. It is now read-only.
Eric/dataset #318
Merged
Eric/dataset #318
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
drothermel
reviewed
Apr 28, 2022
Originally written by @HNR.
have int16 as a valid type.
to gameid, change fundamental sql request.
Add a new integration test testing the construction of the dataset from the altorg and nle_data directories. These altorg games have been checked to see if they are consecutive episodes, and the match up is correct. In this mock directory we include extra mock ttyrecs that should not be included in the dataset. Readmore about the mock dataset in altorg/about.txt Also: change dataset to default shuffle games.
Remove all mention of torch, and instead just ship numpy arrays.
documentation of populate db. We want to blacklist files that we have had trouble rendering in the past. In experience this is only a handful of ttyrecs in millions.
cdmatters
force-pushed
the
eric/dataset
branch
3 times, most recently
from
6be75b7
to
9bc95a4
Compare
NLE does not use the standard ttyrec format, used by alt.org and others. It stores actions, as well as states, albeit in a different channel. This commit introduces a version to the ttyrec format - original ttyrecs are still 'ttyrec.bz2', whereas henceforth NLE generates 'ttyrec2.bz2'. This will be more useful down the line, as readers can easily detect which way to read ttyrecs.
This allows the dataset to know which version of ttyrec is being read, which will allow the downstream Dataset object to chose the right way to read the arrays in future. This ttyrec version is always 1 for 'altorg' or old style ttyrecs, and currently 2 for any nle generated ttyrecs.
Now the converter objects are aware of which version of ttyrec they are decoding, have been passed it from the the dataset.
cdmatters
force-pushed
the
eric/dataset
branch
2 times, most recently
from
f656ec5
to
7319368
Compare
We will now start storing the in-game score in ttyrecs, to provide greater information for offline learning methods. This is done by utilising a different 'channel' for score. As it stands the channels are: - 0 - terminal information - 1 - action input - 2 - in-game score (based on `botl_score()` via `blstats[NLE_BL_SCORE]`) Note the in-game score is recorded once, just before `nle_step` returns, at the same time that `obs->done` is set. This is done to avoid having to store the in-game score everytime the game flushes to screen, and to keep the final ttyrec size low.
Add support for v3 ttyrecs in the converter, and feed through to the dataset. Also add support to the read_tty script. Note the converter now writes a score when the frame comes up, but only "flushes" the main frame when the action comes in. We test with ttyrec with an example of gaining core from picking up gold and killing monsters. We expect the action at index t to correspond to the state and reward just passed.
heiner
approved these changes
May 9, 2022
|
Again, no major comments except that the new versioning function is very convenient. Two minor comments. |
drothermel
approved these changes
May 12, 2022
nle/dataset/dataset.py
Outdated
| :param load_fn: A callback that loads the next file into a converter: | ||
| sig: load_fn(converter) -> bool is_success | ||
|
|
||
| Note actions will only be non-null if the ttyrec read_actions flag is set. |
There was a problem hiding this comment.
This note is no longer relevant I believe
nle/scripts/read_tty.py
Outdated
| @@ -181,6 +181,11 @@ def main(): | |||
| char, *_ = struct.unpack("<B", data) | |||
| data = chr(char).encode("ascii", "backslashreplace") | |||
| arrow = "->" | |||
| elif channel == 2: | |||
| score, *_ = struct.unpack("<i", data) | |||
| # data = chr(score).encode("ascii", "backslashreplace") | |||
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.