human action space + enabled GUI #449

pzhokhov · 2021-02-02T18:53:01Z

adds a "LowLevelInput" mission flag that allows malmo input to be injected at mouse and keyboard level (instead of events level)
when lowLevelInput is on, in-game GUIs are enabled
new env specs - HumanControls (base spec class), HumanSurvival, and EquipWeapon. EquipWeapon is task to equip iron axe that is placed somewhere in the inventory using human inputs. The reward and done signal for equip weapon have to be implemented via a wrapper on the python side.

…. Use button is a little strange - single tick click opens places the crafting table and opens the inventory

…erl into peterz/human_action_space

…action_spaces_pr

pep8speaks · 2021-02-02T18:53:10Z

Hello @pzhokhov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-02-04 00:08:43 UTC

minerl/Malmo/Minecraft/build.gradle

brandonhoughton · 2021-02-02T22:26:39Z

minerl/Malmo/Minecraft/src/main/java/com/microsoft/Malmo/Client/ClientStateMachine.java

@@ -1874,9 +1874,15 @@ protected void onMissionStarted()

            // Force brightness setting
            Minecraft.getMinecraft().gameSettings.gammaSetting = (float) 2.0;


A gamma setting of 2 is brighter than playable without editing the options.txt manually (e.g. the highest in game via the brightness slider)

May also need to look into fov as it is beyond quake-pro in default minerl

good point, but I'd rather get a wider team buy-in before changing these values, since that would affect everyone's experiments. Currently in this PR the behavior (without lowLevelInput flag) should not change.

brandonhoughton · 2021-02-02T22:36:47Z

minerl/Malmo/Minecraft/src/main/java/com/microsoft/Malmo/Client/MalmoModClient.java

-     * Event listener that logs when agents open gui windows
-     * @param event the captured event
-     */
-    @SideOnly(Side.CLIENT)


This was originally here to debug if we were opening a gui window unintentionally, probably OK to remove

brandonhoughton · 2021-02-02T23:09:50Z

minerl/Malmo/Minecraft/src/main/java/com/microsoft/Malmo/Client/VideoHook.java

+        float x = (float) (player.lastTickPosX + (player.posX - player.lastTickPosX) * partialTicks);
+        float y = (float) (player.lastTickPosY + (player.posY - player.lastTickPosY) * partialTicks);
+        float z = (float) (player.lastTickPosZ + (player.posZ - player.lastTickPosZ) * partialTicks);
+        float yaw = player.prevRotationYaw + (player.rotationYaw - player.prevRotationYaw) * partialTicks;


I guess we forgot to zero them out here

brandonhoughton · 2021-02-03T20:56:16Z

minerl/Malmo/Minecraft/src/main/java/com/microsoft/Malmo/Client/MalmoModClient.java

@@ -208,43 +208,6 @@ public void onPressed()
        this.keyManager = new KeyManager(settings, extraKeys);
    }

-    /**


How will we prevent locking out non-gui agents from moving?

oh you make a good point, let me restore this bit of code. Deleted it accidentally with the debug function ^_^

brandonhoughton · 2021-02-03T23:52:20Z

...inecraft/src/main/java/com/microsoft/Malmo/MissionHandlers/CameraCommandsImplementation.java

+                        this.currentPitch = player.rotationPitch;
+                    }
+                }
+                return true;
            }
        } catch (NumberFormatException e) {
            System.out.println("ERROR: Malformed parameter string (" + parameter + ") - " + e.getMessage());


This is ignoring number format exceptions then? That is expected right?

my understanding is that, just like prior to the change, in case of NumberFormatExceptions, we print an error message and return false (which, AFAIK, means that the command will be passed further down the chain)

brandonhoughton · 2021-02-03T23:54:53Z

minerl/Malmo/Minecraft/src/main/java/com/microsoft/Malmo/Mixins/MixinMouse.java

+        FakeMouse.setGrabbed(grabbed);
+    }
+//
+//    @Overwrite(remap = false)


No-longer needed?

brandonhoughton · 2021-02-03T23:56:09Z

minerl/Malmo/Schemas/Mission.xsd

+                <xs:element name="LowLevelInputs" type="xs:boolean" default="false" minOccurs="0" maxOccurs="1">
+                    <xs:annotation>
+                        <xs:documentation>
+                            If set to true, inputs to malmo are injected instead of keyboard and mouse (instead of higher level commands)


Maybe "If set to true, inputs to malmo are disabled and instead given via a virtual keyboard and mouse (instead of higher level commands)"

brandonhoughton

Left a few more comments but otherwise looks good to me!

This reverts commit c70f6cc.

pzhokhov · 2021-03-11T19:15:31Z

...inecraft/src/main/java/com/microsoft/Malmo/MissionHandlers/CameraCommandsImplementation.java

-                    player.setPositionAndRotation(player.posX, player.posY, player.posZ, this.currentYaw, this.currentPitch);
+                    // TODO peterz: I am pretty sure this code is deprecated and does not match the docstring
+                    // i.e. no smooth panning is happening here
+                    // player.setPositionAndRotation(player.posX, player.posY, player.posZ, this.currentYaw, this.currentPitch);


seems to be causing extra flakiness in starting position?

pzhokhov and others added 30 commits November 13, 2020 11:19

human inputs envs - wip. Made FakeKeyboard work, need to modify launcher

d16da17

enable modding the lwjgl

b9c1540

.

e8c5511

fixed rendering the gui for python

2ab9143

add use button == right click, hotbar keys

ba75232

equip weapon task

18374b9

equip item env

52b48f3

equip weapon length is now 10 seconds

a60400f

render in java

2017caa

episode length 1200

2ea9f9e

.

a64c982

remove warning about gui open

0178a88

make hotbar configurable in equip env

4155dbc

wip

89a8d45

got viewer to work, checked causality of inventory and hotbar buttons…

4742c29

…. Use button is a little strange - single tick click opens places the crafting table and opens the inventory

data rendering pipeline fixes an updates

e0691eb

Merge branch 'peterz/human_action_space' of github.com:minerllabs/min…

e460d56

…erl into peterz/human_action_space

fixes

5e1ae4e

Merge branch 'peterz/human_action_space' of github.com:minerllabs/min…

6b1a072

…erl into peterz/human_action_space

.

faa60e8

update anvil jar

dd2a93f

.

dff4972

Merge branch 'peterz/human_action_space' of github.com:minerllabs/min…

66f4be5

…erl into peterz/human_action_space

.

2f4cea5

update render jar

58e8737

re-rendering pipeline

f7b2b12

.

dc4d459

pull latest master, resolve conflicts

0507b31

update render jar, fake mouse

fe63cce

Merge branch 'dev' of github.com:minerllabs/minerl into peterz/human_…

dfd03da

…action_spaces_pr

pzhokhov added 2 commits February 2, 2021 10:45

more cleanups

fc51e0b

.

052c322

pzhokhov added 2 commits February 2, 2021 11:42

.

a26f349

pep8

091f4d5

pzhokhov requested a review from brandonhoughton February 2, 2021 19:50

pzhokhov marked this pull request as ready for review February 2, 2021 19:52

more pep8

ef92e40

ilge reviewed Feb 2, 2021

View reviewed changes

minerl/Malmo/Minecraft/build.gradle Show resolved Hide resolved

pzhokhov added 2 commits February 2, 2021 12:59

do not use gcs

29dd741

.

4b8ad78

brandonhoughton reviewed Feb 2, 2021

View reviewed changes

pzhokhov added 2 commits February 2, 2021 15:05

merge dev

39cf9e9

cleanups, trying to fix build

ece6b95

brandonhoughton reviewed Feb 3, 2021

View reviewed changes

remove logic that denies block usage

f108e3b

brandonhoughton reviewed Feb 3, 2021

View reviewed changes

pzhokhov added 2 commits February 3, 2021 13:47

disable cursor

91a9252

restore logic disabling right click

372f907

brandonhoughton reviewed Feb 3, 2021

View reviewed changes

brandonhoughton approved these changes Feb 3, 2021

View reviewed changes

PR comments

c82dafc

pzhokhov merged commit c70f6cc into dev Feb 4, 2021

pzhokhov deleted the peterz/human_action_spaces_pr branch February 4, 2021 00:24

brandonhoughton added a commit that referenced this pull request Feb 4, 2021

Revert "human action space + enabled GUI (#449)"

04aae20

This reverts commit c70f6cc.

brandonhoughton added a commit that referenced this pull request Mar 11, 2021

Revert "human action space + enabled GUI (#449)"

426c9ec

This reverts commit c70f6cc.

pzhokhov commented Mar 11, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

human action space + enabled GUI #449

human action space + enabled GUI #449

pzhokhov commented Feb 2, 2021 •

edited

Loading

pep8speaks commented Feb 2, 2021 •

edited

Loading

brandonhoughton Feb 2, 2021

brandonhoughton Feb 2, 2021

pzhokhov Feb 3, 2021

brandonhoughton Feb 2, 2021

brandonhoughton Feb 2, 2021

brandonhoughton Feb 3, 2021

pzhokhov Feb 3, 2021

brandonhoughton Feb 3, 2021

pzhokhov Feb 4, 2021

brandonhoughton Feb 3, 2021

brandonhoughton Feb 3, 2021

brandonhoughton left a comment

pzhokhov Mar 11, 2021

		@@ -1874,9 +1874,15 @@ protected void onMissionStarted()

		// Force brightness setting
		Minecraft.getMinecraft().gameSettings.gammaSetting = (float) 2.0;

human action space + enabled GUI #449

human action space + enabled GUI #449

Conversation

pzhokhov commented Feb 2, 2021 • edited Loading

pep8speaks commented Feb 2, 2021 • edited Loading

Comment last updated at 2021-02-04 00:08:43 UTC

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brandonhoughton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pzhokhov commented Feb 2, 2021 •

edited

Loading

pep8speaks commented Feb 2, 2021 •

edited

Loading