Is there a timing plan for the support for ALFRED, Dialfred, and Tidy Task?

[shengyin1224]

Hello, I have read your work and it's great! It seems that HELPER only run in the teach dataset, can I make it run in other datasets which only include nl instruction and final task goal like ALFRED?

[shengyin1224]

And when I run the following script:

python main.py \
  --mode teach_eval_tfd \
  --split valid_unseen \
  --gpt_embedding_dir ./data/gpt_embeddings \
  --teach_data_dir ./data/teach-dataset \
  --server_port 99 \
  --episode_in_try_except \
  --use_llm_search \
  --use_constraint_check \
  --run_error_correction_llm \
  --zoedepth_checkpoint ./checkpoints/ZOEDEPTH-model-00015000.pth \
  --solq_checkpoint ./checkpoints/SOLQ-model-00023000.pth \
  --set_name HELPER_teach_tfd_validunseen\
  --use_openai\

The program will be stopped in the line 2444 of teach/src/teach/simulators/simulator_THOR.py:

self.controller.step(
            "AddThirdPartyCamera", rotation=dict(x=0, y=0, z=90), position=dict(x=-1.0, z=-2.0, y=1.0), fieldOfView=90
        )

Is there a solution to solve it ?

[Gabesarch]

Hi, we currently do not have a timeframe for releasing the other tasks, but hope to get to it in the next few weeks.

Have you verified that your x display is configured properly? For example, are you able to run first steps here?

[shengyin1224]

yes, i can run the first step.

[shengyin1224]

thank you for reply! I want to know if I can change the TeachDataset to my own dataset and do not change other architecture of HELPER?

[Gabesarch]

Can you please provide your PC config and steps you took to reproduce the issue?

Regarding the dataset, could you specify the format of your data? For example, is it in the same format as the TEACh tfd or edh instances? If that is the case, you could read in your custom files as a tfd or edh instance.

[shengyin1224]

I ran the program in

Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Linux ubuntu 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

And I just follow your instructions in Readme to install and then run the script above mentioned.

[shengyin1224]

Regarding the dataset, could you specify the format of your data? For example, is it in the same format as the TEACh tfd or edh instances? If that is the case, you could read in your custom files as a tfd or edh instance.

The data I want to use includes just the instruction, scene_name and the final state for evaluation like below:

{"instruction": "Break the Mirror with the TennisRacket.", "scene_name": "FloorPlan302", "final_state": [{"objectType": "Mirror", "isBroken": true}, {"objectType": "TennisRacket", "isBroken": false}]}

If I can apply this data into HELPER ?

[Gabesarch]

You would need to reformat the data into the tfd instance format that TEACh provides. See their repo for more details. Here's an example of a part of one of their json files:

{"dialog_history": [["Commander", "prepare coffee"], ["Commander", "mug is in the oven"], ["Driver", "coffee is ready"], ["Commander", "knife is in wash basin"], ["Commander", "lettuce beside stove"], ["Commander", "make a slice"], ["Commander", "tomato is inside the bin infront of fridge."], ["Commander", "make a slice"], ["Commander", "potato is on the stove, make a slice."], ["Driver", "then"], ["Commander", "potato slice please"], ["Driver", "then"], ["Commander", "plate is inside the fridge, place those 3 slices onto it."], ["Commander", "2 potato, 1 tomato, 1 lettuce"], ["Commander", "done"], ["Commander", "thanks"]], "driver_action_history": [], "driver_image_history": [], "driver_actions_future": [{"action_id": 8, "action_idx": 8, "obj_interaction_action": 0, "action_name": "Pan Left", "time_start": 88.71959018707275, "oid": null, "x": null, "y": null}, {"action_id": 4, "action_idx": 4, "obj_interaction_action": 0, "action_name": "Turn Left", "time_start": 94.37360501289368, "oid": null, "x": null, "y": null}, {"action_id": 6, "action_idx": 6, "obj_interaction_action": 0, "action_name": "Look Up", "time_start": 113.44104933738708, "oid": null, "x": null, "y": null}, {"action_id": 205, "action_idx": 20, "obj_interaction_action": 1, "action_name": "ToggleOff", "time_start": 161.29252243041992, "oid": "Microwave|+01.04|+01.68|-01.30", "x": 0.20038167938931298, "y": 0.2538167938931298},...

Once you have data in this format, you can input it as an argument to --teach_data_dir.

We have also recently added ALFRED and the Tidy Task to our codebase. You can find more details on these updates in the ALFRED branch and the Tidy Task branch. Support for Dialfred will be available soon.