Questions about validation set

[yueyang130]

hi @mees , thanks for your great work!
I notice in each scene, there are separate training and validation set. I have some questions:

1. What are the validation data for? Is it for the validation loss to know if your training is overfitting?

2. What's the ratio of training and validation set?

3. Are the scene rendering config files in train and validation in a scene identical?

4. Now I download task_ABC_D.zip and task_D_D.zip. Now I want to train on ABCD, can I merge the train set of ABC_D and D_D to get the train set for ABCD? Is the merge one identical with ABCD_D by downloading?

I would appreciate your help!

[yueyang130]

One more question:

If I want to test the generalization ability of my policy such as testing on objects with unseen colors or shapes, can I modify the calvin code to achieve it?

[lukashermann]

1. Yes, we run a validation step after every training step. We separated the data into different folders to make sure the sliding window approach never samples a frame in both the training and validation step. The data comes from the same teleoperation session, but does not overlap.
2. validation size is 10% of the training size (apart from the debug dataset, where the validation is exactly the same as the training data)
3. yes, they were rendered with exactly the same config
4. Simple merging wouldn't do the job, so I would advice against attempting to do that (you would have to mess with file indices etc)
5. Yes, have a look at the calvin_env, for example, you could change the config file of the scene. By default, the config file that was used during rendering is loaded at the beginning of the evaluation (which is located in your dataset/<validation OR training>/.hydra/merged_config.yaml). You can modify the code to load another scene config during evaluation.
   Look at these parts of the code: here, here and here.