Unknown error when submitting to WOSAC

[syl020119]

Hello, I noticed the update in April and added those mandatory options. However, it still shows

Something went wrong with this submission.
Unknown error, please file a new issue at https://github.com/waymo-research/waymo-open-dataset/issues/new and specify submission_key=user_id: "[email protected]" submission_id: "1727957536034000".

Can you kindly help me find the reason why I cannot submit successfully? Thanks. The account is [email protected].

[jlambert-waymo]

Hi @syl020119, thanks for your submission.

My best guess is that since your uploaded file is in ".tar" format, instead of ".tar.gz" format, the server cannot process it.
I see your uploaded file 1727957536034000.tar. Likely you could just run something gzip -9 < 1727957536034000.tar > 1727957536034000.tar.gz.

We give a bit more info about the submission format here: https://waymo.com/open/challenges/2024/sim-agents/, e.g. "Once you have the 150 files, compress them as a single .tar.gz archive and upload the single file." Some example code to do this can be found here.

[syl020119]

Hi, thanks for your advice. I fix this and submit again. This time, it shows

Missing 44520 scenario from the submission. Some examples: ['f1873765058ec1bd', '25eb5b22f6534a19', '6b8ece90cc959527', '8c013281003088d8', 'd0cbbf9a5f59f48']

For my submission, I just try the demo you published on the waymax simulator, and just use one of training data, training_tfexample.tfrecord-00999-of-01000, is this OK? Why is this problem?
Thanks for your kindness.

[syl020119]

@jlambert-waymo
Hi, can you help me?

[jlambert-waymo]

Hi @syl020119, no problem, happy to help.

We offer two evaluation servers -- one for validation set submissions, and one for test set submissions. A submission is only valid if it features results for all scenarios within the respective dataset split. (Otherwise, the evaluation result is undefined).

It looks like you are submitting to the "test set" server, so uploading train split scenarios will always throw an error (whether 1 train shard or all train shards). Please re-submit using all scenarios from the test split.

[syl020119]

Hi, @jlambert-waymo
it means I need to submit using all tfrecord
(from testing_tfexample_tfrecord-00000-of-00150 to testing_tfexample_tfrecord-00000-of-00149)? Or I can just catch one of them but I need to catch all scenarios in this tfrecord.
Thanks. Have a good weekend!

[jlambert-waymo]

Hi @syl020119, so you need to submit simulated rollouts for ~44,520 test scenarios. These scenarios are distributed in two formats -- "TFExample" and "Scenario" format. Your test split submission must contain all scenarios in the test split (from all shards, if you use the TFExample data source).

Please see Table 5 on Page 21 of https://arxiv.org/pdf/2305.12032 for the counts.

[syl020119]

Thanks, I will read this carefully. Happy weekend.