Unable to reproduce the domainbed accuracies

[Harsh-Sensei]

Hello!
I am facing trouble reproducing the accuracies reported in paper(https://arxiv.org/pdf/2206.04046v5.pdf). I used the default hyperparameters mentioned in the paper, and obtained the following results:

1. VLCS : 78.8(Obtained); 80.2(Reported in paper)
2. PACS : 86.9(Obtained); 88.1(Reported in paper)
3. OfficeHome : 72.7(Obtained); 74.2(Reported in paper)
   Can you please help me with this issue? Am I missing any implementation details?

[Luodian]

I am looking for detail configs but here I can provide some results for those three datasets, I can paste the exact lr and wd and layer configurations for GMoE-S/16 used to reproduce these results.

I think the difference may comes from MOE layer configuration. Especially on small datasets (VLCS PACS), they are very sensitive.

For PACS

lr: 3e-05
weight_decay: 1e-06
moe_layers=['F'] * 8 + ['S', 'F'] * 2, mlp_ratio=4., num_experts=6, drop_path_rate=0, router='top'

The results.txt is (in our previous version, the model is called SFMOE)

-------- Dataset: PACS, model selection method: training-domain validation set
Algorithm             A                     C                     P                     S                     Avg                  
SFMOE                 89.9 +/- 0.0          82.1 +/- 0.0          99.2 +/- 0.0          81.2 +/- 0.0          88.1

For VLCS

lr: 4e-05
weight_decay: 1e-06
moe_layers=['F'] * 8 + ['S', 'F'] * 2, mlp_ratio=4., num_experts=6, drop_path_rate=0, router='top'

The results are

-------- Dataset: VLCS, model selection method: training-domain validation set
Algorithm             C                     L                     S                     V                     Avg                  
SFMOE                 98.1 +/- 0.0          66.0 +/- 0.0          75.8 +/- 0.0          80.9 +/- 0.0          80.2                 

For OfficeHome

lr: 1e-05
weight_decay: 1e-06
moe_layers=['F'] * 8 + ['S', 'F'] * 2, mlp_ratio=4., num_experts=6, drop_path_rate=0.1, router='cosine_top'

The results are

-------- Dataset: OfficeHome, model selection method: training-domain validation set
Algorithm             A                     C                     P                     R                     Avg                  
GMOE_Tutel            72.6 +/- 0.0          58.9 +/- 0.0          81.3 +/- 0.0          83.9 +/- 0.0          74.2

[SiyuJi5]

Hello!
I noticed that the bias of Avg is +-0, How many seeds did you run? 1 seed or other seed? And could you please share me the setting of terra?

[Luodian]

Hello! I noticed that the bias of Avg is +-0, How many seeds did you run? 1 seed or other seed? And could you please share me the setting of terra?

Hi! I think at that time, I only ran with 1 seed to quickly provide the reproduced results. I need sometime to find the Terra's configs since it's been a long time, I will get you back as soon as possible!

[SiyuJi5]

Thank you for your reply! I have done some GMoE multi-seed experiments and often found that acc decreases as more seeds are added, could you teach me how to maintain a high acc even with multiple seeds? Or could you please tell me the specific muti-seed that you used in experiments?