Normal baselines

[AkshitaB]

Llamaish setup with the following config:

--fused_loss=false
--model.init_fn=normal
--model.init_std=0.02
--model.init_cutoff_factor=3
--scheduler.warmup_min_lr=0
--scheduler.grad_clip_warmup_steps=null
--model.clip_qkv=null
--scheduler.units=steps
--scheduler.t_warmup=2000

[epwalsh]

this is the default in the config, right?

Suggested change

--global_train_batch_size=512 \

[epwalsh]

same here

Suggested change

--global_train_batch_size=1024 \

[epwalsh]

nit: I don't think there's any need to override this

Suggested change

--save_folder=runs/ \

[epwalsh]

nit: I don't think there's any need to override this

Suggested change

--save_folder=runs/ \