Fix primary metric for training

[atheurer]

Currently, training metric is taken from training_params_and_metrics_global0.jsonl, getting each vale of "average_throughput" from a sample like:

{"epoch": 0, "step": 279, "rank": 0, "loss": 0.010574680753052235, "overall_throughput": 9.970008352307522, "lr": 1.5584415584415585e-07, "cuda_mem_allocated": 13.605413436889648, "cuda_malloc_retries": 0, "num_loss_counted_tokens": 6734, "batch_size": 32, "total_loss": 0.49789267778396606, "gradnorm": 1.8222006559371948, "weight_norm": 456.3302917480469, "timestamp": "2024-08-02T20:02:47.758635"}

This value, however, is not representative of the overall samples per sec, but supposedly a samples per second for GPU 0 (rank: 0 in the sample json), and this value does not match very closely (only somewhat closely) to the actual metric we want, which is "CurrSamplesPerSec", not found in this file, but in the stdout of the training process.

The stdout of the training process is filtered by the ilab-client script and written to ilab-client-stderrout.txt. This file needs to be post-processed and occurrences of this metric logged.