Propose validation loss calculation to imporve accuracy by reducing floating-point errors

[aakashapoorv]

Propose shift from the current approach, where loss is normalized immediately for each batch:

$( \text{Average Loss} = \sum(\text{Normalized Losses}) $)

to a cumulative method followed by a single normalization step:

$( \text{Average Loss} = \frac{\text{Total Accumulated Loss}}{\text{Number of Batches}} $)

This aims to reduce floating-point errors and increase the accuracy of the reported validation loss.

Changes Made

Aspect	Old Approach	New Approach	Reason for Change
Calculation	Loss divided by number of batches before accumulating.	Accumulate all losses, then divide by batch count.	Reduces rounding errors and floating-point imprecisions.
Precision	Potential for early precision loss due to division.	Division occurs only once, preserving precision.	Enhances the reliability of loss metrics.
Error Potential	Higher, due to repeated operations on each batch.	Lower, with fewer operations on critical data.	Minimizes the accumulation of computational errors.

[karpathy]

You're not wrong... I did it mostly that way because I thought it was cognitively simpler to understand. Possibly it wasn't a great idea. I'll think through. At this scale of the project it probably doesn't actually make a difference?

[aakashapoorv]

I agree 😊, simplicity has its appeal. At this scale, it might make only a small difference, if any. I had a confusion though... if someone modifies things to scale up, then this approach may help reduce floating-point errors.