How does model.fit () calculate loss and acc ? Documentation will be helpful.

[whatdhack]

I think it uses functions like binary_entropy , binary_accuracy etc. Is any smoothing used ? Some description in the documentation will be good ( pointers to the exact code will also help).

[raymond-yuan]

Loss and accuracy are calculated as you train, according to the loss and metrics specified in compiling the model. Before you train, you must compile your model to configure the learning process. This allows you to specify the optimizer, loss function, and metrics, which in turn are how the model fit function knows what loss function to use, what metrics to keep track of, etc. The loss function (like binary cross entropy) documentation can be found here and the metrics (like accuracy) documentation can be found here

In summary, in order for you to train (and for fit to calculate loss and acc):

1. Instantiate model

model = Sequential()
...

2. Compile the model. This is where the functions like binary cross entropy are specified for the model to call fit. The built in loss functions are documented and implemented here.

# For a binary classification problem
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

3. Train by calling fit

model.fit(data, one_hot_labels, epochs=10, batch_size=32)

[whatdhack]

Thanks , that is a little bit more information, but not exactly the answer to what I was asking. My question is what exactly is the formula used for calculating loss and accuracy per batch ( e.g. the numbers that get displayed during model.fit() ) . If you know where that is exactly done in the code , that answer should be right there.

[raymond-yuan]

Yes, like I linked, the formulas for calculating loss per batch can be found here. The exact implementation is backend specific. For example if you were using tensorflow look here

[whatdhack]

Thanks again - I have seen those code by the way. Let me try again. The following is an output from a model.fit () session .

5/125 [>.............................] - ETA: 4:35 - loss: 1.1342 - acc: 0.5250

Which exact variable holds the loss (e.g. 1.1342) and accuracy (e.g. 0.5250) ?

[raymond-yuan]

I believe that the loss is stored here; similarly the accuracy is handled by the handle_metrics function. However, you shouldn't be working with these variables directly, if you want to calculate the loss and accuracy for your model given some data use model.evaluate

[whatdhack]

Thanks again. Basically my question is what mathematically the loss and accuracy displayed during model.fit() are. Is it averaged over the batch , or some other function is used to calculate the loss/acc that gets displayed on the screen. The answer is still not there, hence need to keep looking till the answer is found and document it.

[raymond-yuan]

For training loss, keras does a running average over the batches. For validation loss, a conventional average over all the batches in validation data is performed. The training accuracy is the average of the accuracy values for each batch of training data during training.

[whatdhack]

Good. Is the running average over just the batch or there is carryover from previous batch ? Would be great to see the code for that.

[raymond-yuan]

The code is here, BaseLogger is applied to every Keras model and will update things like loss and acc. The training loss is carried over from the previous batch: it's the average of the losses over each batch of training data.

[whatdhack]

You are probably close. If the code ( see below) indeed is responsible for the fit() per batch loss/acc output, then the mystery is probably is how 'v' got into the logs list before entering the on_batch_end() function . This function primarily seems to be for keeping track variables for on_epoch_end () calculations.

def on_batch_end(self, batch, logs=None):

.
.
.
for k, v in logs.items():
if k in self.stateful_metrics:
self.totals[k] = v < ----
else:
if k in self.totals:
self.totals[k] += v * batch_size
else:
self.totals[k] = v * batch_size

[raymond-yuan]

'v' gets added into the batch logs here. outs is the output of the train function, f, which is built when you call fit on your model, which will output your loss and metrics from here given some inputs. An example of the log list looks like this (if you were to add a print(logs) in the on_batch_end method)

{'loss': 1.8399812, 'batch': 2, 'size': 32, 'acc': 0.5625}
{'loss': 1.6450999, 'batch': 3, 'size': 32, 'acc': 0.53125}
{'loss': 1.6244895, 'batch': 4, 'size': 32, 'acc': 0.40625}
{'loss': 1.4333215, 'batch': 5, 'size': 32, 'acc': 0.5625}
{'loss': 1.1506481, 'batch': 6, 'size': 32, 'acc': 0.75}
{'loss': 1.0616728, 'batch': 7, 'size': 32, 'acc': 0.65625}
{'loss': 1.313815, 'batch': 8, 'size': 32, 'acc': 0.46875}
{'loss': 1.2656202, 'batch': 9, 'size': 32, 'acc': 0.5625}
{'loss': 1.2441672, 'batch': 10, 'size': 32, 'acc': 0.5625}
{'loss': 0.99731344, 'batch': 11, 'size': 32, 'acc': 0.65625}
{'loss': 1.0144608, 'batch': 12, 'size': 32, 'acc': 0.71875}
{'loss': 1.2803181, 'batch': 13, 'size': 32, 'acc': 0.4375}
{'loss': 1.2766386, 'batch': 14, 'size': 32, 'acc': 0.5}
{'loss': 0.87459761, 'batch': 15, 'size': 32, 'acc': 0.6875}

[whatdhack]

That is great. So, it is still not clear if the per batch loss/acc is the average over just that batch or something else.

[kaiche12]

Sorry, how do Keras calculate loss (e.g mse) when doing predict_on_batch. For example you have 10 batches, the model normally will return 10 losses

[DayChan]

@raymond-yuan When we choose "acc" to calculate accuracy, how does keras choose the calculating function? I found that there are several accuracy functions.

[chrismaliszewski]

@DayChan, check a comment to this page https://stackoverflow.com/a/59637143/5536388.

[NEGU93]

After having anxiety by reading this. I believe the conclusion is:

• Loss: displays the loss of the current batch
• Validation loss: displays the loss of all the validation batches
• Accuracy and validation accuracy: displays the acc of all the batches (train and val respectively).

Did I understand correctly?

[chrismaliszewski]

All good but the last point training part. I'll sum this up again + extras:

• if acc/accuracy metric is specified, TF automatically chooses it based on the loss function (LF), it can either be tf.keras.metrics.BinaryAccuracy, tf.keras.metrics.CategoricalAccuracy or tf.keras.metrics.SparseCategoricalAccuracy and it's hidden under the name accuracy,
• when a metric is calculated, it usually has two public variables, total and count; the first calculates the total sum of results of the metric, whereas count, how many times we calculated the results; the displayed metric value comes from dividing one by another,
• (the following I'm not 100% sure) most of the LF inherit from MeanWrapper; if it's not, i.e. custom LF, it's wrapped around it automatically to calculate mean results after each batch in the same way metrics are calculated (see the previous point),
• while training: all metrics (and LF) are calculated on batch_end and displayed immediately,
• while validation: all metrics (and LF) are calculated on batch_end but displayed at the very end after averaging.

[NEGU93]

I tried this line of code:

evaluate = model.evaluate(train_ds)
print(evaluate)

And got:

148/148 [==============================] - 23s 122ms/step - loss: 0.2275 - accuracy: 0.8943 
[0.23347871005535126, 0.8992502093315125] 

See how loss and ACC are not the same. We have two values here, the 'computation' value (in this case 0.8943) and the 'returned' value, in this case 0.8992.
Therefore, I believe the 'computation' value is a per batch value only, meaning the accuracy of the specific batch. And the returned value of the function is the actual total / count.
That is how I would explain the difference here.

Another option (which I think less likely) is that the return value is total_per_sample / (count * batch_size). and then the computation metric is the total / count

[chrismaliszewski]

@NEGU93, I followed the code again, here is how it works:

1. fit() runs.
2. self.train_function runs (it's a bit different for evaluation, prediction, test, training), it returns logs.
3. method compute_metrics is called
4. it calls compiled_metrics.update_state() which calls update_state for each metric
5. as I described in the previous post, most of the metrics accumulate their results by creating total and count variables but they return the updated value
6. (we're back to the place from 3.) after the update, there is a loop across all metrics and their results are being returned with result method of a metric, they are the logs
7. a callback on batch_end is called that updates progbar, method _batch_update_progbar, with the logs values.

That's how the code works.

You can check that what I say is true by running this code for 2 "batches"/examples: https://colab.research.google.com/drive/12gTrW-k0TntAawVjXv-tVLp4lmM5lFXx?usp=sharing. I created this in the past to simulate how metrics and loss functions work in Numpy.

I'm not sure what causes the difference in your code.