Collecting frame error, label error, edit distance?

[albertz]

In RETURNN, the losses actually did two things:

• Calculate a differentiable loss (sometimes also called "score", although this is misleading, as we want to minimize it).
• Optionally, calculate some non-differentiable error. Usually frame error, e.g. comparing arg max to the target class which is either correct or wrong. In case of sequence-level losses like CTC, this would often calculate the edit distance.

The error was mostly just used as running statistics, although it was also potentially used for learning rate scheduling.

We don't really have that now. But we need it, to cover similar behavior or at least reporting as before.

We could just use the normal loss mechanism (mark_as_loss or maybe sth different #56) and also handle frame error, edit distance or other things this way. This is maybe already the solution. Maybe it should set scale=0, although for non-differentiable losses, this is anyway not needed.

In any case, there should be some functions to calculate this easily (error, edit_distance, etc), maybe as part of #38.

[albertz]

There are potentially also other statistics. In TF, you would often just call tf.summary.

Maybe we should generalize this, to cover such errors as well? But this needs to be on RETURNN side then first. Also the learning rate scheduling logic needs to be able to access those things.

Or such tf.summary wrapper is just a separate thing. You could do both mark_as_loss and summary or so.

[albertz]

One thing that RETURNN does additional on losses (and errors) is the accumulation over the epoch and the proper averaging (taking e.g. seq length into account).

[albertz]

We have edit_distance now.

Calculating frame-error is trivial.

We also have the mask_as_loss as_error option.

I think for now this is fine.