How to know the training should stop?

[hrbspider]

From my experience in classification training, the val data is used to decide when to stop during training on train data. But in cycleGAN I haven't found this mechanics.
So how to decide to stop training?

[lishen]

People have been throwing a term called "Nash equilibrium" around, which sounds really fancy but I haven't seen a quantitative definition.

[junyanz]

How to automatically evaluate the results is an open research problem for GANs research. In the case of CycleGAN, we may be able to evaluate the results with validation data when a few paired ground truth data are available. See the FCN score evaluation for cityscape label -> photo in the paper (Section 5.1.1) for an example. But for many creative applications, the results are subjective and hard to evaluate without humans in the loop.

[YilinLiu97]

Hi @junyanz, can we know that by seeing if D outputs 0.5 or is that too ideal in practice? Thanks

[junyanz]

I think it's too ideal. It rarely happens in practice.

[paviddavid]

@junyanz I am facing the same issue, when is the right point to stop the training. When do you stop the CycleGAN training? Do you have any criteria? Do you know other approaches/papers when a CycleGAN training should be stopped?

I trained your model for 45 epochs and I am still not sure if it is enough or not. Should I look at the loss curves and stop when the loss is (more or less) converged against a certain value? Should I focus more on Generator or Discriminator loss?
There is a lot to consider and it is probably still a large research topic...

[junyanz]

If you have a validation set and a task-specific metric, you can pick up the best epoch based on the performance on the validation set. Often times, we don't have that. For many of our applications in the paper and for a middle-size dataset (e.g., a few hundreds of images or 1000 images), we just fix the number of epochs as 200, which might not be the optimal epoch. But if you have a large dataset, you may want to reduce the number of epochs.

[paviddavid]

@junyanz A validation set would be quiet perfect. However, I do not see how to validate these images, based on which attribute? The semantic preservation, the optical impression, based on pixel values (some kind of similarity measurement) or the ability of an object detector... There are many different possibilities and it is not clear which is the best point to stop.

I am working with 17k images in domain A and 20k images in domain B. There are a lot of images. Until now, I trained over ~70 epochs and it took around 7 days.

[junyanz]

70 epochs might be enough. The metric is application-specific. As you have mentioned, there is no single metric that works for all the datasets and tasks.