dm_nfnet_f4 not working on rtx 3060ti

[mobassir94]

i tried to use dm_nfnet_f0 and with batch size 8 i could use 456 image size and on rtx 3060ti,however i tried to train dm_nfnet_f4 today on rtx 3060ti and even with batch size 1 and image size 32(i repeat, 32) i get OOM,why is this happeningÉ is there any memory leak or something? or i won't be able to use dm_nfnets>0 on rtx 3060ti which has 8 gb vram?trying dm_nfnet_f1,2,3 now

[rwightman]

@mobassir94 they're huge models, why would you expect anything different? The F0 is bigger than a ResNet-200. F4 is 320M params and almost 5x a ResNet-200! and running by default at a res > 2x the usual ResNet which is 4x the number of pixels (activations).

[mobassir94]

i can use f0,1,2(maximum) when using rtx 3060ti,but they are performing well with small batch size like 4, thank you for sharing those weight files,i will see if i can use b3 on torch xla tpu,thanks

[rwightman]

The one benefit of these models is that without BatchNorm, they should work just as well at really small batch sizes. But they are really really big, can't stress that enough. The F5-F6 is insane. I've filtered out F4 and higher from the CPU based testing because they kill the GitHub CI runner. I might need to add F3 because it's still failing sometimes. Feel free to post any success using the larger ones in torch XLA. I'm sure some others will try.

[mobassir94]

thank you @rwightman
i would like to ask you one thing if you don't mind, if we use small batch size n f0,f1 like models then will it take more epochs to converge compared to large batch size with or without batchnorm layer like models? just curious