ConvNeXt Architecture

[LukeWood]

Sayak is working on this

[sayakpaul]

I have implemented it: https://github.com/sayakpaul/ConvNeXt-TF/. Should be available here soon: https://tfhub.dev/sayakpaul/collections/convnext/1.

[bhack]

I suppose that one had an inital approach of converting the Pytorch reference impl weights.
It would be nice to see how we will handle this with the points we have at #71

[sayakpaul]

Yeah, absolutely. If you serialize the weights of the converted models you will have something similar to what's expected here.

[bhack]

More ore less. I think that some of difference in this repo could be:

• As a library we could care to have a reusable components API of the network components. Probably more generic that just to build the original Networks and its variants that we could find in the same paper.

• The training scirpts, dataset interfaces API and Markdowns to reproduce the weights from scratch

• Extra (IMHO) finte tuning

• Extra++ Having a community CI infra/github self-hosted actions on GKE to launch user contributed training/fine-tuning jobs approved by maintainers revieiws.

[sayakpaul]

I am not absolutely sure about any of these points. I am also not sure how individual contributors could run the models from scratch without any involved support from the maintainers.

For implementing architectures, there could be specific bits while generic bits could still benefit from what the library already offers. If you see what I have implemented, you'd probably notice there's not too many specific bits there.

Also, it's helpful to have examples aiding your points. For example, if you could provide an example on what you meant in your first point in the context of this issue thread, that would be super helpful.

[bhack]

I am not absolutely sure about any of these points. I am also not sure how individual contributors could run the models from scratch without any involved support from the maintainers.

We don't have this infra right now inpalce. So we cannot contribute a Github action orchestrating the training job for reproducibility.
It was just my perspective feature request (this why was tagget as extra++).

For implementing architectures, there could be specific bits while generic bits could still benefit from what the library already offers. If you see what I have implemented, you'd probably notice there's not too many specific bits there. Also, it's helpful to have examples aiding your points. For example, if you could provide an example on what you meant in your first point in the context of this issue thread, that would be super helpful.

Yes other then thinking about obivious reusable components related to a specific network like new layers, optimizers, losses and metrics that It could introduce and exposed here as API. There was also:
#59

So mainly It is just thinking as a library with the network as an e2e integration example of the new (if needed) introduced components API

[LukeWood]

Hey @sayakpaul just a heads up, we are planning to hold off on incorporating models for a little bit longer.

@qlzh727 has some great ideas on changing the structure for Keras applications a bit, and we'd like to iron those out before adding any models.

[sayakpaul]

Thanks for letting me know. Does this also mean keras.applications will be held off from accepting new models for now?

[LukeWood]

Thanks for letting me know. Does this also mean keras.applications will be held off from accepting new models for now?

I'd guess so, this should only last a month or so before we have the new sample model ready though.

[sayakpaul]

Understood. Thank you.

[Tony363]

@LukeWood Is this issue open for contributions as well?! Next to SWIN transformers, ConvNext boasts even higher performance and stats utilizing similar robustness of training datasets! I definitely want to learn more about ConvNext in kerasCV as one of my research projects would have direct benefits having ConvNext integrated into Keras!

[LukeWood]

It is not open for contributions as of now. I believe Sayak will be contributing these to Keras core. Is that correct @sayakpaul? As of now KerasCV won't be accepting models until some details are ironed out on our end.

[sayakpaul]

I am working on it, yes.

[LukeWood]

@sayakpaul we can migrate this to KerasCV when you are ready.

[sayakpaul]

Yes sure. I will start working on it very soon.

[LukeWood]

Sayak fixed this!