The following was a research project initially inspired by CLIPMatrix. However, after the release of CLIP-Actor and AvatarCLIP I've decided to just release my code since those works have extensively documented this domain.
The main difference is that this uses meshes which are much faster and less memory-intensive, additionally I explore multiple independent optimizable parameters
IMO there is still huge for improvement on this topic and hopefully this repo allows everyone to perform quick experimentation without needing large compute - Enjoy! 🎉
This project is for research purposes only
This codebase uses SMPL-X Models and the smplx pip library. By using this code you agree to the SMPL-X Model License and smplx License
This code base relies on nvdiffmodeling and in turn nvdiffrast.
Get started by testing out the features through the collabs
 |
 |
 |
 |
|---|---|---|---|
| Create a character from just a text prompt | Change an expression with text prompt | Pose with a description ( |
Create fantastic creatures like CLIPMatrix |
git clone --recurse-submodules [email protected]:NasirKhalid24/CLIPMesh-SMPLX.git
cd CLIPMesh-SMPLX
# or manually unzip so the .npz files are in the folder as shown below
unzip /models/smplx/smplx_npz.zip -d /models/smplx
pip install virtualenv
virtualenv ENV
source ENV/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
CLIPMesh-SMPLX
│
└───models
│ └───smplx
│ │ SMPLX_NEUTRAL.npz
│ │ ....
│ │ ....
To replicate the demos you can use the configs provided.
# For a single character generation
python main.py --path=configs/single.yaml
# For a single expression generation
python main.py --path=configs/expression.yaml
# For a single pose generation
python main.py --path=configs/pose.yaml
# For a CLIPMatrix type generation
python main.py --path=configs/pose.yaml
# For something else use params
python main.py \
--optim body expression texture normal \
--options face full back \
--epochs 1000 \
--gpu 0 \
--face_text "a thin 3D render of the face of a James Bond" \
--full_text "a thin 3D render of the James Bond" \
--back_text "a thin 3D render of the back of James Bond" \
--debug_log true
--log_int 250