This is the official implementation of the paper Semantic Segmentation in Art Paintings. EuroGraphics 2022.
Nadav Cohen, Yael Newman, Ariel Shamir.
[Arxiv] [EG]
We are organizing our code so it is understandable and easy to use. Until then we invite you to read our paper.
12/29/2022
News: requirements.txt file, detailed descriptions on how to download the relevant data, train AdaIN style transfer and use it for creating pseudo paintings. More descriptions of how to train and test the models from scratch.
Next Update: inference code for paintings out of the DRAM dataset.
06/05/2022
News: DRAM dataset is now available in our project page (seem more info below). Also available are our trained checkpoints, style weights file for DRAM test set
and scripts for combining PascalVOC12 and SBD datasets and filtering the combined dataset to include the 12 classes used in the paper.
Next Update: Final scripts for stylizing the dataset with a specific art style and clear instructions for train and test scripts (requirements.txt file, and command line calls). Hopfully next update will be the final one (besides bug fixes if needed).
04/09/2022
News: Uploaded train and test scripts, and everything needed to run them.
Next Update: Link to DRAM dataset, trained checkpoints and scripts for PascalVOC2012+SBD preprocessing.
Our project page is officially up: [Project Page]
Here you can find the download links for DRAM dataset and some more info about the paper.
Note: The website offers two options: RAW and Processed. The processed version is the dataset in the state we used it before training. RAW holds the dataset
as gathered from original sources without any process.
You can find our trained checkpoints in the following link: [checkpoints.zip]
The .zip file holds checkpoints for Step1 and Step2 for each of the art movements used for training: Realism, Impressionism, Post-Impressionism and Expressionism.
To compose the networks as described in the paper unzip the checkpoints.zip file to the project folder and then run ./test_multi_weighted.py together with the style weights file (found in the style weights folder).
The multi-test config should already be set to use the checkpoints and style weights.
- Install Python 3.8.5
- Create a pip virtual env using python -m venv <path_to_env>
- run 'pip install -r requirements.txt' from the project folder.
- activate pip env: 'source <path_to_env>/bin/activate.
Note: If you are only interested in testing the code, you can skip training pytorch-AdaIN and creating pseudo paintings as you need only the DRAM dataset.
After running the all instructions, your folder tree should look like this:
├── data
│ ├── pascal ---> original pascal dataset
│ ├── sbd ---> original sbd dataset
│ ├── pascal_sbd ---> pascal combined with sbd and filtered (same as paper)
│ ├── pascal_sbd_styled_realism ---> realism pseudo-paintings
│ ├── pascal_sbd_styled_impressionism ---> impressionism pseuso-paintings
│ ├── pascal_sbd_styled_post_impressionism ---> post-impressionism pseudo-paintings
│ ├── pascal_sbd_styled_expressionism ---> expressionism pseudo paintings
│ └── DRAM_500 ---> DRAM art paintings dataset
│
├── SemanticSegmentationInArtPaintings ---> project main directory
├── pytorch-AdaIN ---> AdaIN style transfer network code
└── stylize-datasets ---> dataset stylization repository for pseudo-paintings creation-
Download Data
- Create data dir as described above (located in the same folder as your ProjectDir)
- Download PascalVoc12 dataset link and unzip it in data/pascal.
- Download SBD dataset link and unzip it in data/sbd.
- combine and filter pascal and sbd to one dataset with 12 classes as used in the paper:
from main project dir: python utils/combine_pascal_sbd.py python utils/create_filtered_list.py- Download DRAM dataset link and unzip it to data. (Rename folder from DRAM_processed to DRAM_500, sorry about that...)
-
Train AdaIN style transfer networks:
- clone and install pytorch-AdaIN
- create combined DRAM data folder compatible with pytorch-AdaIN interface:
from main project dir: python utils/organize_dram_for_adain_train.pyThis will create a folder in your data dir: DRAM_for_Adain. This folders hold all of the movements images in a single folder for training the AdaIN network. Feel free to remove 'DRAM_for_Adain' after the next step, as you will not need it further.
- Train AdaIN weights using the following command from pytorch-AdaIN project dir:
from pytorch-AdaIN: CUDA_VISIBLE_DEVICES=<gpu_id> python train.py --content_dir ../data/pascal_sbd/images --style_dir ../data/DRAM_for_Adain -
Create a Psuedo-Paintings dataset for each movement.
- clone and install stylize-datasets
- copy and rename the decoder '.pth' weight file from pytorch-Adain/experiments/decoder2_iter_160000.pth.tar -> stylize-datasets/models/decoder.pth
- create a data folder for each art movement compatible with stylize_datasets interface:
from main project dir: python utils/organize_dram_for_stylization.pyThis will create a folder for each art movement in your data dir: DRAM_for_stylization_movement same as we did before for training AdaIN.
This time DO NOT remove the folders.
- for each created folder, run stylize-datasets/stylize.py script from its project page to create pseudo painting dataset as follows:
from stylize-datasets: python stylize.py --content-dir ../data/pascal_sbd/images --style-dir ../data/DRAM_for_stylization_realism --num-styles 1 --content-size 0 --style-size 0 --alpha 0.5 --output-dir ../data/pascal_sbd_styled_realism python stylize.py --content-dir ../data/pascal_sbd/images --style-dir ../data/DRAM_for_stylization_impressionism --num-styles 1 --content-size 0 --style-size 0 --alpha 0.5 --output-dir ../data/pascal_sbd_styled_impressionism python stylize.py --content-dir ../data/pascal_sbd/images --style-dir ../data/DRAM_for_stylization_post_impressionism --num-styles 1 --content-size 0 --style-size 0 --alpha 0.5 --output-dir ../data/pascal_sbd_styled_post_impressionism python stylize.py --content-dir ../data/pascal_sbd/images --style-dir ../data/DRAM_for_stylization_expressionism --num-styles 1 --content-size 0 --style-size 0 --alpha 0.5 --output-dir ../data/pascal_sbd_styled_expressionism- run the following script from the main project dir to organize the pseudo painting folders for the Semantic Segmentation in Art Paintings training:
from main project dir: python utils/organize_pseudo_painting_dirs.py
After creating all the data above you can train the networks yourself. Each networks trains with two steps, training the expressionism art movement for example will be as follows:
python train_src.py -cfg configs/train/deeplabv2_r101_src_pascal_expressionism.yaml OUTPUT_DIR results/step1/pseudo_expressionism # Step 1
python train_adv.py -cfg configs/train/deeplabv2_r101_adv_expressionism.yaml OUTPUT_DIR results/step2/expressionism resume results/step1/pseudo_expressionism/model_iter020000.pth # Step 2
You can see all training command in train.sh.
you can test a single network or combine the networks based on the GRAM matrices of the test images (As described in paper). The test config files which are found in configs/test should be set according to the test you want to run.
Single Network Test:
python test_single.py -cfg configs/test/deeplabv2_r101_test_single.yaml
Multi-Network Test:
python test_multi_weighted.py -cfg /home/students/nadav/FADA_ArtSeg/configs/test/deeplabv2_r101_test_multi_weighted.yaml
Some of our code is adapted from FADA and Permuted AdaIN. We thank them for their excellent research and for sharing their repositories. We also thank pytorch-AdaIN and stylize-datasets for their excellent repositories which are used in our data creation process.
If you found this project helpful in your research, please consider citing our paper.
@article {ArtSeg-22,
journal = {Computer Graphics Forum},
title = {{Semantic Segmentation in Art Paintings}},
author = {Cohen, Nadav and Newman, Yael and Shamir, Ariel},
year = {2022},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
ISSN = {1467-8659},
DOI = {10.1111/cgf.14473}
}