Training on Crawling @ Home

[afiaka87]

I've been running some experiments with the Crawling@Home dataset (C@H? CAH?) on DALLE. There are also some plans by others in the Discord to do so. I'll be posting some preliminary results over the next few days.

1. Deliverables/Code Snippets
2. Filtering Experiment (videogame captions)
3. 512 Dim DALLE

[afiaka87]

Deliverables/Code Snippets

I felt this code was too specific to include upstream. At any rate; if you would like to train a DALLE-pytorch on C@H; you can do some fun live filtering without a significant performance penalty because DALLE is (usually) the bottleneck.

```python
    ENABLE_METADATA = True
    FILTER_WORDS = ["image", "photo"]
    def filter_dataset(item):
        if mycap not in item:
            return False
        current_cap = item[mycap].decode('utf-8').lower()

        # only keep captions that are long enough
        if len(item[mycap]) <= 64: return False # set minimum caption length

        # only captions with the following words are kept
        has_word = False
        for word in FILTER_WORDS: # list of words/phrases to filter out
            if word in current_cap:
                has_word = True
                break
        if not has_word: return False

        if ENABLE_METADATA: # metadata is only on C@H or img2dataset datasets
            current_metadata = json.loads(item['json'].decode('utf-8')) # json metadata

            width = int(current_metadata['original_width'])
            height = int(current_metadata['original_height'])
            similarity = float(current_metadata['similarity'])


            # only square aspect ratio
            if width != height:
                return False

            if width < IMAGE_SIZE: return False # only images larger than IMAGE_SIZE

            if similarity < 0.4: return False # only images with similarity >= 0.4

        if myimg not in item:
            return False
        
        return True
    
    w_dataset = wds.WebDataset(DATASET, handler=wds.warn_and_continue)
    filtered_dataset = w_dataset.select(filter_dataset)
    ds = filtered_dataset.map_dic

[afiaka87]

Filtering Experiment (videogame captions)

Using caption-filtering; you can start training on subsets containing words right away.

For instance, filtering on various videogame related keywords finds a good deal of videogame box art.

(cherrypicked) You can see the DALLE begins to learn the difference between xbox/ps4:

A link to the full run is here:
https://wandb.ai/dalle-pytorch-replicate/CAH_VG/reports/DALLE-pytorch-on-C-H-videogame-captions---VmlldzoxMDQzNDg3?accessToken=omqfipkjpdjjif1g6kgll736hy208vjdo91erff8mtim765ldkv661r29l2rl879

[afiaka87]

C@H for 900K Samples on 512 Dim, 128 TSL, 16 Depth, 16 Heads, Rotary, Shift

https://wandb.ai/dalle-pytorch-replicate/CAH_dalle_imagenet

[afiaka87]

These were trained with no filtering. This means that aspect-ratio can be non-square (using white for the fillcolor).

[afiaka87]

An interesting experiment to try would be to include the aspect ratio in the caption (it's available in the metadata).

[afiaka87]

My attempts to improve the loss by decreasing the learning rate were perhaps misguided - I don't think the loss had plateaued enough to need that.