plots: introduce support for images

[pared]

For #6145

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here. plots: document images support dvc.org#2839

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

EDIT:

Key takeaways:

• repo.plots no longer returns vega json - now its data collected from particular recvisions - @efiop I think we might need to bump minor version with this one.
• HTML rendering is moved out of repo - it still requires repo for some logic related to templates and config, but we might be able to get rid of that too
• image support is there
• --show-vega remains - backward compatibility, in case of image target we just won't get any result

What we need to document:

• images support
• plots modify will not have an effect on image plots (not restricting it as we support dirs, which can have mixed content)

Further development:

• address "lot of immages" issue (plots: support images #6145 (comment))

[dberenbaum]

Hi @pared, what's the status on this PR? Is there anything we can do to move it along while you are out?

[pared]

Removing - even if we want to have a warning that some file does not exist, we need to try to open that file later, so that we get FileNotFound in results, for API consistiency.

[pared]

@dberenbaum I believe it should be fine as-is for now. There could be some polishing in the future, but it should be fine for now.

[pared]

cc @Suor @rogermparent @mattseddon

[pared]

Sample image plots rendering:

#!/bin/bash

set -ex

src=$(pwd)

pushd $TMPDIR

rm -rf test_workspace
mkdir test_workspace

pushd test_workspace
mkdir test_repo

pushd test_repo

git init --quiet
dvc init --quiet

echo -e "
from tensorflow.keras.callbacks import Callback
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras import utils
from collections import defaultdict
from dvclive.keras import DvcLiveCallback

results = defaultdict(list) 

class MetricsCallback(Callback):
    def on_epoch_end(self, epoch: int, logs: dict = None):
        logs = logs or {}
        for metric, value in logs.items():
            results[metric].append(value)
        results['epoch'].append(epoch)


def load_data():
    (x_train, y_train), (x_test, y_test) = mnist.load_data()

    x_train = x_train.reshape(60000, 784)
    x_test = x_test.reshape(10000, 784)
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    x_train /= 255
    x_test /= 255

    classes = 10
    y_train = utils.to_categorical(y_train, classes)
    y_test = utils.to_categorical(y_test, classes)
    return (x_train, y_train), (x_test, y_test)


def get_model():
    model = Sequential()

    model.add(Dense(256, input_dim=784))
    model.add(Activation('sigmoid'))
    model.add(Dense(10, input_dim=256))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy',
    metrics=['accuracy'], optimizer='sgd')
    return model


(x_train, y_train), (x_test, y_test) = load_data()
model = get_model()

model.fit(x_train,
          y_train,
          validation_data=(x_test, y_test),
          batch_size=128,
          epochs=5,
          callbacks=[MetricsCallback(), DvcLiveCallback()])

def write_results(results, d):
    import matplotlib
    import matplotlib.pyplot as plt
    import os
    os.makedirs(d)
    x = results.pop('epoch')
    for key, y in results.items():
        plt.figure()
        plt.plot(x, y)
        plt.ylabel(key)
        plt.savefig(os.path.join(d, key + '.png'))
    

write_results(results, 'plots')
" >> train.py
dvc run -d train.py --plots plots --plots dvclive -n train python train.py
git add -A
git commit -am "init"

sed -i "" "s/sgd/adam/g" train.py
dvc repro train
dvc plots diff

[efiop]

This change is isolated, so let's merge and address things in follow-ups (docs too).

[daavoo]

@pared What is the reason for this change?

Should we update on DVCLive that {path}.html is no longer the default html output?

[dberenbaum]

Good catch! To support images, the image files must be stored alongside the html, so the output is now a directory instead of a single file.

I would probably opt to do something like dvcvlive/dvc_plots rather than append them with underscores. Any other ideas are welcome.

[daavoo]

I have no strong preference but, if we go with dvcvlive/dvc_plots, the .html output will now be handled like the .tsv plots (i.e. based onlive's cache option).

Previously, the .html was not tracked by DVC and it was up to the user whether to track it with Git/DVC or just ignore.

[dberenbaum]

Ah, that's probably why @pared did it this way. Maybe just dvclive_plots then?

Another option would be to reorganize dvclive outputs so that the tsv files are under a subdirectory like dvclive/logs or dvclive/tsv and the html is another subdirectory like dvclive/plots or dvclive/html.

[daavoo]

I kind of like dvclive/tsv & dvclive/html.

Honestly, the word plots for referring to the HTML sounds kind of confusing to me because the .tsv files are also dvc plots in a sense.

Moving the outputs to subdirectories would require some changes in DVC core but it feels like a good direction. I might consider also moving the summary .json to dvclive/summary.json.

[daavoo]

Maybe we can continue the discussion in iterative/dvclive#148 as this is a comment in a closed P.R.

[jorgeorpinel]

Should be /plots

[jorgeorpinel]

I think you can probably inject HTML code here:

>>> os.path.relpath('#"> <script...')
'#"> <script...'