Multiple metrics in one file

[guysmoilov]

I think being able to define a list of metric type + xpath for a single output file will come in VERY handy. An additional "metric name" property would also help a lot when you are actually looking back at your metrics and trying to decide what to do about them.

As an example of why this should come in handy, check out the following repo:
https://dagshub.com/Guy/uri_nlp_ner_workshop/src/175bbd99d7dcc833049429d04334e7afe5763340

The training stage has a bunch of metric outputs, and I'm left with one of two choices:

1. Dump them all in one big json (or other) file, and treat it as one big raw metric. That means no good visualization.
2. Dump each one in its own json file, as I did in the linked repo. This allows visualization but is very cumbersome, not to mention it makes the DAG blow up and requires the dvc run command to be very long and tightly coupled to the training code.

[efiop]

Hi @guysmoilov !

Great idea! This can be solved in a pretty streight forward way by making metric: field in our dvc files to support lists. And also there should be respective CLI changes for dvc metrics and dvc run, surely. Is this something that you would be interested to submit a PR for? 🙂

Thanks for the feedback!

[guysmoilov]

@efiop Yeah, I'll start working on a PR when I have the chance

[schlichtanders]

@guysmoilov what is the status for this?

getting started with dvc, this is the first unexpected missing feature I came accross

[guysmoilov]

@schlichtanders Unfortunately, I haven't yet had time. It would be interesting to hear how you envisioned this working, before you were surprised.

[schlichtanders]

dvc already supports json structures and others in metric files

it is natural that such a json structure can have multiple metrics

{
"score1": 1.2312
"score2": -12300.122233
}

it would be great if we could seamlessly work with both metrics in this json, without overwriting the other.
(or is it already possible and I misunderstood this issue?)

[guysmoilov]

@schlichtanders AFAIK, you understood correctly.

[dmpetrov]

Now we support only JSON with multiple metrics. DVC understand the JSON structure and does proper metrics diff.

As far as I understand, this solves the issue. Closing. Please met know if something else is needed.

[efiop]

JSON and yaml 🙂