From 2a6eeb8c2d0cdbdf91b2e8465baaa4246f9e390a Mon Sep 17 00:00:00 2001 From: dberenbaum Date: Fri, 19 Mar 2021 12:40:57 -0400 Subject: [PATCH 1/6] initial draft for exp-show-target --- proposals/exp-show-target.md | 232 +++++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 proposals/exp-show-target.md diff --git a/proposals/exp-show-target.md b/proposals/exp-show-target.md new file mode 100644 index 0000000..3e35ad0 --- /dev/null +++ b/proposals/exp-show-target.md @@ -0,0 +1,232 @@ +- Enhancement Proposal PR: (leave this empty) +- Contributors: dberenbaum, (add your Github handle) + +# Summary + +The `dvc exp show` command shows a comparison of the results of different +experiments along with the parameters and metrics values for each experiment. +The output currently includes all values that are defined in `params.yaml` or +any other referenced parameters files in any DVC stage. Output also includes all +metrics in any referenced metrics files in any DVC stage. This proposal allows +users to see only parameters and metrics relevant to specified pipeline stages. + +See https://github.com/iterative/dvc/issues/5451 for more background. + +# Motivation + +`dvc exp show` is the main view by which users may compare experiments, and it's +becoming useful for other purposes, like comparing multiple commits (see +https://discord.com/channels/485586884165107732/563406153334128681/822464119906369606). + +It's also likely the busiest DVC output: + +```console +┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━ +┃ Experiment ┃ Created ┃ auc ┃ prepare.split ┃ prepare.seed ┃ featurize.max_features ┃ featurize.ngrams ┃ train.seed ┃ train.n_estimators ┃ topic_modeling.seed ┃ top +┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━ +│ workspace │ - │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ 20170428 │ 10 +│ topic_modeling │ 04:41 PM │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ 20170428 │ 10 +│ master │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - +│ └── exp-56d11 │ 04:40 PM │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ 20170428 │ 10 +│ 263732a │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ - │ 100 │ - │ - +│ b297969 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ - │ 100 │ - │ - +│ 4edd930 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ - │ 100 │ - │ - +│ 72ed9cd │ Nov 15, 2020 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 50 │ - │ - +│ ├── exp-47dfa │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - +│ └── exp-44136 │ Feb 17, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 50 │ - │ - +│ f8e9d93 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 50 │ - │ - +``` + +This table view is from a simplified example and is still missing several +columns, so this view can become cluttered quickly. Eliminating irrelevant +information is a priority. + +There are several ways in which the included parameters might be irrelevant: +* Parameters files may include values that aren't tracked by DVC at all (for + example, debug level). +* The parameters may be used in a different pipeline branch (for example, the + `topic_modeling` stage is in a separate branch of the pipeline from the + `evaluate` stage, which produces the relevant experiment metrics). +* The parameters may be used in a downstream pipeline branch (for example, the + post-processing `combine` stage parameters aren't relevant to experiment + evaluation). + +```console + +-------------------+ + | data/data.xml.dvc | + +-------------------+ + * + * + * + +---------+ + | prepare | + +---------+ + * + * + * + +-----------+ + | featurize | + **+-----------+** + **** * **** + **** * *** + ** * *** ++-------+ * ** +| train | ** * ++-------+ * * + ** ** * + ** ** * + * * * + +----------+ +----------------+ + | evaluate | | topic_modeling | + +----------+ +----------------+ + *** **** + * *** + ** ** + +---------+ + | combine | + +---------+ +``` + +There are no apparent reasons that users want to see untracked parameters in +`dvc exp show`. Unlike `dvc params diff` and `dvc metrics diff`, `dvc exp show` +is not applicable to arbitrary files. Likewise, there are no apparent reasons +that users want to see parameters or metrics from irrelevant stages of their +pipelines. Users could keep untracked parameters in other files if they don't +want them shown in `dvc exp show`, and they could exclude parameters or metrics +from irrelevant stages, but DVC already has enough information to understand +that these should not be shown. + +# Detailed design + +`dvc exp show` could take stages as (positional?) arguments, similar to `dvc exp +run`. The help message could be similar: + +```console +positional arguments: +stages Stages for which to print experiment info. 'dvc.yaml' +by default. +``` + +If no stages are passed, parameters and metrics defined in any stage would be +shown. This differs from current behavior only because untracked parameters +would be excluded. + +If stages are passed, parameters and metrics defined anywhere in the pipeline +for those stages would be shown. + +Using the above examples, `dvc exp show evaluate` would be equivalent to using +the `--exclude-params` and `exclude-metrics` options to exclude all parameters +and metrics for the `topic_modeling` and `combine` stages: + +```console +┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ prepare.split ┃ prepare.seed ┃ +featurize.max_features ┃ featurize.ngrams ┃ train.seed ┃ train.n_estimators ┃ +train.random_state ┃ … ┃ +┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━┩ +│ workspace │ - │ 0.51625 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ topic_modeling │ Mar 18, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ └── exp-44136 │ 10:29 AM │ 0.51625 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ master │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ └── exp-56d11 │ Mar 18, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ 263732a │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 +│ 2 │ - │ 100 │ 20170428 │ - │ +│ b297969 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 +│ 2 │ - │ 100 │ 20170428 │ - │ +│ 4edd930 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 +│ 2 │ - │ 100 │ 20170428 │ - │ +│ 72ed9cd │ Nov 15, 2020 │ 0.5674 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 50 │ - │ - │ +│ ├── exp-47dfa │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ └── exp-44136 │ Feb 17, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 50 │ - │ - │ +│ f8e9d93 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 50 │ - │ - │ +│ 377c988 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 500 +│ 1 │ 20170428 │ 50 │ - │ - │ +│ 27d4e7c │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 +│ 1 │ 20170428 │ 50 │ - │ - │ +│ 8bf2091 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 +│ 1 │ 20170428 │ 50 │ - │ - │ +│ ff9e2fa │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 +│ 1 │ 20170428 │ 50 │ - │ - │ +│ 11e1705 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ a82585e │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ c778a78 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ 551082e │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +│ 15bef96 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 +│ 2 │ 20170428 │ 10 │ - │ - │ +└────────────────┴──────────────┴─────────┴───────────────┴──────────────┴────────────────────────┴──────────────────┴────────────┴────────────────────┴────────────────────┴───┘ +``` + +This table includes parameters and metrics from the `evaluate` stage and all +stages upstream of it. It's possible to also have a `--single-item` argument to +exclude upstream changes, but this is out of scope for now. + +# How We Teach This + +See [Detailed design](#detailed-design) for a proposed help message. + +This is consistent with other commands, and it's only one additional argument to +an existing command, so it likely doesn't require much teaching. It might be +necessary to explicitly add that parameters and metrics from upstream stages +will be shown. + +# Drawbacks + +This proposal may require changes to how DVC collects and tracks parameters and +metrics. Implementation could break existing functionality, and it is +inconsistent with the current behaviors of `dvc params`, `dvc metrics`, and `dvc +exp diff` commands. + +Further, this proposal adds to an already crowded set of `dvc exp show` options, +and there is no single way to show parameters and metrics that's going to work +for every use case. For example, this proposal does nothing to filter out +parameters for upstream stages, which may frequently be irrelevant if only the +stages near the end of the pipeline change during experimentation. + +# Alternatives + +There are many alternative methods to refine the output of `dvc exp show`: + +* Current functionality: `exp show` already has options to include/exclude any + column, so the flexibility to get the same output as suggested here already + exists. +* Expand the include/exclude options to accept wildcard/glob/regex arguments for + more flexibility to exclude an entire file or section of a file like + `exclude-params=params.json:local_config*`. +* Save configuration options for `dvc exp show` so they can be reused and + modified without typing out all options at the command line each time. +* Add a tsv/csv output format option in `dvc exp show` to enable users to + manipulate the table with other tools. + +This proposal does not preclude any of these options. The current output +will continue to be available except for untracked parameters. The other +alternatives may still be implemented. + +Each of the alternatives provide flexibility to modify the `dvc exp show` +outputs, which is important but not the goal of this proposal. Instead, the +proposed feature uses the information DVC has to make smart decisions about what +to show and save the users from needing to make modifications that would always +be desirable. + +# Unresolved questions + +* What about consistency with `parameters`, `metrics`, and `exp diff` commands? + Should these also be modified or left as is? As mentioned above, `parameters` + and `metrics` in particular may be used on arbitrary files outside of DVC + projects. +* Should experiments themselves be excluded based on what stages were run? Using + the example above, if a user ran both `dvc exp run evaluate` and `dvc exp run + topic_modeling` experiments, should all of these experiments show when running + `dvc exp show evaluate`? From 3cbdf53170e956eb63697d584a50b20cf21c92ff Mon Sep 17 00:00:00 2001 From: dberenbaum Date: Fri, 19 Mar 2021 12:46:36 -0400 Subject: [PATCH 2/6] fix table in exp-show-target --- proposals/exp-show-target.md | 68 ++++++++++++------------------------ 1 file changed, 23 insertions(+), 45 deletions(-) diff --git a/proposals/exp-show-target.md b/proposals/exp-show-target.md index 3e35ad0..b3acac6 100644 --- a/proposals/exp-show-target.md +++ b/proposals/exp-show-target.md @@ -120,52 +120,30 @@ and metrics for the `topic_modeling` and `combine` stages: ```console ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━┓ -┃ Experiment ┃ Created ┃ auc ┃ prepare.split ┃ prepare.seed ┃ -featurize.max_features ┃ featurize.ngrams ┃ train.seed ┃ train.n_estimators ┃ -train.random_state ┃ … ┃ +┃ Experiment ┃ Created ┃ auc ┃ prepare.split ┃ prepare.seed ┃ featurize.max_features ┃ featurize.ngrams ┃ train.seed ┃ train.n_estimators ┃ train.random_state ┃ … ┃ ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━┩ -│ workspace │ - │ 0.51625 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ topic_modeling │ Mar 18, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ └── exp-44136 │ 10:29 AM │ 0.51625 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ master │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ └── exp-56d11 │ Mar 18, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ 263732a │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 -│ 2 │ - │ 100 │ 20170428 │ - │ -│ b297969 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 -│ 2 │ - │ 100 │ 20170428 │ - │ -│ 4edd930 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 -│ 2 │ - │ 100 │ 20170428 │ - │ -│ 72ed9cd │ Nov 15, 2020 │ 0.5674 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 50 │ - │ - │ -│ ├── exp-47dfa │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ └── exp-44136 │ Feb 17, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 50 │ - │ - │ -│ f8e9d93 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 50 │ - │ - │ -│ 377c988 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 500 -│ 1 │ 20170428 │ 50 │ - │ - │ -│ 27d4e7c │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 -│ 1 │ 20170428 │ 50 │ - │ - │ -│ 8bf2091 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 -│ 1 │ 20170428 │ 50 │ - │ - │ -│ ff9e2fa │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 -│ 1 │ 20170428 │ 50 │ - │ - │ -│ 11e1705 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ a82585e │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ c778a78 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ 551082e │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ -│ 15bef96 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 -│ 2 │ 20170428 │ 10 │ - │ - │ +│ workspace │ - │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ topic_modeling │ Mar 18, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ ├── exp-e2784 │ 12:15 PM │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ └── exp-44136 │ 10:29 AM │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ master │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ └── exp-56d11 │ Mar 18, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ 263732a │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ - │ 100 │ 20170428 │ - │ +│ b297969 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ - │ 100 │ 20170428 │ - │ +│ 4edd930 │ Feb 11, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ - │ 100 │ 20170428 │ - │ +│ 72ed9cd │ Nov 15, 2020 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 50 │ - │ - │ +│ ├── exp-47dfa │ Feb 17, 2021 │ 0.51625 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ └── exp-44136 │ Feb 17, 2021 │ 0.5674 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 50 │ - │ - │ +│ f8e9d93 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 50 │ - │ - │ +│ 377c988 │ Nov 15, 2020 │ 0.54175 │ 0.2 │ 20170428 │ 500 │ 1 │ 20170428 │ 50 │ - │ - │ +│ 27d4e7c │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 │ 1 │ 20170428 │ 50 │ - │ - │ +│ 8bf2091 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 │ 1 │ 20170428 │ 50 │ - │ - │ +│ ff9e2fa │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 500 │ 1 │ 20170428 │ 50 │ - │ - │ +│ 11e1705 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ a82585e │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ c778a78 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ 551082e │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ +│ 15bef96 │ Nov 15, 2020 │ - │ 0.2 │ 20170428 │ 1500 │ 2 │ 20170428 │ 10 │ - │ - │ └────────────────┴──────────────┴─────────┴───────────────┴──────────────┴────────────────────────┴──────────────────┴────────────┴────────────────────┴────────────────────┴───┘ ``` From dac9c00989975e0b2e8475d2e838ba236d4c0999 Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Mon, 22 Mar 2021 20:02:53 -0400 Subject: [PATCH 3/6] Update proposals/exp-show-target.md Co-authored-by: Jorge Orpinel --- proposals/exp-show-target.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/exp-show-target.md b/proposals/exp-show-target.md index b3acac6..1061f8e 100644 --- a/proposals/exp-show-target.md +++ b/proposals/exp-show-target.md @@ -5,8 +5,8 @@ The `dvc exp show` command shows a comparison of the results of different experiments along with the parameters and metrics values for each experiment. -The output currently includes all values that are defined in `params.yaml` or -any other referenced parameters files in any DVC stage. Output also includes all +The command's output currently includes all values that are defined in `params.yaml` or +any other referenced parameters files in any DVC stage. It also includes all metrics in any referenced metrics files in any DVC stage. This proposal allows users to see only parameters and metrics relevant to specified pipeline stages. From 738d77052e91a8f439b310c5fb607cd6ebda446d Mon Sep 17 00:00:00 2001 From: dberenbaum Date: Mon, 22 Mar 2021 20:21:38 -0400 Subject: [PATCH 4/6] summary edits --- proposals/exp-show-target.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/proposals/exp-show-target.md b/proposals/exp-show-target.md index 1061f8e..0d2554b 100644 --- a/proposals/exp-show-target.md +++ b/proposals/exp-show-target.md @@ -1,3 +1,5 @@ +# Add targets/stages argument to dvc exp show + - Enhancement Proposal PR: (leave this empty) - Contributors: dberenbaum, (add your Github handle) @@ -5,10 +7,12 @@ The `dvc exp show` command shows a comparison of the results of different experiments along with the parameters and metrics values for each experiment. -The command's output currently includes all values that are defined in `params.yaml` or -any other referenced parameters files in any DVC stage. It also includes all -metrics in any referenced metrics files in any DVC stage. This proposal allows -users to see only parameters and metrics relevant to specified pipeline stages. +The command's output currently includes: +* All values that are defined in `params.yaml`. +* All values in other referenced parameters files in any DVC stage. +* All values in any referenced metrics files in any DVC stage. +This proposal allows users to see only parameters and metrics relevant to +specified pipeline stages. See https://github.com/iterative/dvc/issues/5451 for more background. From 6c73ed13ffc32ab3fbf69a07ef68d3ed47187be8 Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Wed, 24 Mar 2021 12:14:18 -0400 Subject: [PATCH 5/6] Update proposals/exp-show-target.md Co-authored-by: Jorge Orpinel --- proposals/exp-show-target.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/exp-show-target.md b/proposals/exp-show-target.md index 0d2554b..cda8410 100644 --- a/proposals/exp-show-target.md +++ b/proposals/exp-show-target.md @@ -11,6 +11,7 @@ The command's output currently includes: * All values that are defined in `params.yaml`. * All values in other referenced parameters files in any DVC stage. * All values in any referenced metrics files in any DVC stage. + This proposal allows users to see only parameters and metrics relevant to specified pipeline stages. From 1f46c0ccf4fc98c2fc5b3422343e2696e92aad04 Mon Sep 17 00:00:00 2001 From: dberenbaum Date: Wed, 24 Mar 2021 16:10:57 -0400 Subject: [PATCH 6/6] links to issues for other exp show enhancements --- proposals/exp-show-target.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/proposals/exp-show-target.md b/proposals/exp-show-target.md index cda8410..7e24b60 100644 --- a/proposals/exp-show-target.md +++ b/proposals/exp-show-target.md @@ -185,13 +185,15 @@ There are many alternative methods to refine the output of `dvc exp show`: * Current functionality: `exp show` already has options to include/exclude any column, so the flexibility to get the same output as suggested here already exists. -* Expand the include/exclude options to accept wildcard/glob/regex arguments for - more flexibility to exclude an entire file or section of a file like +* [Wildcards](https://github.com/iterative/dvc/issues/5642): Expand the + include/exclude options to accept wildcard/glob/regex arguments for more + flexibility to exclude an entire file or section of a file like `exclude-params=params.json:local_config*`. * Save configuration options for `dvc exp show` so they can be reused and modified without typing out all options at the command line each time. -* Add a tsv/csv output format option in `dvc exp show` to enable users to - manipulate the table with other tools. +* [CSV](https://github.com/iterative/dvc/issues/5446): Add a tsv/csv output + format option in `dvc exp show` to enable users to manipulate the table with + other tools. This proposal does not preclude any of these options. The current output will continue to be available except for untracked parameters. The other