params: bulk params from file

[dberenbaum]

Params need to be specified individually, which can be repetitive and inconvenient where users want to simply include all parameters from a file.

For background, see:

• Introduce hyper parameters and config #3393 (comment)
• `params` function registry #6506 (comment)

Instead of specifying every parameter individually, it should be possible to do dvc stage add foo -p my_params.yaml python foo.py or dvc stage add foo -p "my_params.yaml:*" python foo.py, which could generate this dvc.yaml:

stages:
  foo:
    cmd: python foo.py
    params:
    - my_params.yaml

or

stages:
  foo:
    cmd: python foo.py
    params:
    - my_params.yaml:*

After running dvc repro, dvc.lock could treat my_params.yaml like a regular file dependency:

stages:
  foo:
    cmd: python foo.py
    deps:
    - path: my_params.yaml
    - md5: abc123

However, commands that collect parameters to show info about them, like dvc params diff and dvc exp show, would parse my_params.yaml and show each individual parameter.

[skshetry]

Some questions that I have:

1. Does this mean tracking all the params of the file at a time or all the time? Do we need support for both?
   See run: allow tracking of all params in a file with -p #4112.
2. Do we want to save the contents of it or the complete file?
   We have never saved the params file before. We have only used the data from dvc.lock to see if they have changed.
3. What should be the order of resolving params in the following case, foo from params.yaml or foo the params file?

params:
- foo

Regarding the structure, I prefer dvc.lock mirror dvc.yaml, having params section from dvc.yaml to not refer to params section in dvc.lock maybe confusing.

This will also affect run-cache. It's not a big issue, as it's just a dvc.lock like file, but this change will also affect run-cache.

[dberenbaum]

@skshetry Returning to this. Thanks for the reference to #4112. I would probably go so far as to mark this a duplicate, close this, and continue conversation there. WDYT?

[skshetry]

Duplicate of #4112

[skshetry]

Let's close this then @dberenbaum.