Add versioning scheme in the dvc.lock

[skshetry]

dvc.lock

• New version encloses all of the lock details of all the stages inside stages keyword.
• Previously it uses to be just a dictionary of lock details of stages.
• New meta keyword has version identifier. We could add more metadata there in the future.
• 1.0 will be read-only and will be migrated to 2.0 format when any of the stages in the lock file are reproduced (should there be a command?).

v1.0

copy:
  cmd: cp foo bar
  deps:
  - path: foo
    md5: d3b07384d113edec49eaa6238ad5ff00
    size: 4
  outs:
  - path: bar
    md5: d3b07384d113edec49eaa6238ad5ff00
    size: 4

v2.0

meta:
  version: '2.0'
stages:
  copy:
    cmd: cp foo bar
    deps:
    - path: foo
      md5: d3b07384d113edec49eaa6238ad5ff00
      size: 4
    outs:
    - path: bar
      md5: d3b07384d113edec49eaa6238ad5ff00
      size: 4

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[efiop]

v2.0
meta:
version: '2.0'

Could you please elaborate on using meta for version instead of putting it at the top level?

[efiop]

Change version to schema?
If we tie our version to DVC's version, then version might make sense (as in, having new file format updates each major version of DVC).

To major releases or to minor ones too? What if it stabilizes and dvc v101 will generate the same schema s dvc v2?

Should version be strict or just informative?
docker compose for example, uses versioning, but it's for backward compatibility and is for informative purpose.

So far we've been implicitly treating lock files as strict, and when unknown fields are found - we are deeming the whole lockfile corrupted. So strict version might continue the trend, but informative (+ warnings about unknown fields) is definitely friendlier (people were complaining about nfiles/size fields additions recently).

Should we always default version keyword to 1.0?
Alternately, we could support uppercased, single-word variables or break compatibility completely (% of users that will be affected by this change would be small). This is in addition to the dvc.yaml/dvc.lock having version identifier. This would allow us to always have the "version-less" dvc.yaml to be considered the latest version, but make DVC always add them by default and recommend user to do the same as well (if they want compatibility).

Could you elaborate on variables, please? Not sure I understand.

[skshetry]

To major releases or to minor ones too?

Better to not break compatibility on minor releases. But, no strong opinion with schema either (feels like it's too much of a technical term).

Could you elaborate on variables, please? Not sure I understand.

The only thing that makes v1 -> v2 incompatible is the parametrization syntax (foreach .. do and vars can be made compatible). As an example, if user were using ${TMPDIR} in the cmd, v2 will start complaining considering it as a parametrization syntax rather than an env var and they'd have to escape them with \ (as, \${TMPDIR}).

But, we could make them compatible if we consider uppercased single-word inside ${} as a special case and avoid interpolating them. This way, we could make the version informative rather than being a schema This way v1 will still support foreach and vars, but DVC can warn users on missing/new attributes. If in the future, if only there will be any incompatible changes in some attributes' schema, DVC will take the version into account (and, refuse to run if it cannot proceed).

Or, we could just break compatibility (how many users will be using ${VAR} vs $VAR syntax in cmd?).

Making the version strict has it's advantage, as we could always abandon the schema completely, but it comes at a cost of user-facing version identifer.

Could you please elaborate on using meta for version instead of putting it at the top level?

No reason, just wanted to house all possible meta fields there.

[efiop]

For the record: accidental merge, but still need to review post-merge 🙂

[efiop]

Maybe let's just have top-level schema: 2.0? Meta is confusing with meta from dvc.yaml and version is confusing with dvc version.

[efiop]

This will definitely get confused with stage meta :(

[efiop]

My only concern is with meta and version, otherwise looks good. 👍