Release 3.0

[efiop]

Data Management

Beta Give feedback

1. drop dos2unix behaviour #4658
   p1-important +1
   
2. drop external outputs #9531 (@efiop)
3. Rename dvc add --jobs to --remote-jobs (frequently gets confused with core.checksum_jobs)
4. cache migrate: add utility for migrating local cache files to 3.x #9591
   +0
   

Experiments and Pipelines

Beta Give feedback

1. Drop/Revisit usage of checkpoints #9221
   A: experiments discussion +2
   
2. add dvc stage add --run
3. Remove Stage-level vars (@skshetry)
4. dvc exp push changes? (see https://iterativeai.slack.com/archives/C01116VLT7D/p1686570450222809?thread_ts=1686321601.633079&cid=C01116VLT7D) decided not to make breaking changes now; will revisit dropping --rev HEAD

Deprecations

Beta Give feedback

1. deprecate old .dvc file based pipeline stages? #5940. (@skshetry )
2. Remove exp gc, exp init, exp show --html, exp show --pcp.
3. Deprecate dvc add --recursive flag.
4. Remove dvc run.
5. Remove --show-json/--show-csv/--show-md in favor of existing --json/--csv/--md flags.
6. ‍‍‍‍‍Drop top-level plots definition as a dictionary, can be defined as a list instead.
7. Drop 1.0 dvc.lock format support.
8. Soft deprecate desc/type/labels/meta in dvc.yaml and .dvc files, now that we have artifacts (drop CLI flags, dvc data ls, and drop them from docs).
9. Remove repro/exp run --metrics flag #9457 (and replace with --message short flag)
10. Deprecate dvc add --file (obscure and more likely to confuse than help)

Other release blockers

• --allow-missing: skips stages even if deps have changed #9530
• data status --not-in-remote: not working #9541
• add/import-url: fix cloud versioning #9584

Studio Readiness for 3.0 release

• https://github.com/iterative/studio/issues/5944

[dberenbaum]

Some possible ideas to add looking through dvc add options:

• Deprecate dvc add --file (obscure and more likely to confuse than help)
• Rename dvc add --jobs (frequently gets confused with core.checksum_jobs)

[skshetry]

Two more proposals from my side:

• Remove Stage-level vars. I haven't seen this being used and makes implementation complex.
• Make dvc exp push work by default, or remove the need for explicit remote argument.
  (If we make dvc exp push work by default, what should it push?)

[dberenbaum]

* [Remove Stage-level vars](https://dvc.org/doc/user-guide/project-structure/dvcyaml-files#:~:text=Stage%2Dspecific%20values%20are%20also%20supported). I haven't seen this being used and makes implementation complex.

😅 https://discord.com/channels/485586884165107732/1111557378735865856/1111636564183883876

It's not clear to me whether the stage-level vars are needed there. @skshetry Maybe you have an idea for how else to handle that use case?

* Make `dvc exp push` work by default, or remove the need for explicit `remote` argument.
  (If we make `dvc exp push` work by default, what should it push?)

I've come around to not minding the explicit remote requirement since it has occasionally saved me from accidentally pushing to origin when it was an upstream remote (like when I clone a demo project but don't want to mess with the studio demo). Especially now that we have VS Code to make pushing easier, I don't mind keeping the CLI explicit.

[dberenbaum]

* [Remove Stage-level vars](https://dvc.org/doc/user-guide/project-structure/dvcyaml-files#:~:text=Stage%2Dspecific%20values%20are%20also%20supported). I haven't seen this being used and makes implementation complex.

😅 https://discord.com/channels/485586884165107732/1111557378735865856/1111636564183883876

It's not clear to me whether the stage-level vars are needed there. @skshetry Maybe you have an idea for how else to handle that use case?

@skshetry It turns out they aren't needed in that case, so I'm on board to drop them.

[efiop]

One last thing we forgot to include is to stop validating cache on every minor operation. This is a leftover from the past where symlink/hardlink were default and you ran a greater risk to corrupt your cache. These days we default to reflink/copy, so there is no reason really to validate cache on every operation like checkout. We should instead only validate cache where we really need to (before push?) and just introduce dvc cache check (or something like that) for manual validation.

In practical terms, I don't think we document that certain operations check cache, so we are not really tied to 3.0 release and can just stop doing it whenever.

[dberenbaum]

In practical terms, I don't think we document that certain operations check cache, so we are not really tied to 3.0 release and can just stop doing it whenever.

Why don't we create a separate issue for it then and just make it high priority so we get to it soon?

[skshetry]

Stage-level vars was created to support loading different parameters for different stages. Take following as an example:

stages:
  train_model:
    foreach: [us, uk]
    do:
      wdir: ${item}
      vars: [params.yaml]  # uk/params.yaml, us/params.yaml, etc.
      cmd: cat params.yaml

But I don't find it very elegant, is cumbersome, has a lot of complexity and haven't seen anyone use this. Hydra support may have made this redundant.

[sjawhar]

Stage-level vars was created to support loading different parameters for different stages. Take following as an example:

stages:
  train_model:
    foreach: [us, uk]
    do:
      wdir: ${item}
      vars: [params.yaml]  # uk/params.yaml, us/params.yaml, etc.
      cmd: cat params.yaml

But I don't find it very elegant, is cumbersome, has a lot of complexity and haven't seen anyone use this. Hydra support may have made this redundant.

We actually use this feature extensively. Our piplines tend to have a lot of this:

stages:
  preprocess_feature_one:
    foreach: ${sessions}
    do:
      vars:
        - preprocessing_key: feature_one
          input_dir: load
      <<: &stage_preprocess
        cmd: >-
          python src/preprocess.py
          --params-file=params.yaml
          ${preprocessing_key}
          data/interim/${input_dir}/${key}
          data/interim/${preprocessing_key}/${key}
        params:
          - preprocess.${preprocessing_key}
        deps:
          - ../../poetry.lock
          - data/interim/${input_dir}/${key}
          - src/preprocess.py
        outs:
          - data/interim/${preprocessing_key}/${key}

  preprocess_feature_two:
    foreach: ${sessions}
    do:
      vars:
        - preprocessing_key: feature_two
          input_dir: feature_one
      <<: *stage_preprocess

While removing this functionality might not completely block us from upgrading to DVC 3.x—one could copy/paste the stage definition with small changes—it would definitely increase the maintenance burden of our pipelines and make mistakes much easier to make (oops, forgot to change X in all the stages). It's possible I'm overlooking some functionality made possible by hydra?

I might have even gloated about our use of this pattern in my last DVC office hours 😅

[efiop]

@dberenbaum Created #9561

[dberenbaum]

Thanks @efiop! Does #9561 relate to 3.0 items?

[dberenbaum]

@sjawhar That's a really creative way to do things! If we were to implement #5172, do you think that would solve the same use case? From what I can tell, you are limited by needing to loop over both sessions and features.

@skshetry WDYT about the comment from @sjawhar above? What do you think would be the best way to do it?

[efiop]

@dberenbaum I will try to create dvc cache/remote check if I have time this week and if I will - it will be nice to drop cache checks in a few places right away. If not - i think we can do that later, even though that's not ideal.

[skshetry]

@sjawhar, that's an interesting way to do nested looping with merge-key.

How many sessions do you run? If they aren't that many, I'll suggest using a (duplicated) map/dictionary and use that loop.

# params.yaml
sessions:
  feature1_session1:
    key: feature_one
    input_dir: feature_one
  feature1_session2:
    key: feature_one
    input_dir: feature_one
  feature2_session1:
    key: feature_two
    input_dir: feature_two
  feature2_session2:
    key: feature_two
    input_dir: feature_two

This might be more readable, although you can still use the same merge-key to avoid duplications.

sessions:
  feature1_session1: &feature1
    key: feature_one
    input_dir: feature_one
  feature1_session2: *feature1
  ...
  feature2_session1: &feature2
    key: feature_two
    input_dir: feature_two
  feature2_session2: *feature2
  ...

#5172 might be a better solution to fix this.

[sjawhar]

@sjawhar, that's an interesting way to do nested looping with merge-key.

How many sessions do you run? If they aren't that many, I'll suggest using a (duplicated) map/dictionary and use that loop.

@skshetry When we use DVC in a data registry pattern, there could be hundreds or thousands of sessions. Yes, though, you're right that we could have an additional step the one runs before running dvc commands which generates this params.yaml, but supporting it natively in DVC seems more logical (to me at least)