guide: add "Best Practices"

[efiop]

UPDATE: Possibly as a How To guide (see #899)

Looks like we need a special section describing ways how to organize your projects:

• how to use DVC with DB (see how to use DVC when the data is stored in a DB #594)
• our default Dvcfile trick
• manually editing dvc.yaml + dvc commit or dvc repro (see also how to: "update your pipeline by editing DVC-files" #230 (comment))
  it's safe to edit DVC files, no need to touch or update md5, DVC will take care of it
  UPDATE: See dvc.yaml: encourage manual writing/editing stages (and pipelines) #2578
• specify meaningful stage names with -f
• creating a pipeline in a 'debug' directory and then moving it to different data sets
• creating a pipeline in a 'debug' directory and then modifying respective DVC files to set different data sets as an input
• add "use meta to preserve your content" - describe "meta" field in stage file and that we are preserving comments #306
• never store user credentials in the DVC project config
• one vs many dvc.yaml files (from pipelines: clarify whether it's possible to have more than one _pipeline file_ for a DVC project in the docs #2170 (comment))

See also the latest relevant #72 (comment) and below.

[jorgeorpinel]

Should this be merged with #230 and featured in the #899 epic? We're trying to avoid so many sections now.

Also, the Questions part of What is DVC? (currently in https://dvc.org/doc/user-guide/what-is-dvc/collaboration-issues#questions) probably overlaps with this.

[efiop]

@jorgeorpinel Yeah, that indeed seems suitable.

[shcheklein]

@jorgeorpinel how would it looks like? like a subsection in How To?

[jorgeorpinel]

Just a single document under How To.

I updated the description of this issue and in fact I think #230 is already included here, in the "manually editing dvc.yaml + dvc commit or dvc repro" checkbox.

[jorgeorpinel]

More:

• how to work with overlapping stage output locations (e.g. hopefully with wildcards in deps/outs soon) — see https://discuss.dvc.org/t/managing-pipelines-operating-per-dataset-element/613/4 for a current alternative.
• DVC in Production setup (see cases: DVC in Production #862 (comment))
• Add dvc version to your first DVC repo commit? Or another way to know what version(s) you've used (since file formats may change, especially between major versions).

[jorgeorpinel]

Looking at open check boxes, most or all of these topics are addressed I think (the relevant ones at least).

how to use DVC with DB

This would be a how-to, but is it still something we want to have official docs for? Not really matching DVC's approach

creating a pipeline in a 'debug' directory and then ... set different data sets as an input

@efiop is this some sort of bootstrapping method? Is it really something people do? What problem does it solve?

never store user credentials in the DVC project config

We do stress the use of --local for sensitive configurations (especially in remote modify). Should be enough I believe.

Avoid dynamic names

We mention non-deterministic behavior in general in https://dvc.org/doc/command-reference/stage/add#avoiding-unexpected-behavior and avoiding ad hoc file naming for versioning is a core use case..

[jorgeorpinel]

all of these topics are addressed

So we could close this ticket. That said we still don't have a "Best Practices" section or guide(s). Do we want to? Maybe Use Cases or the trails of the Get Started already cover this need (informing users of the main/recommended patterns for DVC project setup/usage).

WDYT @shcheklein @dberenbaum ? Thanks

[shcheklein]

I'm fine to close this. Not all items are covered though and yes, we could have done a good page that contains tips/faq for pipelines, for data, general project structure (e.g. never store user credentials in the DVC project config) ...