stage collection: do not collect .gitignored dvcfiles

[skshetry]

Closes #5244

A few fixes has been made along the way, including:

• better error message if the dvc.yaml and .dvc files are dvc-ignored.

  $ echo "dvc.yaml" >> .dvcignore
  $ dvc repro
  ERROR: 'dvc.yaml' is dvc-ignored. 

• better error message if the dvc.yaml and .dvc files are git-ignored.

  $ echo "dvc.yaml" >> .gitignore
  $ dvc repro
  ERROR: 'dvc.yaml' is git-ignored.

An open question is, what should we do about lockfiles if they are git-ignored or dvc-ignored?
Right now, we assume the lockfile as empty if they are dvc-ignored and we don't care about git-ignores
for the lockfiles. I feel that this behaviour is good, but if we want them to be consistent, please do tell me.

Another thing to keep in mind is, since the implementation depends on isinstance(repo.tree, LocalTree), if we start using LocalTree for get/list/import, it'll break, though I don't think it should as we better apply it to only the user's workspace (IIRC dvcx uses this mode).
Though, this might be handled by a state (eg: repo.is_external_repo or something like that).

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[skshetry]

This check might not work everywhere, especially in for_write=True scenarios in dvcx. But as we are adjusting our APIs in ExternalRepo, Repo.clone() should set some kind of state to workaround that. And, we'll add additional check for repo.is_external here and that should be enough.

[pared]

An open question is, what should we do about lockfiles if they are git-ignored or dvc-ignored?

I think users should be able to ignore lockfile. Dvc is able (to some extent) to work without it, so removing this possibility would limit dvc's functionality. And I guess one can come with some use cases where this is actually a feature. For example any educational material that's purpose is to show how running the pipeline looks like.

[pmrowla]

An open question is, what should we do about lockfiles if they are git-ignored or dvc-ignored?

I think users should be able to ignore lockfile. Dvc is able (to some extent) to work without it, so removing this possibility would limit dvc's functionality. And I guess one can come with some use cases where this is actually a feature. For example any educational material that's purpose is to show how running the pipeline looks like.

One thing to keep in mind is that experiments runs in temp directories (and on remote machines in the future) have to git commit dvc.lock in order for us to retrieve the experiment result. We can (and already do) force-add gitignored lock files internally for these types of runs, and on the user's local exp apply the dvc.lock file would no longer be staged/committed like normal. But this does seem like the kind of thing that may lead to unexpected behavior in the future and it may be safer for us to forbid it?

[pared]

@pmrowla Fair point, I didn't consider experiments in my thought process. In that case it might be better to just require it.