A 'dev' mode to preserve local changes in git/hg repositories

[cdevienne]

Describe the problem/challenge you have

We use vendir in meta-projects, and work directly in the git/hg synced sub-repositories.
After doing some changes in them, we cannot do a "vendir sync" without loosing local changes (and lazy doesn't work if we pull some ref changes).
Also, after commit & pushing changes in the sub-repositories, we would like to automatically update the vendir configuration file with the current commit id/tag of each repo.

Describe the solution you'd like

We think vendir could benefit from a "dev" mode, which would be enabled by using a "dev-sync" command or a "--dev" switch on the sync command.

When turned on, vendir would change its sync behavior for mercurial and git (at least) in the following way:

• if the target directory already exists, keep it
• when using already existing target :
  • check if non-committed changes exists, in which case the sync is aborted
  • fetch & checkout directly in the target directory

A new command "dev-baseline" (for example) would, for each vcs directory, determine what the current ref is:

• if current revision has a tag, the tag
• otherwise, the sha of the current revision
  If the ref in the vendir file is not a branch (or a topic the mercurial case), and does not match the current ref determined above, the configuration is updated with the new current ref.

Anything else you would like to add:

The overall experience is inspired by the "hg-confman" mercurial extension that we intend to replace with vendir.

We are willing to work on an implementation for these features, and would like your opinion before diving into coding.

[praveenrewar]

We use vendir in meta-projects, and work directly in the git/hg synced sub-repositories.

@cdevienne Could you through some more light on this use case. vendir is used to pull config from a source. It seems like you also want to make changes in the same config and then be able to pull more changes from the source, which would make vendir work on two different source of truths.
Also, it would require vendir to do things like merging the config from source and local.
cc @joaopapereira

[cdevienne]

One of our use cases is the following: we have repositories for odoo modules, that we sync in a top-level client project. When working on the client project, the developer updates some code in synced sub-repo(s), runs some tests locally and when satisfied commit changes in both the synced sub-repo(s) and the top-level repo.

When working a sub-repo, there is a period of time where the synced directory is not in-sync. So not really 2 sources of truth, just a gray area, until one those 2 events happen:

• vendir sync is called, and the reference expected by the vendir configuration is checked out
• the vendir configuration is updated to match the local reference

We can already do this with vendir but it requires to be very careful if we don't want to lose any local change because 'sync' will drop everything and start over.

The changes we suggest in "vendir sync" provide safety for the local changes, and would work a little like the "--lazy" option but safer. So no added need for merging configurations.

And the "dev-baseline" command is to make the vendir configuration update easier.

[rohitagg2020]

@cdevienne We had a discussion on this in the last community meeting. Some of the ideas shared were:

1. As the dev can means a lot of things, we can have something like --skip <comma separated path> to simply skip a content type/section in the vendir.yml file while doing vendir sync. And also at the same time, add a field (skip: true/false) in the content type inside vendir.lock.yml file to indicate that this section was skipped when vendir sync is run.
2. See if setting the manual content Type helps. Benefit of it is that it is already implemented today. But the con is to change the vendir.yml file everytime you decide to switch between sync/not sync upstream.

We can come up with proposal together on the same.

[cdevienne]

@rohitagg2020

I agree "dev" might not be the ideal name. But it is not a "skip" either because the key ideas are:

• keep the local commits (ie do a fetch instead of a new clone when syncing)
• skip syncing only if uncommitted changes are found

I guess it applies only to git & mercurial repositories.

[rohitagg2020]

@cdevienne

We are thinking more from the generic behavior on the vendir side. For the following scenario:

keep the local commits (ie do a fetch instead of a new clone when syncing)

I believe you will do merge later on to get the changes. As there is already a manual step involved down the line, you can select not to sync it using vendir and then do git fetch and git merge manually.

[cdevienne]

Our main goal is to secure local changes by default, so a "vendir sync" can never mean loosing some work.

If we cannot do a fetch (which btw would be a lot faster) instead of cloning, then we need to make sure, before deleting any git/hg repo, that they have no local commits and no local change (including staged ones).

If we favor this approach, then a "safe" mode could make sense.

[rohitagg2020]

then a "safe" mode could make sense.

Are you thinking about safe as a flag to the vendir sync command? As of today, the default behavior is to delete and recreate. To keep the backward compatibility, we cant move away from this behavior. Thus, if user wants to retain the changes, he has to provide this flag explicitly/ add as part of vendir.yml configuration (depends on how it is implemented).

[cdevienne]

Yes, a "--safe" flag is what I have in mind. It could be turned on with an environment variable too (also maybe in the vendir.yml configuration).
It would not change the behavior when proceeding with the sync, only cancel it in any case of a local change.
Here is what we propose to work on:

• Add a '--safe' flag to 'sync', that refuses to sync altogether if any local change or unpushed commit is detected.
• Add a 'baseline' command that updates the vendir.yml with currently checked out refs. It would stop if any local uncommitted change is found
• Add a 'status' command that display a list of the local changes and local commits, and whether the current checked out ref matches the one in vendir.yml. For non hg/git directories, it would only tell if the target directory is present or not.