unpack-trees:virtualfilesystem: Improve efficiency of clear_ce_flags

[neerajsi-msft]

When the virtualfilesystem is enabled the previous implementation of
clear_ce_flags would iterate all of the cache entries and query whether
each one is in the virtual filesystem to determine whether to clear one
of the SKIP_WORKTREE bits. For each cache entry, we would do a hash
lookup for each parent directory in the is_included_in_virtualfilesystem
function.

The former approach is slow for a typical Windows OS enlistment with
3 million files where only a small percentage is in the virtual
filesystem. The cost is
O(n_index_entries * n_chars_per_path * n_parent_directories_per_path).

In this change, we use the same approach as apply_virtualfilesystem,
which iterates the set of entries in the virtualfilesystem and searches
in the cache for the corresponding entries in order to clear their
flags. This approach has a cost of
O(n_virtual_filesystem_entries * n_chars_per_path * log(n_index_entries)).

The apply_virtualfilesystem code was refactored a bit and modified to
clear flags for all names that 'alias' a given virtual filesystem name
when ignore_case is set.

n_virtual_filesystem_entries is typically much less than
n_index_entries, in which case the new approach is much faster. We wind
up building the name hash for the index, but this occurs quickly thanks
to the multi-threading.

Signed-off-by: Neeraj Singh [email protected]

[ghost]

All CLA requirements met.

[derrickstolee]

Thanks! installers are being built here: https://dev.azure.com/gvfs/ci/_build/results?buildId=19487&view=results

[neerajsi-msft]

One thing I'd like to mention for review is that the previous code was clearing flags for cache entries with a nonzero ce_stage. I don't think we were setting those flags in the caller, but it's worth noting that subtlety.

[derrickstolee]

There is a VFS for Git functional test failure that I will investigate tomorrow. I'll see if I can reproduce it in a Git test to give us confidence that it is fixed within this PR.

[neerajsi-msft]

Hi Derrick,
Thanks for pointing out the failure. I'm still investigating and already have a local repro. I don't think you should need to investigate.

[neerajsi-msft]

I figured out the root cause of the test failure, which was a real regression introduced by this change:

When updating the index flags in clear_ce_flags_virtualfilesystem_1 we need to iterate through all entries that match the name case-insensitively so that we don't miss clearing the bits if two names exist in the index.

Here's a minimal repro script in case that's interesting to you. Every step is required to observe the failure:

set GIT_EXE="git.exe"
set REPO_ROOT=c:\repos\gvfs_repo
set GIT_ROOT=%REPO_ROOT%\src
rd /s /q %REPO_ROOT%

GVFS.exe clone "https://gvfs.visualstudio.com/ci/_git/ForTests" "%REPO_ROOT%" --branch "FunctionalTests/20201014" --local-Cache-path "C:\Repos\GVFSFunctionalTests\enlistment\..\.gvfsCache"  --internal_use_only "{\"ServiceName\":\"Test.GVFS.Service\",\"StartedByService\":false,\"MaintenanceJob\":null,\"PackfileMaintenanceBatchSize\":null}"

pushd %GIT_ROOT%

%GIT_EXE% config core.abbrev 40
%GIT_EXE% config user.name "Functional Test User"
%GIT_EXE% config user.email "[email protected]"
%GIT_EXE% config advice.statusUoption false

%GIT_EXE% checkout FunctionalTests/20201014
%GIT_EXE% branch -D tests/functional/CaseOnlyRenameFileAndChangeBranches
%GIT_EXE% checkout -b tests/functional/CaseOnlyRenameFileAndChangeBranches FunctionalTests/20201014
%GIT_EXE% mv Readme.md README.md
%GIT_EXE% commit -m "Change for CaseOnlyRenameFileAndChangeBranches"
%GIT_EXE% checkout FunctionalTests/20201014
%GIT_EXE% checkout tests/functional/CaseOnlyRenameFileAndChangeBranches
%GIT_EXE% status
%GIT_EXE% reset --hard -q HEAD
%GIT_EXE% status

popd


GVFS.exe unmount "%REPO_ROOT%" --internal_use_only "{\"ServiceName\":\"Test.GVFS.Service\",\"StartedByService\":false,\"MaintenanceJob\":null,\"PackfileMaintenanceBatchSize\":null}"

[derrickstolee]

Thanks for this contribution! The VFS for Git functional tests pass with this change, so I'm happy to take it!

[neerajsi-msft]

Thanks Derrick! Please feel free to merge whenever you're ready (I guess I don't have write access to this repo even after linking my corp account).