LibGit2Sharp.LockedFileException: The index is locked. This might be due to a concurrent or crashed process #1031
Comments
|
The very next build through TeamCity failed with the exact same message and stack trace. I'm going to revert to |
|
Id switch to the exe, run it with /updateAssemblyInfo before building. We would have to grab a mutex, and block builds until the first task has finished calculating the version then unblock the others and they get the version from the cache. |
|
Perhaps we could use |
|
Im happy with that |
|
any updates here? |
|
@bbowman: I'm sorry, but nothing has happened besides agreeing on a solution that is yet to be implemented. We need someone in the team to have time to implement this, or get someone else to submit a PR. I unfortunately have my hands full. Perhaps you or @michaelnoonan can submit a PR? |
|
any update on this? builds take 4x as long without parallelism :( |
|
@jnevins-gcm: Sorry, there's nothing new to report on this. If you are able to provide a pull request that fixes this as described in #1031 (comment), it will be merged quickly provided it passes all tests. |
|
Thanks for the update. Even if I fix, how would I manage to get that into GitVersionTask? Is there any short term workaround I could do with retries? |
|
@jnevins-gcm, if you submit a pull request to this repository, it will be merged and a new package of GitVersionTask will show up on NuGet that you can use. |
|
Now that we've upgraded to LibGit2Sharp version 0.26 in #1713, I hope this problem might be fixed. Please test with the latest v5 beta and reopen if the problem persists. |
|
Hi, We are having the same issue using the latest beta version (5.0.0-beta5.11). |
|
Also hitting this issue on linux and Windows (but not MacOS) build agents using GitVersion 5.0.1 |
|
@StevenDevooght and @borrillis, can you please tell me more about your build environments that might help us figure out why this is happening? |
|
@asbjornu This is the repo and branch I am currently testing with: https://github.com/axiom3d/axiom/tree/azure-pipelines and I am only seeing it fail in the CI/CD builds in Azure DevOps (https://dev.azure.com/axiom3d/Axiom/_build/results?buildId=37&view=results), locally it is working using the command |
|
@StevenDevooght, are you also seeing this on Azure DevOps? |
|
@asbjornu, We have been having the same issue in our CI/CD builds on our private GitLab instance |
|
We also have the issue when building on Azure DevOps. Locally we haven't experienced any issues. |
|
Also see it in Azure Dev Ops, intermittently. Never happens when we run the cake build locally though. |
|
We also have this issue. dotnet core 2.2.203 Often it's "build->fail, build->fail, build->fail, build->succeed". Doesn't ever seem to happen locally when building through VS. We've tried adding "-p:maxCpuCount=1" to the build command to see if we could force it to build without parallelization (thinking that might have something to do with it), but no dice. edit: also this doesn't just happen on build, it also sometimes happens on "dotnet test" as well. |
|
@dieterv, since you're able to reproduce this in a local GitLab instance, we might have a chance to figure out why this is occurring. Are there any parallelization going on in your GitLab build that might cause this? |
|
@asbjornu, I'm operating a single Gitlab build runner that accepts a single build job at a time ( I've been experimenting with our setup these last couple of weeks and seem to have found something interesting. Since these changes we have seen 40 builds succeed without encountering the issue. Either we're simply extremely lucky or one or a combination of these changes "fixed" it for us. I was going to wait a bit longer to be sure, but well, maybe this information can help...
mode: ContinuousDeployment
branches:
master:
tag: master
increment: Patch
prevent-increment-of-merged-branch-version: true
track-merge-target: false
release:
tag: beta
increment: Patch
prevent-increment-of-merged-branch-version: true
track-merge-target: false
feature:
tag: useBranchName
increment: Inherit
prevent-increment-of-merged-branch-version: true
track-merge-target: false
ignore:
sha: []Before we had: diff --git a/gitversion.yml b/gitversion.yml
index 11204d4c..4bacb5b4 100644
--- a/gitversion.yml
+++ b/gitversion.yml
@@ -1,32 +1,19 @@
mode: ContinuousDeployment
-tag-prefix: '[vV]'
-continuous-delivery-fallback-tag: ci
-major-version-bump-message: '\+semver:\s?(breaking|major)'
-minor-version-bump-message: '\+semver:\s?(feature|minor)'
-patch-version-bump-message: '\+semver:\s?(fix|patch)'
-no-bump-message: '\+semver:\s?(none|skip)'
-legacy-semver-padding: 4
-build-metadata-padding: 4
-commits-since-version-source-padding: 4
-commit-message-incrementing: Enabled
branches:
master:
- mode: ContinuousDeployment
tag: master
increment: Patch
prevent-increment-of-merged-branch-version: true
track-merge-target: false
release:
- mode: ContinuousDeployment
tag: beta increment: Patch prevent-increment-of-merged-branch-version: true track-merge-target: false feature: - mode: ContinuousDeployment tag: useBranchName increment: Inherit - prevent-increment-of-merged-branch-version: false + prevent-increment-of-merged-branch-version: true track-merge-target: false ignore: sha: []Hope this helps! |
|
One thing I'm trying as a workaround right now (Azure DevOps Pipeline) is to do this:
#Removes GitVersionTask from all csproj files before we build
Get-ChildItem .\*.csproj -Recurse | % {
[Xml]$xml = Get-Content $_.FullName
$xml | Select-Xml -XPath '//*[@Include="GitVersionTask"]' | ForEach-Object{$_.Node.ParentNode.RemoveChild($_.Node)}
$xml.Save($_.FullName)
}
Since we don't use AssemblyInfo.cs at all (dotnet core SDK), this seems to work. Whether it will fix the issue is unknown since it's wildly unpredictable as to when it happens. edit: added the powershell script I used. |
This is working for us - the only annoying thing is that package version does not contain the full semver for commits ahead in the same way the gitversion package does. i.e. 1.9.0-alpha0123 Vs. 1.9.0-alpha.122. Much like the others, we're seeing this using on Azure DevOps, using the .Net Core build step. We're building a solution with 19 projects, of which around 16 have the gitversion NuGet package. edit: using "Automatic package versioning" option on the .Net Core task with the Pack option, we can set the Environment Variable option to be GITVERSION_FullSemVer. This is working well. |
|
We're experiencing this too on a local GitLab instance when building solutions which contain multiple projects that reference My workaround:Put the following in each project file (or a <PropertyGroup Condition=" '$(GitVersion_SemVer)' != ''">
<GetVersion>false</GetVersion>
<WriteVersionInfoToBuildLog>false</WriteVersionInfoToBuildLog>
<UpdateAssemblyInfo>false</UpdateAssemblyInfo>
<Version>$(GitVersion_FullSemVer)</Version>
<VersionPrefix>$(GitVersion_MajorMinorPatch)</VersionPrefix>
<VersionSuffix Condition=" '$(UseFullSemVerForNuGet)' == 'false' ">$(GitVersion_NuGetPreReleaseTag)</VersionSuffix>
<VersionSuffix Condition=" '$(UseFullSemVerForNuGet)' == 'true' ">$(GitVersion_PreReleaseTag)</VersionSuffix>
<PackageVersion Condition=" '$(UseFullSemVerForNuGet)' == 'false' ">$(GitVersion_NuGetVersion)</PackageVersion>
<PackageVersion Condition=" '$(UseFullSemVerForNuGet)' == 'true' ">$(GitVersion_FullSemVer)</PackageVersion>
<InformationalVersion Condition=" '$(InformationalVersion)' == '' ">$(GitVersion_InformationalVersion)</InformationalVersion>
<AssemblyVersion Condition=" '$(AssemblyVersion)' == '' ">$(GitVersion_AssemblySemVer)</AssemblyVersion>
<FileVersion Condition=" '$(FileVersion)' == '' ">$(GitVersion_AssemblySemFileVer)</FileVersion>
<RepositoryBranch Condition=" '$(RepositoryBranch)' == '' ">$(GitVersion_BranchName)</RepositoryBranch>
<RepositoryCommit Condition=" '$(RepositoryCommit)' == '' ">$(GitVersion_Sha)</RepositoryCommit>
</PropertyGroup>
</Project>Then run Edit: PowerShell Scriptgitversion /output json | Out-String | ConvertFrom-Json | ForEach-Object { foreach ($item in $_.PSObject.properties) { Set-Item -Path "env:GitVersion_$($item.Name)" -Value $item.Value } }Edit 2You also need to set the version properties (which is normally performed within the |
|
Just as a followup. The approach I posted back in October has been running fine since. I had to do a little bit of adjustments to the way the GitVersion.yml was setup because it seemed to behave just slightly differently than when using the msbuild way, but otherwise it's been running without issue since. So at least now things seem stable and reliable. Having to only run gitversion once at the beginning seems to be the way to go. |
|
During my inquiries I realized that each task is executing I'd suggest to try to restructure the build tasks, so that:
|
|
@admin-simeon thanks for this solution. I found some issues with it, maybe we can work together to get a fully working workaround:
|
|
|
Somehow I feel sad about GitVersionTask. It seems not thought to the end. It does not take into account a multi project solution and even less it is built for concurrency. Why do you rely on locking machanism of LibGit2Sharp? It does not seems to work. So just don't rely on it. Caching is nice, but as long you need make lookup git-metadata for each call to GetVersion to look for such a cache on MSBuild-level (you call GetVersion each time), you will run into these lock exception at some time. Currently I am working on an add-in, that just disables the task GetVersion of the package GitVersionTask and does this stuff by itself with the help of GitVersion.exe. What's the reason you not using GitVersion.exe by yourself in GitVersionTask(.MSBuild) (@asbjornu)? GitVesionTask could carry it on by itself and could replace the current implementation of the task GetVersion with some clean lines of code. In such an implementation it would be easy to implement a global lock mechanism and even to implement a kind of global identifier for caching purposes of solution with multi projects that rely heavily on GitVersionTask. |
|
I'm sad about the current state of GitVersionTask too, @teroneko. Parallelism is something Microsoft introduced as default in later versions of .NET, which is why the issue with locking is "new". It has always existed, though. I think the proper solution to this problem may come with version 6 of GitVersion (see #2275) , where we modularize and separate things so a build can be created from a sequence of steps instead of just a single GitVersion step as is the current way of operation. A suggested way to do this in version 6 would be:
I think this should work and that it is doable. The only optimization I would like to do, if possible, is to add some sort of hook into Fixing this in version 5 would require someone to submit a pull request that provides a working file locking implementation. Until such a PR exists, this issue is going to stay open, I'm afraid. |
|
As I am in working for an add-in that also allows nested GitVersion.yml's, but same .git-folder, I would agree to contribute a proper locking implementation. I see two options: Mutex or a file lock with a simple "sleep-and-retry-until-lock-release"-implementation. I think I will use last option. May I ask you, if your interpretation of dotnet build belongs to a solution or a set of projects, or only to a single project, like GitVersionTask is designed? Because then, you are going to shift logic from GitVersionTask.MSBuild to GitVersion(Core), for modularity and simplicity right? I just want to give you here three scenarios in which GitVersionTask is necessary.
As you can see, it is necessary, that each project should have a package reference to GitVersionTask for its independency. To have the calculation and such logic like described (scope resolving and scoped caching) above in one product, let call it GitVersion.exe, would be great. I red your proposal, and I think you want GitVersionTask behave the same like before, but with just moved logic like I described above, right? |
🙏 👏 🙏 👏 🙏
As long as the method chosen is cross-platform, any method will do.
In version 6 we are moving towards executing
Yes, absolutely.
That's the plan. Each GitVersionTask instance don't need to calculate their own version number, as the result will be the same each and every time.
Yes. |
|
I have implemented a simple and generic lock file utility. My image of current GitVersion implementation is, that each executable project implements his own concrete logic to grab the native binaries to be able to work with GitVersionCore. By the way here is the code related to file locking I want to contribute: public static class LockFileTools
{
public const FileMode DefaultFileMode = FileMode.OpenOrCreate;
public const FileAccess DefaultFileAccess = FileAccess.ReadWrite;
public const FileShare DefaultFileShare = FileShare.None;
public static bool TryAcquire(string filePath, out FileStream? fileStream, FileMode fileMode = DefaultFileMode,
FileAccess fileAccess = DefaultFileAccess, FileShare fileShare = DefaultFileShare)
{
filePath = filePath ?? throw new ArgumentNullException(nameof(filePath));
try {
fileStream = File.Open(filePath, fileMode, fileAccess, fileShare);
return true;
} catch (Exception error) when (error.GetType() == typeof(IOException)) {
fileStream = null;
return false;
}
}
public static bool WaitUntilAcquired(string filePath, out FileStream? fileStream, FileMode fileMode = DefaultFileMode,
FileAccess fileAccess = DefaultFileAccess, FileShare fileShare = DefaultFileShare, int timeoutMilliseconds = Timeout.Infinite)
{
FileStream spinningFileStream = null;
var spinHasBeenFinished = SpinWait.SpinUntil(() => {
return TryAcquire(filePath, out spinningFileStream, fileMode: fileMode, fileAccess: fileAccess, fileShare: fileShare);
}, timeoutMilliseconds);
if (spinHasBeenFinished) {
fileStream = spinningFileStream ?? throw new ArgumentNullException(nameof(spinningFileStream));
return true;
} else {
fileStream = null;
return false;
}
}
public static bool WaitUntilAcquired(string filePath, out FileStream? fileStream, FileMode fileMode = DefaultFileMode,
FileAccess fileAccess = DefaultFileAccess, FileShare fileShare = DefaultFileShare, TimeSpan? timeout = null)
{
timeout = timeout ?? Timeout.InfiniteTimeSpan;
var timeoutInMilliseconds = Convert.ToInt32(timeout.Value.TotalMilliseconds);
return WaitUntilAcquired(filePath, out fileStream, fileMode: fileMode, fileAccess: fileAccess, fileShare: fileShare, timeoutMilliseconds: timeoutInMilliseconds);
}
} |
|
@teroneko, that's awesome! Would it be possible to put a lock on the cache file GitVersion already stores within the I'd also like to see a non- |
|
It is just the abstraction, and are we really talking about the name? If you want read that below. About your question, it would be somehow possible to lock the folder, but it seems not very straightforward, so we should keep on file locking. Please help me, is it correct, that you associate a cache by meta data from git? Because then you couldn't lock on cache before you didn't got the data from .git with LibGit2Sharp. So a simple .git/gitversion_cache/gitversion.lock would do it too. You have to rely on the fact, that only you are using this directory, so why worry about any behaviour that can interrupt you. Because then you would need to do a lot more to ensure a "atomic" action for access git and cache to prevent any influence from outside. My view of this is: You always have first an abstraction of something, then you are going to concrete something. In this case you have a generic file lock mechanism, somewhere written now to LockFileTools. You can call it whatever you want like I_Can_Lock_A_File. Tools refers for me a kind of static construction that applies on something. In this case: Lock(ing a )File. As the string is too generic to be an extension, Because of this, I decided to enclose it to the topic Lock(ing a )File. Now based on code, it will will happen to craft a class, for DI, or whatever, to wrap the static functions to reduce arguments of course and increase encapsulation and complexity to serve its actual purpose. |
|
Of course, you are right. Yes, the cache file requires some information from LibGit2Sharp, specifically the SHA of the Preferably, we would have a My distaste for
Even though we provide a The IOC work laid down by @arturcic is, however, a step in the direction of making GitVersion easier to consume as a library. So it is on the roadmap, although not a high-priority goal. With version 6, I think we might be closer and during the refactorings we are going to do over the lifetime of version 6, I hope we might be in a good place by version 7, where GitVersion should have its own core domain model independent of LibGit2Sharp and where all implementations of everything infrastructure related can be injected through the DI container. |
|
Thanks for your answering. When I find time, I will try to look around the code in GitVersionCode and seek for a possiblity to wrap each single action that involves GitVersion.yml, caching or LibGit2Sharp by an file lock. Another approach would be to lock on startup and unlock on exit. But that is quick and dirty and reminds me of the time, where you instantiated one SqlConnection during lifetime. 😃 Regarding to your distate of Tool and Utility. I can understand you and agree with you, that such a Tool whatever static prupose it does serve, can be lost quickly. But a library is not a library without such functions. Those functions cannot be related to only one purpose easily, So when you have two classes that serves completely its own purposes, but have both one method that share a bit of the same logic like Directory.Exists or File.Exists, then you should outsource them definetely to let them become Directory.Exists and File.Exists. My taste for Tools (or Utilities) are coming for this reason, that I don't want to collide with existing classes from System.*, but want to state its relation to them. So the name LockFileTools comes more or less by habit. When I think more over it: FileLocker does associate a kind of encapsulation between these static functions, so even when I would call it FileLocker: The class itself does not provide a file locker, but each of its static functions. Moreover they do not provide a file locker instead they do lock a file it without an instanstiated context so the name would irritate somehow. But this would change if FileLocker would be instantiable and would require a file path and provides methods like "Lock", then the class itself would be a file locker and therefore named FileLocker. Haha, but I do respect your programming style. To your last words: I am looking forward that this library will become a strong library. 😊 |
Sounds great! 👍
It's a viable choice if other options are too difficult. It's much better than crashing and will still yield better performance with parallelization than a serial build would.
I don't know either.
Good. We can instead harmonize this a bit in v6 and gather everything GitVersion related into a
Awesome!
I think we should be able to surface this bug with enough projects in a single solution, so a very simple repository with just a bare minimum solution with perhaps 100 projects all using GitVersionTask should be able to consistently reproduce this bug, I'd assume. I've set up and invited you to GitVersion.TestCases so we can collaborate on the reproduction there. I'm thinking this one repository can have one branch per test case, starting with this parallelization problem.
My point exactly! I have a general distaste for
So do I! 😃 |
|
Guys, the builds failing because of GitVersion creating |
|
Pending PR #2581 has some mitigations (retries) for this issue. Makes a big difference at least for my builds. |
|
🎉 This issue has been resolved in version 5.6.5 🎉 Your GitReleaseManager bot 📦🚀 |
|
@arturcic I don't think this issue should be closed. I'm currently using GitVersion.MsBuild 5.6.8 and I'm still seeing The problem is that there is a race condition in the NormalizeGitDirectory where two projects building at the same time will both attempt to create the needed branches, and one will fail with Seems like this code needs to be more resilient to failures, because if it just attempted it again, it would see that the required branch does exist. EDIT: Instead of a retry, seems like a mutex would be a good way to ensure that the normalization code is not being run in parallel since that's the part that needs to be protected. |
|
Unfortunately I'm getting this error also using GitVersion.MsBuild 5.6.8 on my TeamCity 2020.1.3. |
|
I have opened #2669 with my attempt to fix this problem. |
|
Yup, issue is still there with v5.6.9 (GitLab CI) |
|
@bording i will have a look on the PRs in progress and see when we can release |
and others
Wasn't sure if this was an issue or whether we shouldn't be using
msbuild /mto enable concurrent builds.This occurs on TeamCity
10.0.0building ~30 projects. We've only just upgraded toGitVersionTask.3.6.3fromGitVersionTask.3.6.1which had thegitversion_cacheissue.In the meantime I'll try some more builds and see if it recurs frequently.
The text was updated successfully, but these errors were encountered: