Internal Server Error when restoring packages

[ADD-Juan-Perez]

We are starting to use BaGet as our package repository instead of Azure DevOps Server. After one month of using it, builds started to fail due to Internal Server Errors when restoring packages. Looking at the BaGet logs we saw that this exception is being logged many times:

Category: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer
EventId: 2
RequestId: 80000839-0003-9500-b63f-84710c7967bb
RequestPath: /v3/package/mypackage/2022.1.11.9/mypackage.2022.1.11.9.nupkg
SpanId: |b99f9b84-43bfa581a4123fff.
TraceId: b99f9b84-43bfa581a4123fff
ParentId: 

Connection ID "10736581526146779187", Request ID "80000839-0003-9500-b63f-84710c7967bb": An unhandled exception was thrown by the application.

Exception: 
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithPropagationAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(DbContext _, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at BaGet.Core.PackageService.TryUpdatePackageAsync(String id, NuGetVersion version, Action`1 action, CancellationToken cancellationToken) in /_/src/BaGet.Core/PackageService.cs:line 145
   at BaGet.Core.DefaultPackageContentService.GetPackageContentStreamOrNullAsync(String id, NuGetVersion version, CancellationToken cancellationToken) in /_/src/BaGet.Core/Content/DefaultPackageContentService.cs:line 65
   at BaGet.Web.PackageContentController.DownloadPackageAsync(String id, String version, CancellationToken cancellationToken) in /_/src/BaGet.Web/Controllers/PackageContentController.cs:line 49
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at BaGet.Web.OperationCancelledMiddleware.Invoke(HttpContext context) in /_/src/BaGet.Web/OperationCancelledMiddleware.cs:line 57
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

To solve this, we had to reduce the number of packages stored in the database, from 100.000 to 50.000 packages, and the Internal Server Errors disappeared although that exception is still being recorded in the BaGet log. The reason so many packages are stored is because we have a continuous integration process involving many builds to publish and restore packages.

For more information, BaGet is installed in an IIS on a Windows Server 2019 Standard and the database server is a SQL Server 2019.

Is there anything we can do without modifying the code to avoid this exception? Change the database server to PostgreSQL or MySQL?

Thank you in advance for your dedication to this project.

[inkysquid]

Hi, I may have just fixed this for you. #716

Otherwise, if you don't care about updating the download count, you could simply comment out the call

await _packages.AddDownloadAsync(id, version, cancellationToken);

in BaGet.Core.DefaultPackageContentService

[prcdpr]

We use private NuGet feed for providing binary updates to worker machines and I had to put download count increment into try/catch block to stop error propagation which caused HTTP 500 errors.

When dozens of downloads are initiated at the same moment, the HTTP 500 is guaranteed to occur.