Value cannot be null. (Parameter 'key') issue on Concurrent/Duplicate updates

[gungorenu]

Hello,

I am getting this exception when two operations try to do update on same row using EFCore 8.0.5.

I fixed the issue on my side but I created this issue for a better error message on what the problem is.

Error:

[19#15.3 = ANECDOTE]: Value cannot be null. (Parameter 'key')
System.ArgumentNullException: Value cannot be null. (Parameter 'key')
   at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
   at Microsoft.EntityFrameworkCore.Update.ModificationCommand.<GenerateColumnModifications>g__HandleJson|41_4(List`1 columnModifications, <>c__DisplayClass41_0&)
   at Microsoft.EntityFrameworkCore.Update.ModificationCommand.GenerateColumnModifications()
   at Microsoft.EntityFrameworkCore.Update.ModificationCommand.get_ColumnModifications()
   at Microsoft.EntityFrameworkCore.Update.Internal.CommandBatchPreparer.CreateCommandBatches(IEnumerable`1 commandSet, Boolean moreCommandSets, Boolean assertColumnModification, ParameterNameGenerator parameterNameGenerator)+MoveNext()
   at Microsoft.EntityFrameworkCore.Update.Internal.CommandBatchPreparer.BatchCommands(IList`1 entries, IUpdateAdapter updateAdapter)+MoveNext()
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Talepreter.Operations.Grains.EntityGrain`3.<>c__DisplayClass5_0.<<Execute>b__0>d.MoveNext() in /app/Operations/Talepreter.Operations.Grains/EntityGrain.cs:line 96
--- End of stack trace from previous location ---
   at Talepreter.Operations.Grains.GrainWithStateBase`1.SaveState(Func`2 action) in /app/Operations/Talepreter.Operations.Grains/GrainWithStateBase.cs:line 35
   at Talepreter.Operations.Grains.EntityGrain`3.Execute(CommandId commandId) in /app/Operations/Talepreter.Operations.Grains/EntityGrain.cs:line 87

Repro: unfortunately it is a little sporadic. error occurred randomly on a data pair I did not realize it was duplicate target (I will explain). EFCore did not give this error on other data pairs so I cannot give you clear repro steps but I will try to explain.

I use Orleans grains. Imagine a grain A:123. I send two modification requests to the actor/grain. First will update columns A, B, C. Second will update A, B, C, D. order of requests are a little random, fetched from a queue, one of them can be before other. it is not necessary to be different either, I have realized sometimes request data is full duplicate.
so columns to update will look like below

Request1: A + B + C
Request2: A + B + C + D
or Request3: A + B + C

I use scopes and fetch a new DBContext for each incoming call (I use IIncomingGrainCallFilter) so I am not sure why I am getting this error but that is out of scope for this ticket perhaps (it could be my issue or Orleans related).

I cleared the duplicate updates and everything works, and interestingly if I revert the data back to duplicate sometimes it works, sometimes fails with same error. I think order of requests processed matters in case requests have different update columns, not same.

When I had the error, restarting svcs or DB does not matter, I always get the error on same data pair. surprisingly I realized there are other data pairs that simply passed without issue. it gets stuck on some specific data pairs, maybe related with columns.

Note: my columns are regular Guid, int and string columns/fields on objects. I have one Json field on the failing type but I am not sure if it matters because the data columns I listed above did not include that field to be updated. generally there are no changes on that json field. it could be related though

Unfortunately I cannot provide a repo for reproduction. code of my project is here though. https://github.com/gungorenu/talepreter-public sadly it is just too big to describe it. maybe you can detect registration error straight away, I put link below

DBContext: code is here for registering model.

https://github.com/gungorenu/talepreter-public/blob/main/Talepreter/DB/Talepreter.AnecdoteSvc.DBContext/AnecdoteSvcDBContext.cs

DB is SqlServer in docker container

Expectation: just error message. it would be very helpful if exception explains why I am getting this error (duplicate requests to update same record and different columns). for example if it had some hints describing concurrent update or something like that then I could retry with new options or dbcontext or similar ideas. with error message as title it is hard to retry. the request comes from a queue so I have opportunity to retry it anyway but again, I need to know if error occurs because of concurrency or multiple updates.

Orleans Stuff: state of grain is not same with data that is used and failing in above exception. so grain state is stored in DB A but error occurs on DB B. No other grain or flow updates the same data. every grain has a data row in DB dedicated to them (not state data, for a specific reason they are separate). silo is single instance, so no two silos messing with each other. that specific grain lives in one silo only.

Thanks