object/put: fix concurrent PUT data corruption

If ants pool is busy and cannot take task, early `return` without `wg.Wait()`
leads to `iterateNodesForObject`'s `return` and all the buffers for binary
replication from now may be reused while are still in use by the other routines
inside the pool. Wait for WG and try other nodes more instead, it also can
increase the rate of successful PUTs at high loads. Closes #2978.

Signed-off-by: Pavel Karpy <[email protected]>

CHANGELOG.md

@@ -35,6 +35,7 @@ attribute, which is used for container domain name in NNS contracts (#2954)
 - Panic in event listener related to inability to switch RPC node (#2970)
 - Non-container nodes never check placement policy on PUT, SEARCH requests (#3014)
 - If shards are overloaded with PUT requests, operation is not skipped but waits for 30 seconds (#2871)
+- Data corruption if PUT is done too concurrently (#2978)
 
 ### Changed
 - `ObjectService`'s `Put` RPC handler caches up to 10K lists of per-object sorted container nodes (#2901)

pkg/services/object/put/distributed.go

@@ -331,7 +331,7 @@ func (x placementIterator) iterateNodesForObject(obj oid.ID, f func(nodeDesc) er
 					if e, _ := lastRespErr.Load().(error); e != nil {
 						err = fmt.Errorf("%w (last node error: %w)", err, e)
 					}
-					return errIncompletePut{singleErr: err}
+					wg.Wait()
 				}
 			}
 			wg.Wait()