Unexpected PublishConfirmExceptions with 2.x

[cocowalla]

I'm seeing some odd behaviour when the broker goes down when using using pub/sub.

After the broker comes back up, every single publish results in an immediate PublishConfirmException with message The broker did not send a publish acknowledgement for message # within 0:00:10.

Messages are still being delivered though.

[pardahlman]

Thanks for reporting this. I must admit that I haven't tried this myself, but it is something that definitely should work. It is very possible that the publish sequence resets after the connection goes down and recovers. There is always the option to disable publish confirms alltogether by calling you outing operations (publish etc) with .UsePublishAcknowledge(false).

[cocowalla]

Yes, after the broker restarts I see a single entry in PublishAcknowledgeMiddleware with NextPublishSeqNo set back to 1.

I'd like to keep publish confirmations if at all possible :)

[pardahlman]

I've looked into this and adjusted the internal counter for the publish sequence to reset if the sequence has for some reason been reset.

This solution seems to be robust for multiple concurrent publishers using the same channel.

[cocowalla]

This seems to work, unless any of the publish tasks were cancelled while the broker was down.

For example, if the publish call looks something like this (if the publish doesn't succeed in 5s, cancel the task):

using (var timeoutCts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
{
    await bus.PublishAsync(message, configAction, timeoutCts.Token);
}

When the broker comes back up, every subsequent publish results in an immediate PublishConfirmException.

[pardahlman]

Are you sure? I get an OperationCanceledException, which is expected as the task has been cancelled. Personally I think this is odd behavior - but that is how cancellation is handled.

try
{
  await _client.PublishAsync(new ValueCreationFailed(), token: timeoutCts.Token);
}
catch (OperationCanceledException) { }

[cocowalla]

Gah, my mistake, I was working with the wrong source - your commit has fixed the issue in all cases!

[xwrs]

Hi. I have faced the same issue in rc5. With moderate load of ~400 events. Singular publishes work fine. Increasing wait time for publish confirm to 1 minute is not helping. Events are all delivered though. Temporarily disabled publish ack with .UsePublishAcknowledge(false) but it would be nice to understand why it is happening.

"Depth": 0,
"ClassName": "",
"Message": "The broker did not send a publish acknowledgement for message 281 within 0:01:00.",
"Source": "System.Private.CoreLib",
"StackTraceString": " at RawRabbit.Operations.Publish.Middleware.PublishAcknowledgeMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Operations.Publish.Middleware.ReturnCallbackMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Pipe.Middleware.PooledChannelMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Pipe.Middleware.HeaderSerializationMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Pipe.Middleware.ExchangeDeclareMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.BusClient.InvokeAsync(Action1 pipeCfg, Action1 contextCfg, CancellationToken token)\n at .......

P.s. @pardahlman thanks for awesome framework