Fix for deadlock issue #3682

[rajagopalanand]

What does this PR do?

This PR is a fix for #3682. In some instances, mem.Alerts.Subscribe() and store.gc() can get deadlocked

1. Lock acquisition in store.Alerts.gc():
   • The method store.Alerts.gc() acquires a lock on its internal mutex store.Alerts.mtx
   • While holding the lock, it then calls the callback function which will try to acquire a lock on mem.Alerts.mtx
2. Concurrent Execution:
   • mem.Alerts.Subscribe() acquires a lock on its internal mutex mem.Alerts.mtx and calls store.Alerts.List()
3. Deadlock situation:
   • Callback function tries to acquire a lock on mem.Alerts.mtx. However this lock is already being held by mem.Subscribe()
   • Similarly mem.Subscribe() cannot proceed because store.List() cannot acquire lock (store.Alerts.mtx) because it is being held by store.gc()

Another way of summarizing this is store.Alerts.gc() was holding the lock until callback function completed which in turn was waiting to acquire the lock. Callback function could not acquire the lock because Subscribe() was holding the lock. Subscribe() cannot progress because it calls store.Alerts.List() which was waiting for lock acquisition which was being held by store.Alerts.gc(). This fix releases the lock held by store.Alerts.gc() prior to calling the callback function

[gotjosh]

@rajagopalanand thank you very much for your contribution - can you please fix the linter?

[gotjosh]

This is a duplicate of https://github.com/prometheus/alertmanager/pull/3648/files, right?

[rajagopalanand]

This is a duplicate of https://github.com/prometheus/alertmanager/pull/3648/files, right?

Was not aware of this but I will take a look

[rajagopalanand]

This is a duplicate of https://github.com/prometheus/alertmanager/pull/3648/files, right?

Was not aware of this but I will take a look

The other PR addresses issues more than just the deadlock issue mentioned in #3682. If this can be reviewed and merged, then when the other PR gets reviewed and merged it can address race conditions and can retain a test from this PR

[gotjosh]

Thank you so much for the detailed response and PR description.

This LGTM, but there's one more thing we need to do. Please take a look at my comment.

[gotjosh]

We need to stop passing []*types.Alert and instead now send a slice of values to the callback.

Suggested change

	var resolved []*types.Alert
	var resolved []types.Alert

[rajagopalanand]

I can submit a separate PR for this