Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to remove items from persistent queue storage if the storage is full #7198

Closed
swiatekm opened this issue Feb 14, 2023 · 2 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@swiatekm
Copy link
Contributor

swiatekm commented Feb 14, 2023

Describe the bug
The persistent queue removes items from storage after they're successfully exported. This removal happens in a transaction which also updates the list of currently dispatched items. Depending on the implementation details of the underlying storage, this transaction may fail if the storage device is full.

As a result, we can take items out of the queue, but they're not actually removed from the storage, and no new items can be put in.

Steps to reproduce
See the unit test in the linked PR.

Additional context
I've confirmed that filestorage can behave this way via the following test: open-telemetry/opentelemetry-collector-contrib@dbe3105. I suspect that this will be true of any transactional storage engine, as some amount of transaction data needs to be persisted to disk before it can be committed.

How often this can happen in practice is difficult to estimate. It depends heavily on how the size of queue items aligns with available disk space. Anecdotally, I've seen it happen during an incident, on a volume with multiple queues sharing space.

@swiatekm swiatekm added the bug Something isn't working label Feb 14, 2023
@swiatekm
Copy link
Contributor Author

CC @djaglowski for the storage part. Anyone who sees this with the necessary power, please assign this to me.

@swiatekm
Copy link
Contributor Author

Fixed in #7396

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant