Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix database dead locks in Aggregator #517

Closed
jpraynaud opened this issue Sep 20, 2022 · 1 comment · Fixed by #521
Closed

Fix database dead locks in Aggregator #517

jpraynaud opened this issue Sep 20, 2022 · 1 comment · Fixed by #521
Assignees
Labels
bug ⚠️ Something isn't working D-medium Difficulty: medium P-high Priority: high

Comments

@jpraynaud
Copy link
Member

jpraynaud commented Sep 20, 2022

Issue

We have noticed some database locks occurring on the GCP Aggregator which these error messages:

  • cannot start a transaction within a transaction
  • database is locked

⚠ The problem occurred after activating the pruning feature of the stores #516

Here are the full logs:

mithril-aggregator_1                | {"msg":"STATE MACHINE: an error occurred: ","v":0,"name":"slog-rs","level":50,"time":"2022-09-20T14:44:11.436860471Z","hostname":"1db9e5f989b2","pid":1,"error":"StoreError(AdapterError(QueryError(Error { code: Some(1), message: Some(\"cannot start a transaction within a transaction\") })))"}
mithril-aggregator_1                | {"msg":"… Cycle finished, Sleeping for 60000 ms","v":0,"name":"slog-rs","level":30,"time":"2022-09-20T14:44:11.436936131Z","hostname":"1db9e5f989b2","pid":1}
mithril-aggregator_1                | {"msg":"⇄ HTTP SERVER: certificate_pending","v":0,"name":"slog-rs","level":20,"time":"2022-09-20T14:44:11.436978054Z","hostname":"1db9e5f989b2","pid":1}
mithril-aggregator_1                | {"msg":"certificate_pending::error","v":0,"name":"slog-rs","level":40,"time":"2022-09-20T14:44:11.437023811Z","hostname":"1db9e5f989b2","pid":1,"error":"AdapterError(ParsingDataError(Error { code: Some(5), message: Some(\"database is locked\") }))"}
mithril-aggregator_1                | {"msg":"⇄ HTTP SERVER: certificate_pending","v":0,"name":"slog-rs","level":20,"time":"2022-09-20T14:44:12.626648308Z","hostname":"1db9e5f989b2","pid":1}
mithril-aggregator_1                | {"msg":"certificate_pending::error","v":0,"name":"slog-rs","level":40,"time":"2022-09-20T14:44:12.626705143Z","hostname":"1db9e5f989b2","pid":1,"error":"AdapterError(ParsingDataError(Error { code: Some(5), message: Some(\"database is locked\") }))"}

Workaround

Restart the container

@ghubertpalo
Copy link
Collaborator

The tactic is to upgrade the SQLite version in order to support the RETURNING keyword. This would allow not to use a transaction.
Update the Signer & Aggregator documentation + SPO guide to inform about SQLite minimum version (3.35)
Remove the useless harness test in SQLiteAdapter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug ⚠️ Something isn't working D-medium Difficulty: medium P-high Priority: high
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants