-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support dual mode data storage #369
Conversation
7911837
to
cb3e5e2
Compare
b5351c2
to
fae3f73
Compare
Allow for dual mode, where data is written to the primary, but the secondary is read in order to 'back fill'. For simplicity, Dual presumes that the primary data store is Bigtable and the secondary is DynamoDb. This presumes that the `db_settings` contains serialized settings for both data stores. Closes: SYNC-3452
9d65c01
to
c00cb44
Compare
… feat/3452-dual_data
… feat/3452-dual_data
This flag makes writing to the secondary an active feature. This is required due to the potential for clients to be arbitrarily assigned to the dual mode storage or the older single mode storage. This flag is set to `true` as the default since we want both data stores active until we have sufficient cross coverage.
… feat/3452-dual_data
… feat/3452-dual_data
… feat/3452-dual_data
… feat/3452-dual_data
… feat/3452-dual_data
… feat/3452-dual_data
…opush-rs into feat/3452-dual_data
… feat/3452-dual_data
… feat/3452-dual_data
…opush-rs into feat/3452-dual_data
… feat/3452-dual_data
…opush-rs into feat/3452-dual_data
dd93d0c
to
036ba4b
Compare
036ba4b
to
577ca3f
Compare
… feat/3452-dual_data
Cargo.toml
Outdated
sentry-core = { version = "0.31", default-features = false, features = [ | ||
"client", | ||
] } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this client
feature is being used anywhere?
sentry-core = { version = "0.31", default-features = false, features = [ | |
"client", | |
] } | |
sentry-core = { version = "0.31" } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do wonder if i should audit the libraries and features we're pulling in. I know there's a cargo feature that allows for that, but it needs nightly...
autopush-common/src/db/dual/mod.rs
Outdated
#[derive(Clone, Debug, Default, Deserialize)] | ||
pub struct DualDbSettings { | ||
#[serde(default)] | ||
primary: DbSettings, | ||
#[serde(default)] | ||
secondary: DbSettings, | ||
#[serde(default = "set_true")] | ||
write_to_secondary: bool, | ||
} | ||
|
||
fn set_true() -> bool { | ||
true | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#[derive(Clone, Debug, Default, Deserialize)] | |
pub struct DualDbSettings { | |
#[serde(default)] | |
primary: DbSettings, | |
#[serde(default)] | |
secondary: DbSettings, | |
#[serde(default = "set_true")] | |
write_to_secondary: bool, | |
} | |
fn set_true() -> bool { | |
true | |
} | |
#[derive(Clone, Debug, Deserialize)] | |
pub struct DualDbSettings { | |
primary: DbSettings, | |
secondary: DbSettings, | |
write_to_secondary: bool, | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair, I had originally wanted to do these as optional and resortable, but that's not happened.
} | ||
|
||
async fn router_table_exists(&self) -> DbResult<bool> { | ||
self.primary.router_table_exists().await |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not check secondary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was debating that.
I wanted the call to be reasonably fast and working through the logic around which sources to poll might be weird. Plus, since we don't pass the UAID, we're kinda arbitrarily picking which data store to read from anyway.
Picking just primary ensures that the prime database is correct, with the secondary being just that, and lesser priority. (Honestly, once we're on bigtable, we could drop this entirely, but there's probably good reasons to keep it in case we ever have to move to another data store).
Allow for dual mode, where data is written to the primary, but the
secondary is read in order to 'back fill'. For simplicity, Dual presumes
that the primary data store is Bigtable and the secondary is DynamoDb.
This presumes that the
db_settings
contains serialized settings forboth data stores.
Closes: SYNC-3452
Blocks on #364