Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sled-agent] Initialize multi-node CockroachDB, rather than single-node #727

Closed
smklein opened this issue Mar 7, 2022 · 3 comments · Fixed by #3450
Closed

[sled-agent] Initialize multi-node CockroachDB, rather than single-node #727

smklein opened this issue Mar 7, 2022 · 3 comments · Fixed by #3450
Assignees
Labels
mvp Sled Agent Related to the Per-Sled Configuration and Management
Milestone

Comments

@smklein
Copy link
Collaborator

smklein commented Mar 7, 2022

Note: This issue tracks a follow-up from #686

CockroachDB is initialized using "start-single-node" instead of "start".

This works for testing, but is obviously not representative of an actual production-ready rack. We should ensure that CRDB instances can be initialized / booted jointly, across multiple nodes.

@smklein smklein added the Sled Agent Related to the Per-Sled Configuration and Management label Mar 7, 2022
@smklein
Copy link
Collaborator Author

smklein commented Mar 7, 2022

As an FYI for whoever picks this up - maybe future me - CRDB is picky about the clocks of all nodes being in sync. This will require clock synchronization between nodes, likely via NTP or some equivalent.

@davepacheco
Copy link
Collaborator

Yeah, I recommend we use chrony for this. When I was testing CockroachDB in late 2020, I found that the stock illumos ntpd was not even close to good enough to keep clocks in sync. CockroachDB expects something like 250ms of drift. After a few days or a few VM restarts in AWS, it was common for the clocks to drift further than this. After setting up crony (which was pretty straightforward), the clocks were generally within 1ms (as measured by CockroachDB).

There are open questions about what we'd point any NTP server at. RFD 34 remains open.

@smklein
Copy link
Collaborator Author

smklein commented Jan 27, 2023

Blocked on #2219

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mvp Sled Agent Related to the Per-Sled Configuration and Management
Projects
None yet
3 participants