Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce cluster lock so that solo doesn't allow parallel execution of commands #43

Closed
Tracked by #38 ...
jeromy-cannon opened this issue Feb 13, 2024 · 3 comments · Fixed by #707
Closed
Tracked by #38 ...
Assignees
Labels
New Feature A new feature, service, or documentation. Major changes that are not backwards compatible. P1 High priority issue. Required to be completed in the assigned milestone.

Comments

@jeromy-cannon
Copy link
Contributor

jeromy-cannon commented Feb 13, 2024

depends on: #569

@nathanklick nathanklick transferred this issue from hashgraph/full-stack-testing Feb 20, 2024
@nathanklick nathanklick changed the title CLI[Solo]: refactor Node.checkNetworkNodeStarted() to avoid sleep or only toggle under correct conditions Refactor Node.checkNetworkNodeStarted() to avoid sleep or only toggle under correct conditions Feb 20, 2024
@nathanklick nathanklick removed this from FST Suite Feb 20, 2024
@nathanklick nathanklick added this to Solo Feb 20, 2024
@github-project-automation github-project-automation bot moved this to 🆕 New in Solo Feb 20, 2024
@leninmehedy leninmehedy added the Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. label Feb 21, 2024
@leninmehedy leninmehedy moved this from 🆕 New to 🔖 Ready in Solo Feb 21, 2024
@leninmehedy leninmehedy moved this from 🔖 Ready to 🏗 In progress in Solo Feb 21, 2024
@leninmehedy leninmehedy changed the title Refactor Node.checkNetworkNodeStarted() to avoid sleep or only toggle under correct conditions Introduce process lock so that solo doesn't allow parallel execution of commands Feb 22, 2024
@leninmehedy leninmehedy moved this from 🏗 In progress to 👀 In review in Solo Feb 22, 2024
@leninmehedy
Copy link
Member

leninmehedy commented Feb 28, 2024

We should use cluster level lock instead of process lock. Kubernetes provide lease lock, we can use that.

@leninmehedy leninmehedy changed the title Introduce process lock so that solo doesn't allow parallel execution of commands Introduce cluster lock so that solo doesn't allow parallel execution of commands Feb 28, 2024
@leninmehedy leninmehedy moved this from 👀 In review to 🔖 Ready in Solo Feb 29, 2024
@leninmehedy leninmehedy added New Feature A new feature, service, or documentation. Major changes that are not backwards compatible. P1 High priority issue. Required to be completed in the assigned milestone. and removed Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. labels Feb 29, 2024
@leninmehedy leninmehedy moved this from 🔖 Ready to 📋 Backlog in Solo Mar 5, 2024
@leninmehedy leninmehedy moved this from 📋 Backlog to 🔖 Ready in Solo Mar 13, 2024
@leninmehedy leninmehedy moved this from 🔖 Ready to 📋 Backlog in Solo Apr 22, 2024
@leninmehedy
Copy link
Member

Some notes on locks + solo state:

  • Solo state should be versioned and supports migrations
  • Locks should be namespace level leases
  • Two users should be able to operate against different network deployments aka namespaces simultaneously
  • Use the native K8S lease construct
  • Look at how the Java or Go Operator SDK implements expirable mutex locks for cluster leadership decisions

@leninmehedy leninmehedy moved this from 📋 Backlog to 🔖 Ready in Solo Apr 23, 2024
@jeromy-cannon jeromy-cannon self-assigned this Sep 10, 2024
@jeromy-cannon jeromy-cannon removed their assignment Sep 25, 2024
@jeromy-cannon
Copy link
Contributor Author

pulled this from the design doc on multi-cluster, not sure how useful it will be:

# TODO: locking shouldn't be in the config, it should be part of the lease
	lock: # as solo acquires all locks successfully, it should update the solo-state with the metadata so that we can contact the person and override locks if required
		status: true
		expiresAt: 2024-04-25T13:30:32Z
		createdAt: 2024-04-25T13:23:32Z
		contact: [email protected] # person who has the lock
		fromHost: Lenin-RaspberryPi # some sort of host name
		fromIp: 149.167.236.138

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Feature A new feature, service, or documentation. Major changes that are not backwards compatible. P1 High priority issue. Required to be completed in the assigned milestone.
Projects
Status: ✅ Done
3 participants