-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decentralize the Bisq Markets API service #165
Comments
Yes, please. Glad to hear you're willing to take this on.
I'd recommend doing this separately from the rewrite to Java. Let's get a single instance up, stable, performant, well-defined, etc., then consider decentralizing / federating it.
Let's please discuss implementing the new API service using Spring. It has excellent security and caching support, along with everything else we'd need. The pricenode is already implemented in Spring, I'm an expert in the framework, and countless thousands of good Java developers know it well too. |
Per my comment above, I recommend renaming this proposal to "Port Markets API service to Java". |
Do you have any performance benchmarks on Spring, compared to other traffic engines we could use? We would likely want to use an actual HTTP reverse proxy, like Apache Trafficserver or nginx, in front of the API service anyway, so it's more about raw power over internal functionality IMO. And while I love Java as much as the next Bisq developer, I really don't feel Java is the best tool for an API service. Since the Bisq Markets API service currently needs to serve over 100 queries per second, and will likely grow over time, can a JVM backend really handle that? The last PHP implementation certainly could not, even with caching enabled, which is why we needed to move to a rust backend in the first place. Since rust compiles to machine code it is able to meet the demand, together with 12 instances of Google Appengine Java runtime in front of it to smart-cache and serve all the requests serving as a CDN. I really don't think a single Java node is going to work for the amount of load we are seeing. But to clarify, I am proposing to only use the JSON file dump feature of Bisq java nodes, and use a native code backend for the API service. The new API daemon can simply import these static JSON files and serve the API. In order to stay decentralized, I'd like to simply add more nodes for scaling, load balanced by DNS, so we don't need add any CloudFlare type CDN service in front of it as a MITM / CPOF.
Well this proposal is to specifically to decentralize the Bisq Markets API service by running multiple nodes, which requires more roles and budget expense, and to load balance these nodes using DNS. The way we implement that technically is something we can discuss on Keybase and over pull requests, but not really the scope of this DAO proposal... |
https://github.com/cbeams/perfdemo Performance is not an issue.
I will downvote the proposal if it attempts to decentralize and/or prematurely optimize via load balancing etc. The simple service should be written first, made correct and measured. Then considerations can be made as to whether it needs further performance improvements like load balancing across multiple instances, whether it needs (and merits the resources and funding) to be decentralized right now, etc. |
Well yes, the current code is written to run on a centralized instance of Google Appengine so it wouldn't be possible to decentralize it or run multiple nodes before making the new simple API service layer... Like I said we can work out the technical implementation details separately, this proposal is for the general goal of "decentralize all the things" along with the overall spirit of Bisq, so I think we are on the same page? Not sure why that would be controversial, as it's better and cheaper to do self-hosted.
The purpose of decentralized self-hosted nodes isn't primarily for performance reasons (although that is an added benefit). The primary purpose of this proposal is to remove the centralized CPOF of having a single instance, and to move away from Google, thereby increasing the availability, privacy, and censorship resistance of the API service. There's no reason why we shouldn't do this, as it would save money over Google which is relatively expensive compared to cheap VMs we can rent anywhere. |
@cbeams how about I start by simply moving back to the old Bisq + PHP codebase, load balanced across a few servers on my network ? I'd be happy to host this for free and save some money for Bisq, currently it's running $300 a month from Google |
Even better, please do that. This means no code changes, correct?
… On Jan 15, 2020, at 12:37 AM, wiz ***@***.***> wrote:
@cbeams how about I start by simply moving back to the old Bisq + PHP codebase, load balanced across a few servers on my network ? I'd be happy to host this for free and save some money for Bisq, currently it's running $300 a month from Google
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Well the code needs some work for sure, but since Bisq doesn't have the budget for this project I'll do the work for free. I'll get the old Java+PHP markets API service running on some of the older servers on my network and load-balance across them, and I won't request any compensation for code changes, maintaining, or hosting the Bisq Markets API for at least the next 10 dao cycles. If that's cool with you I think we can simply close this proposal for now, and we can figure out any technical details over Keybase chat... |
Summary
Following the success of the previous project to decentralize the BSQ explorer into several BSQ explorer nodes, I now propose to decentralize the Bisq Markets API in a similar fashion.
Background
A few months ago when I took over the Bisq Markets API maintainer role, the API service was in bad shape, mostly overloaded having grown to serve millions of requests, and suffering from slow load times and frequent downtime
As a quick temporary fix back then, @bodymindarts and I used his Risq incubator project and some Google Appengine caching code to re-implement the Bisq Markets API, and it has worked well for the past few months. But the move to Google was only temporary, as they do not respect user privacy, and keep IP logs of everyone who accesses the API service.
Additionally, I understand that @bodymindarts will no longer be able to maintain the Risq project due to his personal situation, as well as Bisq implementing new budget controls so there won't be any funding for incubator projects like Risq. We should move back to self-hosted as soon as reasonably possible to preserve the decentralized and self-hosted spirit of Bisq.
Proposal
Similar to how I implemented a one-command installation script for the Bisq Seednode and BSQ explorer nodes, I will implement a simple and easy way to install a local Bisq Markets API service. We will return to 100% java codebase Bisq nodes, using the JSON file dump. I came up with some RAM backed performance enhancements for the BSQ explorer use case which I feel can be applied to the Bisq Markets node as well.
Instead of using the previous PHP codebase to serve the API itself, we can research how to use some existing various open source projects as a minimal layer middleware to serve the static JSON filed dumped from Bisq Markets node to the API service - this shouldn't be a large amount of work, and there are a lot of projects that do this with high performance now.
After the new codebase is working well, we can have 4 new Bisq Markets API nodes run by one each of the current infrastructure operators, @wiz @Emzy @devinbileck @mrosseel, and we can use DNS round-robin to load balance across the 4 markets API nodes. This will reduce the load on each API node, eliminate the CPOF of having a centralized server, and hopefully ensure high uptime for the service going forward.
The text was updated successfully, but these errors were encountered: