Testing BTC-RPC-Proxy to simulate an unpruned node #722
Comments
|
Hi, happy you're interested!
You don't need to install Rust by piping commands to I'm now looking into re-packaging it in my Debian repository, so you will have even easier time installing. :) Also your instructions describe cloning Regarding configuration, daemon cookie authentication is supported and recommended. I plan to implement it for users as well eventually. Also you seem to be missing
Would you be willing to try again and use Wireshark to dump the exact request? (right click on one of the request packets -> follow TCP stream) (feel free to censor private information)
It's still an alpha feature. :) But it's intended to be entirely transparent so that the application doesn't even know it's not a real bitcoind. Therefore if there's any problem, definitely ping us. |
|
Thank you will test and update the instructions. $ cargo --version did now: then git clone https://github.com/Start9Labs/btc-rpc-proxy.git
cd /home/brp/btc-rpc-proxy
git checkout feature/refactor
and I get this with: |
|
Sorry, misread |
ok so instead of
|
|
Ah, yeah, I reported it and it seems they forgot to add it. The suggestion from the compiler works, so for the time being you can manually add |
|
It works now: Can see a wall of RPC calls passing through the Proxy: no errors in the Specter server and no popup about the pruned node either! Updating instructions and will explore more. |
|
Next problem is that Specter still perceives as it is a pruned node: No errors in the Proxy. Updated the instructions to show how I got this far: https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6 |
|
Nice!!! I still want to test it a bit more before merging and signing as the official version. Perhaps doing a second pass of review as well. If you know of other projects that need non-pruned node, then you can help push this forward by trying out the proxy with them as well and reporting your findings. I also want to add a new configuration option for log level later so that production users are not spammed with those messages. The logs are quite useful for the time being. |
|
Weird, GitHub showed me your latest message only after I commented... Looks like specter attempted wallet rescan and that seems to be unimplemented in |
|
Thinking about it again, I think adding |
|
now tried to import my backed up wallets while connected to BTC-RPC-Proxy backed by a real unpruned bitcoind node. The scanning was triggered in bitcoind, but as it seems the wallets couldn't be saved while going through the Proxy. Specter loads and unloads wallets to bitcoind and that must be not playing well together with BTC-RPC-Proxy. |
|
Proxy should have no effects on this behavior. It looks like you forgot to restore the wallet files or something. |
|
You are right, it was a problem with restoring with a used bitcoind wallet. I've recreated the bitcoind wallet.dat and emptied the wallets directory from .specter and the errors are gone. I am inclined to close this issue since it is not (yet) possible to do rescan on a pruned node without redownloading the full blockchain or plugging an electrs instance to either Specter itself or to BTC-RPC-Proxy. Looking forward to the further developments on BTC-RPC-Proxy and it was well proven that it is capable of setting the fine-grained bitcoin RPC permissions as it was intended to. |
|
I'm not sure if this is even solvable problem at all. If you don't have any kind of index or wallet metadata, then downloading the whole chain (in theory does not have to be stored) is the only trustless way to recover a wallet. But if you're still interested in some of the other tradeoffs available, I'd love to hear your opinion which of them sounds most reasonable to you so that maybe I can look into it one day. Options that seem to be realistic:
|
|
I think
would be the most desirable option given there would be more nodes serving them. |
|
Yeah, that sounds pretty good to me too. We might also check if bitcoind can keep them despite pruning and set it to do so. But AFAIK only Knots can send filters over RPC. |
just a note, this is a privacy leak. Ideally it should be configurable as to whether you download the whole chain, or only blocks that you need. |
|
Not extremely big privacy leak, I believe. It should be hard to correlate transactions from whole blocks (which even have false-positives) and each block (candidate) could be downloaded over separate Tor connection. Having it configurable for paranoid users would be nice of course. |
|
Simulating |
|
I'm definitely not opposed to the idea of having the proxy a bit heavier if it is configurable (opt-in) and the code is properly organized. Especially if it means Bitcoin users no longer have to store 300G of chain data to access very useful Bitcoin features. I don't think I can find the time to work on it myself in the following month or two but I could review/merge a PR if you're motivated to do it sooner. |
|
FYI, @chrisguida is working on this. It requires giving proxy control over pruning though, to do it efficiently, so that it can ensure undo files are not deleted before they are used. |
|
That's great news! I'm not very happy with controlling pruning over RPC as it'll either break my setup or I'll have to do some serious redesign. Would be nice if fallback was possible. But I do realize the benefits so I do intend to think about redesigning. It'll just take some time to do it. |
|
@chrisguida yes I know, I wrote that part of the docs :) It should be compatible with RPC proxy faking non-pruning though as long as |
|
@Kixunil the alternative is to maintain a second UTXO set, separate from core in order to know the scripts associated with spends. It's much slower and takes a lot more storage. |
|
@dr-bonez ah, didn't realize those scripts are needed to build block filters. Would appreciate if it was possible to turn this feature off which would also stop control of pruning. At least for some time. What I'm more concerned about is that if a user has already installed pruned version and wants to start using btc-rpc-proxy later, it'd fail. So some kind of fallback even if slow and temporarily using more storage would be useful. I can understand if this is not priority, just something to keep in mind and perhaps design the code to not be hard to add. |
|
oh for sure. All of these kinds of features should be opt in. But they won't get block filters if they don't turn it on. the overhead of an additional fallback system probably isn't worth it right now. But they are adding support for block filters to pruned nodes in 0.22, so this is a temporary holdover anyway. |
|
Ah, my apologies, I hadn't noticed that 😅 |
|
Waiting for 0.22 is not too bad. @chrisguida no problem, no offense taken. Just had some laugh. :) |
From: #298 (comment)
This could help Specter to run a with pruned node and maybe would make the use of external block explorers unnecessary (see #718)
Quick installation steps as I did on Debian Linux: https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#file-btc-rpc-proxy-md
Need to allow all RPC commands used. Got these when searched for the self.rpc.commands in specter-desktop: https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#configure , but likely missing something still.
When running Specter (
python3 -m cryptoadvance.specter server) the detection of the bitcoinRPC on localhost (BTC-RPC-proxy) works and connection is tested successfully but immediately runs into these errors:https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#file-specter-desktop-log
Same time errors in BTC-RPC-Proxy are showing the bad requests:
https://gist.github.com/openoms/2a94a8ad756bf93c8cd57d68958450a6#file-btc_rpc_proxy-log
Either using BTC-RPC-Proxy is really close and I am just missing something or the next step could be to request @Kixunil for the feature if we figure it out what is exactly needed.
The text was updated successfully, but these errors were encountered: