Improve monitoring tool

[ManfredKarrer]

I set up a "quick and dirty" monitoring tool in December for monitoring the seed nodes and Bitcoin nodes using the Bisq infrastructure (not just Tor connection checks as that reported too many false positives).
http://seedmonitor.0-2-1.net/
https://github.com/bisq-network/exchange/tree/master/monitor

It is working more or less ok and reports no false positives anymore as far I observed, but still would require some fixes, improvements and probably a heavy refactoring (was really done quick and dirty in 1 or 2 days). It does not touch too much the overall Bisq code and the subject for refactoring is mainly the way how the monitored data are structured.
It could be extended much more but first task would be to get it more reliable and get Slack notifications enabled again (integrated but deactivated as it reports too many messages).

It would also be good to separate Seed node and Bitcoin node monitoring because with the recent Tor relay node Dos protections a host with a lot of connections might get blocked.

Further extensions would be to monitor host parameters (CPU/RAM load, network load,...) of all monitored nodes. @Emzy has some nice tool running for his nodes.

To get price provider nodes monitored in the same way (using actual Bisq code not just Tor requests as it is now at http://188.166.2.139:8080/status) would be good as well.

[ujjwalwahi]

Hi @ManfredKarrer
I am interested in working on this issue

[ManfredKarrer]

Yeah would be great!

[ManfredKarrer]

If you work in regtest environment use those prog args:
–baseCurrencyNetwork=BTC_REGTEST --useDevPrivilegeKeys=true --useLocalhostForP2P=true --nodePort=3402 --appName=bisq-BTC_REGTEST_Monitor --useAllProvidedNodes=true --numConnectionForBtc=30

For mainnet:
–baseCurrencyNetwork=BTC_MAINNET --appName=bisq-BTC_MAINNET_Monitor --useAllProvidedNodes=true --numConnectionForBtc=30

If you test with localhost you might want to change the default port 80 to something like 8080 in MonitorMain: port(8080);

[ujjwalwahi]

@ManfredKarrer thanks for help. I can now run the monitor locally using mainnet.

I can now start working on the issue. First issue that I understand you want to address is related to refactoring of code. Is this related to logic and template mixing in MetricsModel.java?

I would like to understand what will make the system more reliable. Can you point me to first task that should be done?

[ManfredKarrer]

I think it should be rewritten and also the way what is displayed in the webpage.
There are requests all 10 minutes to the nodes. The page should show if the nodes have responded in that request cycle and how many retries have been needed until a response succeeded.
To show the data and bigger deviations would be good as well.
We should keep in mind further metrics like CPU/RAM/bandwidth which get requested separately from local server monitoring tools. @Emzy has such a tool running, maybe he can give some hints.

Most important is that we filter out false positives (thats why there are retries because if Tor is very slow or unstable the first request might fail) and show clearly when a node is failing.

So it is pretty open how to do it in details. The current version was done really very quick and dirty and should not be considered as "optimal".

[Emzy]

I'm using munin http://munin-monitoring.org/
It should be inn every linux distribution. You have a server that is saving the data and do the graphs. The clients (munin-node) that are collecting the data can run on many other hosts that should be monitored.

There are may plugins included. And this is my plugin to collect the bitcoin-core data:

/usr/share/munin/plugins/bitcoind_

#!/bin/bash
type=`cut -d "_" -f 2 <<< "$0"`
type_upper="$(tr '[:lower:]' '[:upper:]' <<< ${type:0:1})${type:1}"
if [ "$1" = "config" ]; then
  echo "graph_category bitcoin"
  echo "graph_title Bitcoin $type_upper"
  echo "graph_vlabel $type_upper"
  echo "$type.label $type"
else
  bcpath="/usr/local/bin/bitcoin-cli"
  if [ "$type" = "connections" ]; then
    value=`su - emzy -c "$bcpath getconnectioncount"`
  elif [ "$type" = "blocks" ]; then
    value=`su - emzy -c "$bcpath getblockcount"`
  elif [ "$type" = "difficulty" ]; then
    value=`su - emzy -c "$bcpath getdifficulty"`
  elif [ "$type" = "mempoolsize" ]; then
    value=`su - emzy -c "$bcpath getmempoolinfo" | grep size | grep -oP '\d+'`
  elif [ "$type" = "mempoolbytes" ]; then
    value=`su - emzy -c "$bcpath getmempoolinfo" | grep bytes | grep -oP '\d+'`
  fi
  echo "$type.value $value"
fi

Example

[ghost]

Hello,
Please could you tell what is the status of this issue/bounty ?
Is this issue still of interest (or not) ?
Is somebody working on it ?
(I guess the actual code behind http://seedmonitor.0-2-1.net/ is in java ?)
Thanks

[ManfredKarrer]

Yes it is still open and of interest! And yes its in Java, though can be combined with different tools (e.g. http://munin-monitoring.org).

[ghost]

I'm just looking at munin, and see that it exists also as an "official" debian package. Nice.
https://packages.debian.org/stretch/munin
(My worry is that I'm absolutely not a java dev, and since Bisq is greatly coded in java, I feel disqualified for much tasks, at least when it narrows the core).
Did you use mrosseel's webapi Manfred for the actual monitor's implementation ? or maybe it's not even possible/pertinent ?
(Apologies for the noob questions)

[mrosseel]

the webapi was not used for monitoring, as the webapi is mostly to read/create offers, trades, fund your wallet etc. The monitoring checks the health of all nodes in the P2P network (peers, seednodes, ... ). Most of the checks happen over the P2P network, some like the pricenodes are over http.

[ghost]

Thanks for the answer @mrosseel
At the moment, do all bisq monitored/special nodes have munin-node installed ? (is it or may it be a requirement for those nodes ?)
I guess most of the bisq monitored/special nodes run under linux, but maybe I'm wrong ?

I dive a bit into this munin tool.

[ManfredKarrer]

Thanks for your feedback. The main task is java and p2p network based. The munin tool would be an additional tool to check the state of the machine.

[ghost]

Hello @Emzy ,
I guess you have some munin master node working,
do you have mail alerts enabled ? does it work ?
I have also a munin master node running, the graphs are fine
... but impossible to generate the mail alerts.
I use postfix/sendmail which works properly and I tested a lot of syntaxes ... but nothing works :-(

[Emzy]

Hello @HarryMacfinned,

no I have not. But I now configured it and will test it.
Meybe the "mail" program is missing?

It seems you can test it with:

# mail -s "Munin notification" [email protected]
testmail
.
CC:
#

[ghost]

Hello @Emzy ,
I have installed postfix which only gives me the sendmail command. But normally this should not be an issue.
I may try with another MTA, but I'll first try to have it working with sendmail/postfix.
(The concerned VPS is a debian stretch 64 bits.)

[Emzy]

Hello @HarryMacfinned,
please try to install mailx and use the mail command.

apt-get install mailx

[ghost]

Hello @Emzy ,
Thanks for the hint. There are 2 mailx flavors available for debian stretch : bsd-mailx and heirloom-mailx
( https://packages.debian.org/search?keywords=mailx&searchon=names&suite=all&section=all ),
no pure mailx.
I'm reading the docs.
heirloom-mailx seems the choice, because BSD-mailx seems really raw ( https://packages.debian.org/stretch/bsd-mailx )
Did you try mailx ?

[freimair]

Started work on the issue here: https://github.com/freimair/bisq/tree/monitor