diff --git a/README.md b/README.md index 4aea388..cf5360b 100644 --- a/README.md +++ b/README.md @@ -1,137 +1,74 @@ -# Bisq BSQ Explorer +# BSQ Block Explorer -## Overview -The BSQ explorer provides basic information about BSQ transactions. -It is based on static html pages which are rendered from json files generated by a Bisq application running on the -server. Beside http server and Bisq (we run a seednode instance) we require bitcoind as the Bisq node is running in -fullnode mode. +Follow these instructions to setup a [BSQ Block Explorer](https://explorer.bisq.network) using data from your [Bisq Seednode](https://github.com/bisq-network/bisq/tree/master/seednode). Keep in mind you need a dedicated seednode for the BSQ explorer, as the JSON data dump puts too much load on it to seed the network properly. +## Bisq Seednode -### Installation -Requires Bisq seednode with `--dumpBlockchainData=true` +First, [setup your Bisq Seednode](https://github.com/bisq-network/bisq/tree/master/seednode#bisq-seed-node) so you have Tor, Bitcoin, and bisq-seednode running and fully synced. Then, enable BSQ data output on your Bisq Seednode with the following command -#### Web Server +```bash +sudo sed -i -e 's!BISQ_DUMP_BLOCKCHAIN=false!BISQ_DUMP_BLOCKCHAIN=true!' /etc/default/bisq-seednode.env +sudo service bisq-seednode restart ``` -sudo apt-get update -sudo apt-get install nginx -``` - -Check that it's running -``` -systemctl status nginx -``` -Copy nginx.config to /etc/nginx/sites-available and symlink from /etc/nginx/sites-enabled. Remove sym link to -default site. Replace `hostname` with site name. +It will take 10+ minutes before the seednode starts saving BSQ transaction data. -Install certificate +### Firewall -``` -sudo apt-get install software-properties-common -sudo add-apt-repository universe -sudo add-apt-repository ppa:certbot/certbot -sudo apt-get update -sudo apt-get install python-certbot-nginx -sudo certbot --nginx +Open ports 80 and 443 on your firewall for HTTP and HTTPS +```bash +sudo ufw allow 80/tcp +sudo ufw allow 443/tcp ``` -Set automatic certificate renewal in crontab. Try to renew every month. Edit crontab... +### Let's Encrypt -``` -sudo crontab -e -``` -...and paste this line2 -``` -0 0 1 * * certbot renew -``` +Next, request an SSL certificate for your server from Let's Encrypt using certbot-nginx -#### Explorer -Some needed pyhton packages -``` -sudo pip install python-bitcoinrpc -sudo pip install simplejson -sudo pip install gitpython +```bash +sudo apt-get update -q +sudo apt-get install -q -y nginx-core python-certbot-nginx +sudo certbot --nginx --agree-tos --non-interactive -m ssl@example.com -d explorer.example.com ``` -Setup variables for other commands below to use: -``` -export EXPLORER_HOME= -export DATADIR= -``` +After obtaining your SSL certificate, you should be able to see the default page at https://explorer.example.com/ -examples: -``` -#we assume you cloned this repo to: /home/user/bisq-explorer -#we assume bisq-seednode was run with --appName=seed_2002 and testnet is used as network -export EXPLORER_HOME=/home/user/bisq-explorer -export DATADIR=/home/user/.local/share/seed_2002/btc_testnet/db -``` +### BSQ Indexer -HTML need to point to explorer/www -``` -cd /var/www -sudo ln -s ${EXPLORER_HOME}/www html +Clone this repo to bisq user's homedir +```bash +curl -s https://raw.githubusercontent.com/bisq-network/bisq-explorer/master/install_bsq_explorer_debian.sh | sudo bash ``` -Copy Bisq seednode db directory contents to `$EXPLORER_HOME` -``` -mkdir ${EXPLORER_HOME}/data -cp -r ${DATADIR}/* ${EXPLORER_HOME}/data -``` +### Nginx -For the update script `inotifywait` is needed, it's part of inotify-tools -``` -sudo apt install inotify-tools +Install the nginx.conf from this repository, substituting explorer.example.com for your server hostname +```bash +sudo wget -O /etc/nginx/nginx.conf https://raw.githubusercontent.com/bisq-network/bisq-explorer/master/nginx.conf +sudo sed -i -e 's!__HOSTNAME__!explorer.example.com!g' /etc/nginx/nginx.conf +sudo service nginx restart ``` -#### Bitcoin -Bitcoin blocknotify script needs `nc` -``` -sudo apt install netcat-openbsd -``` +You should now be able to access your BSQ explorer at https://explorer.example.com/ -Example `bitcoin.conf` -``` -# Uncomment to run on testnet -#testnet=1 -lang=en - -whitelist=127.0.0.1 -rpcallowip=127.0.0.1 -rpcport=18332 - -server=1 -txindex=1 -rpcuser=bisquser -rpcpassword=bisqpasswd -blocknotify=bash $HOME/.bitcoin/blocknotify %s -``` -Example `blocknotify` +### Tor onion (optional) +Add these lines to the bottom of /etc/tor/torrc ``` -#!/bin/bash -echo $1 | nc -w 1 127.0.0.1 5110 - +HiddenServiceDir /var/lib/tor/bsqexplorer/ +HiddenServicePort 80 127.0.0.1:80 +HiddenServiceVersion 2 ``` -### Run - -Copy update_monitor.sh to a location outside of the explorer directory where the -script will be run from. Then edit it to configure the DATADIR and EXPLORER_HOME derictories. -You can use these commands to edit the files: -``` -sed -Ei 's|DATADIR=.*|DATADIR='"${DATADIR}"'|' update_monitor.sh -sed -Ei 's|EXPLORER_HOME=.*|EXPLORER_HOME='"${EXPLORER_HOME}"'|' update_monitor.sh +Then restart Tor with the following command +```bash +sudo service tor restart ``` -To start the update monitor -``` -./update_monitor.sh -``` +After Tor restarts, it will generate your onion hostname, get it by doing: -To update the explorer state, using the data from bisq seednode it's possible -to run the update manually. -``` -cd ${EXPLORER_HOME} -./update_data.sh ${DATADIR} +```bash +sudo cat /var/lib/tor/bsqexplorer/hostname ``` + +Then you can also access your BSQ explorer over Tor at http://foo.onion/ diff --git a/bsq-explorer b/bsq-explorer new file mode 100755 index 0000000..d7d632a --- /dev/null +++ b/bsq-explorer @@ -0,0 +1,32 @@ +#!/usr/bin/env bash + +# auto-export all variables +set -a + +# load environment from bisq-seednode configuration +source /etc/default/bisq-seednode.env + +# variables used by this script and bsq-index script +BISQ_DATA_DIR="${BISQ_HOME}/${BISQ_APP_NAME}/${BISQ_BASE_CURRENCY}" +BISQ_JSON_DIR="${BISQ_DATA_DIR}/db/json" +BSQ_EXPLORER_ROOT="${BISQ_HOME}/bisq-explorer/www" +BSQ_EXPLORER_DATA="${BSQ_EXPLORER_ROOT}/data" + +while true +do + # sync + echo "Syncing BSQ transaction data JSON from ${BISQ_JSON_DIR} to ${BSQ_EXPLORER_DATA}/" + rsync -hrt --stats --exclude '*.tmp' "${BISQ_JSON_DIR}" "${BSQ_EXPLORER_DATA}/" + + # index + echo "Indexing BSQ transaction data JSON and updating explorer HTML in ${BSQ_EXPLORER_ROOT}" + /usr/local/bin/bsq-index + + # wait + echo "Waiting for BSQ transaction data update from Bisq seednode" + inotifywait -qq -r -e modify,move,create,delete "${BISQ_JSON_DIR}" + echo "Detected BSQ transaction data update from Bisq seednode" + + # allow time for seednode to complete JSON data dump + sleep 10 +done diff --git a/bsq-explorer.service b/bsq-explorer.service new file mode 100644 index 0000000..b53231d --- /dev/null +++ b/bsq-explorer.service @@ -0,0 +1,20 @@ +[Unit] +Description=BSQ Explorer +After=bisq-seednode.service + +[Service] +ExecStart=/usr/local/bin/bsq-explorer +ExecStop=/bin/kill -TERM ${MAINPID} +Restart=on-failure + +User=bisq +Group=bisq + +PrivateTmp=true +ProtectSystem=full +NoNewPrivileges=true +PrivateDevices=true +MemoryDenyWriteExecute=false + +[Install] +WantedBy=multi-user.target diff --git a/bsq-index b/bsq-index new file mode 100755 index 0000000..1ec3ef8 --- /dev/null +++ b/bsq-index @@ -0,0 +1,361 @@ +#!/usr/bin/python3 + +import subprocess +import inspect +import simplejson +import time +import git +import os +import logging +import sys +from pprint import pprint + +def run_command(command, input_str=None, ignore_stderr=False): + if ignore_stderr: + if input_str!=None: + p = subprocess.Popen(command, shell=True, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE) + return p.communicate(input_str) + else: + p = subprocess.Popen(command, shell=True, + stdout=subprocess.PIPE) + return p.communicate() + else: + if input_str!=None: + p = subprocess.Popen(command, shell=True, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + return p.communicate(input_str) + else: + p = subprocess.Popen(command, shell=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + return p.communicate() + +def error(msg): + last_block_msg='' + func_name='unknown' + try: + func_name=inspect.stack()[1][3] + except IndexError: + pass + # on parse: update last block + if func_name.startswith('parse'): + # store last parsed block + try: + last_block_msg='('+str(last_block)+')' + except IOError: + pass + print('[E] '+func_name+': '+str(msg)+last_block_msg) + exit(1) + +def info(msg): + func_name='unknown' + try: + func_name=inspect.stack()[1][3] + except IndexError: + pass + print('[I] '+func_name+': '+str(msg)) + +def debug(msg): + if d == True: + func_name='unknown' + try: + func_name=inspect.stack()[1][3] + except IndexError: + pass + print('[D] '+func_name+': '+str(msg)) + +def formatted_decimal(float_number): + s=str("{0:.8f}".format(float_number)) + if s.strip('0.') == '': # only zero and/or decimal point + return '0.0' + else: + trimmed=s.rstrip('0') # remove zeros on the right + if trimmed.endswith('.'): # make sure there is at least one zero on the right + return trimmed+'0' + else: + if trimmed.find('.')==-1: + return trimmed+'.0' + else: + return trimmed + +def format_time_from_struct(st, short=False): + if short: + return time.strftime('%Y%m%d',st) + else: + return time.strftime('%d %b %Y %H:%M:%S GMT',st) + +def format_time_from_epoch(epoch, short=False): + return format_time_from_struct(time.localtime(int(epoch)), short) + +def get_now(): + return format_time_from_struct(time.gmtime()) + +def get_today(): + return format_time_from_struct(time.gmtime(), True) + +def get_string_xor(s1,s2): + result = int(s1, 16) ^ int(s2, 16) + return '{:x}'.format(result) + +def load_dict_from_file(filename, all_list=False, skip_error=False): + tmp_dict={} + try: + f=open(filename,'r') + if all_list == False: + tmp_dict=simplejson.load(f)[0] + else: + tmp_dict=simplejson.load(f) + f.close() + except IOError: # no such file? + if skip_error: + info('dict load failed. missing '+filename) + else: + error('dict load failed. missing '+filename) + return tmp_dict + +# mkdir -p function +def mkdirp(directory): + if not os.path.isdir(directory): + os.makedirs(directory) + +# dump json to a file, and replace it atomically +def atomic_json_dump(tmp_dict, filename, add_brackets=True): + # check if filename already exists + # if exists, write to a tmp file first + # then move atomically + + # make sure path exists + path, only_filename = os.path.split(filename) + mkdirp(path) + + f=open(filename,'w') + if add_brackets: + f.write('[') + f.write(simplejson.dumps(tmp_dict, sort_keys=True, use_decimal=True)) + if add_brackets: + f.write(']') + f.write('\n') + f.close() + +def load_json_file(filename): + f=open(filename,'r') + json_data=f.read() + f.close() + data=simplejson.loads(str(json_data)) + return data + +def get_git_details(): + return ("","") + # repo=git.Repo(config.bisqHome); + # assert repo.bare == False + # head_commit=repo.head.commit + # timestamp=format_time_from_epoch(int(head_commit.authored_date), True) + # return(head_commit.hexsha,timestamp) + +def main(): + lines_per_page=100 + lines=[] + last_block=0 + chainstate_dict=load_json_file(os.getenv('BSQ_EXPLORER_ROOT') + '/data/json/all/blocks.json') + for block in chainstate_dict['blocks']: + last_block=block['height'] + for tx in block['txs']: + txid=tx['id'] + time=tx['time'] + txType=tx['txType'] + burntFee=tx['burntFee'] + outputsNum=0 + txBsqAmount=0 + # take address from first output as tx details + if "u'address'" in list(tx['outputs'][0].keys()): + address=tx['outputs'][0]['address'] + else: + address="n/a" + + if txType == 'UNVERIFIED': + txTypeDisplayString='Unverified' + elif txType == 'INVALID': + txTypeDisplayString='Invalid' + elif txType == 'GENESIS': + txTypeDisplayString='Genesis' + elif txType == 'TRANSFER_BSQ': + txTypeDisplayString='Transfer BSQ' + elif txType == 'PAY_TRADE_FEE': + txTypeDisplayString='Pay trade fee' + elif txType == 'PROPOSAL': + txTypeDisplayString='Proposal' + elif txType == 'COMPENSATION_REQUEST': + txTypeDisplayString='Compensation request' + elif txType == 'REIMBURSEMENT_REQUEST': + txTypeDisplayString='Reimbursement request' + elif txType == 'BLIND_VOTE': + txTypeDisplayString='Blind vote' + elif txType == 'VOTE_REVEAL': + txTypeDisplayString='Vote reveal' + elif txType == 'LOCKUP': + txTypeDisplayString='Lockup' + elif txType == 'UNLOCK': + txTypeDisplayString='Unlock' + elif txType == 'ASSET_LISTING_FEE': + txTypeDisplayString='Asset listing fee' + elif txType == 'PROOF_OF_BURN': + txTypeDisplayString='Proof of burn' + else: + txTypeDisplayString='Undefined' + + for o in tx['outputs']: + index=o['index'] + if ('opReturn' not in o): + bsqAmount = o['bsqAmount'] + txBsqAmount += bsqAmount + addr=o['address'] + unspent=o['isUnspent'] + outputsNum+=1 + txo_entry={'bsqAmount':bsqAmount, 'time':time, 'txType':txType, 'txTypeDisplayString':txTypeDisplayString, 'txId':txid, 'index':str(index)} + if addr in addr_dict: + if unspent==True: + stats_dict['Unspent TXOs']+=1 + if 'utxos' in addr_dict[addr]: + addr_dict[addr]['utxos'].append(txo_entry) + else: + addr_dict[addr]['utxos']=[txo_entry] + else: + stats_dict['Spent TXOs']+=1 + if 'stxos' in addr_dict[addr]: + addr_dict[addr]['stxos'].append(txo_entry) + else: + addr_dict[addr]['stxos']=[txo_entry] + + else: + if unspent==True: + stats_dict['Unspent TXOs']+=1 + addr_dict[addr]={'utxos':[txo_entry]} + else: + stats_dict['Spent TXOs']+=1 + addr_dict[addr]={'stxos':[txo_entry]} + + if (o['txOutputType'] == 'GENESIS_OUTPUT' or + o['txOutputType'] == 'ISSUANCE_CANDIDATE_OUTPUT') and \ + o['isVerified'] == True: + # collect minted coins for stats + stats_dict['Minted amount']+=o['bsqAmount'] + + # collect the fee for stats + stats_dict['Burnt amount']+=float(tx['burntFee']) + + line_dict={'bsqAmount':txBsqAmount, 'txType':txType, 'txTypeDisplayString':txTypeDisplayString, 'txId':txid, 'time':time, 'burntFee':burntFee, 'outputsNum':outputsNum, 'height':last_block} + lines.append(line_dict) + + # divide by 100 Satoshi/BSQ (1 BSQ = 100 Sat) + stats_dict['Minted amount']/=100 + stats_dict['Burnt amount']/=100 + stats_dict['Existing amount']/=100 + + # calculate more stats + # TODO missing issued amount + stats_dict['Existing amount']=stats_dict['Minted amount']-stats_dict['Burnt amount'] + + stats_dict['Addresses']=len(list(addr_dict.keys())) + + # TODO not provided yet in jsons (though in app available) + # stats_dict['Price']=0.001234 + # stats_dict['Marketcap']=stats_dict['Price']*stats_dict['Existing amount'] + + + stats_json=[] + for k in ["Existing amount", "Minted amount", "Burnt amount", "Addresses", "Unspent TXOs", "Spent TXOs", "Price", "Marketcap"]: + stats_json.append({"name":k, "value":stats_dict[k]}) + + atomic_json_dump(stats_json, os.getenv('BSQ_EXPLORER_ROOT') + '/general/stats.json', add_brackets=False) + + + # split recent tx to pages + lines.reverse() + pages=int((len(lines)-1)/lines_per_page)+1 + for i in range(pages): + strnum=str(i+1).zfill(4) + atomic_json_dump(lines[i*lines_per_page:(i+1)*lines_per_page], os.getenv('BSQ_EXPLORER_ROOT') + '/general/BSQ_'+strnum+'.json', add_brackets=False) + + atomic_json_dump({"currency": "BSQ", "name": "BSQ token", "pages": pages}, os.getenv('BSQ_EXPLORER_ROOT') + '/values.json') + + + # update field in addr + for addr in list(addr_dict.keys()): + balance=0 + totalReserved=0 + burntNum=0 + genesisTxNum=0 + receivedOutputsNum=0 + spentOutputsNum=0 + totalGenesis=0 + totalReceived=0 + totalSpent=0 + if 'utxos' in addr_dict[addr]: + for u in addr_dict[addr]['utxos']: + balance+=u['bsqAmount'] + if u['txType']=='GENESIS': + genesisTxNum+=1 + totalGenesis+=u['bsqAmount'] + else: + receivedOutputsNum+=1 + totalReceived+=u['bsqAmount'] + if 'stxos' in addr_dict[addr]: + for s in addr_dict[addr]['stxos']: + if s['txType']=='GENESIS': + genesisTxNum+=1 + totalGenesis+=s['bsqAmount'] + spentOutputsNum+=1 + totalSpent+=s['bsqAmount'] + else: + receivedOutputsNum+=1 + totalReceived+=s['bsqAmount'] + spentOutputsNum+=1 + totalSpent+=s['bsqAmount'] + + totalBurnt=totalGenesis+totalReceived-totalSpent-balance + + addr_dict[addr]['address']=addr + addr_dict[addr]['balance']=balance + addr_dict[addr]['genesisTxNum']=genesisTxNum + addr_dict[addr]['totalGenesis']=totalGenesis + addr_dict[addr]['receivedOutputsNum']=receivedOutputsNum + addr_dict[addr]['totalReceived']=totalReceived + addr_dict[addr]['spentOutputsNum']=spentOutputsNum + addr_dict[addr]['totalSpent']=totalSpent + addr_dict[addr]['burntNum']=burntNum + addr_dict[addr]['totalBurnt']=totalBurnt + addr_dict[addr]['totalReserved']=totalReserved + + + for addr in list(addr_dict.keys()): + atomic_json_dump(addr_dict[addr], os.getenv('BSQ_EXPLORER_ROOT') + '/addr/'+addr+'.json', add_brackets=False) + + (commit_hexsha,commit_time)=get_git_details() + now=get_now() + revision_dict={"commit_hexsha":commit_hexsha, "commit_time":commit_time, "last_block":last_block, "last_parsed":now, "url":"https://github.com/bisq-network/bisq"} + + atomic_json_dump(revision_dict, os.getenv('BSQ_EXPLORER_ROOT') + '/revision.json', add_brackets=False) + +# global variables +d=False # debug mode +last_block=0 +bsqutxo_dict={} +bsqo_dict={} +tx_dict={} +addr_dict={} +chainstate_dict={} +stats_dict={"Existing amount":0, + "Minted amount":0, + "Burnt amount":0, + "Addresses":0, + "Unspent TXOs":0, + "Spent TXOs":0, + "Price":0, + "Marketcap":0} + +if __name__== "__main__" : + main() diff --git a/bsq_globals.py b/bsq_globals.py deleted file mode 100644 index d4278e7..0000000 --- a/bsq_globals.py +++ /dev/null @@ -1,40 +0,0 @@ -########################################### -# # -# Copyright Bisq 2018 # -# # -########################################### - -# globals.py - -# rpc_user and rpc_password -from config import * -from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException - - -def init(): - global rpc_connection - global last_block - global d # debug mode - global bsqutxo_dict - global bsqo_dict - global tx_dict - global addr_dict - global stats_dict - global chainstate_dict - last_block=0 - rpc_connection=AuthServiceProxy("http://%s:%s@%s:%s"%(rpc_user, rpc_password,rpc_host,rpc_port)) - bsqutxo_dict={} - bsqo_dict={} - tx_dict={} - addr_dict={} - stats_dict={"Existing amount":0, - "Minted amount":0, - "Burnt amount":0, - "Addresses":0, - "Unspent TXOs":0, - "Spent TXOs":0, - "Price":0, - "Marketcap":0} - chainstate_dict={} - d=False - diff --git a/bsq_json.py b/bsq_json.py deleted file mode 100644 index e1c8f1a..0000000 --- a/bsq_json.py +++ /dev/null @@ -1,201 +0,0 @@ -#!/usr/bin/python - -########################################### -# # -# Copyright Bisq 2018 # -# # -########################################### - -import logging -from bsq_utils_general import * -import sys -from pprint import pprint - -# debug and last_block: -import bsq_globals - -bsq_globals.init() -bsq_globals.chainstate_dict=load_json_file('www/data/json/all/blocks.json') - -lines_per_page=100 - -lines=[] -last_block=0 -for block in bsq_globals.chainstate_dict[u'blocks']: - last_block=block[u'height'] - for tx in block[u'txs']: - txid=tx[u'id'] - time=tx[u'time'] - txType=tx[u'txType'] - burntFee=tx[u'burntFee'] - outputsNum=0 - txBsqAmount=0 - # take address from first output as tx details - if "u'address'" in tx[u'outputs'][0].keys(): - address=tx[u'outputs'][0][u'address'] - else: - address="n/a" - - if txType == 'UNVERIFIED': - txTypeDisplayString='Unverified' - elif txType == 'INVALID': - txTypeDisplayString='Invalid' - elif txType == 'GENESIS': - txTypeDisplayString='Genesis' - elif txType == 'TRANSFER_BSQ': - txTypeDisplayString='Transfer BSQ' - elif txType == 'PAY_TRADE_FEE': - txTypeDisplayString='Pay trade fee' - elif txType == 'PROPOSAL': - txTypeDisplayString='Proposal' - elif txType == 'COMPENSATION_REQUEST': - txTypeDisplayString='Compensation request' - elif txType == 'REIMBURSEMENT_REQUEST': - txTypeDisplayString='Reimbursement request' - elif txType == 'BLIND_VOTE': - txTypeDisplayString='Blind vote' - elif txType == 'VOTE_REVEAL': - txTypeDisplayString='Vote reveal' - elif txType == 'LOCKUP': - txTypeDisplayString='Lockup' - elif txType == 'UNLOCK': - txTypeDisplayString='Unlock' - elif txType == 'ASSET_LISTING_FEE': - txTypeDisplayString='Asset listing fee' - elif txType == 'PROOF_OF_BURN': - txTypeDisplayString='Proof of burn' - else: - txTypeDisplayString='Undefined' - - for o in tx[u'outputs']: - index=o[u'index'] - if (not o.has_key(u'opReturn')): - bsqAmount = o[u'bsqAmount'] - txBsqAmount += bsqAmount - addr=o[u'address'] - unspent=o[u'isUnspent'] - outputsNum+=1 - txo_entry={u'bsqAmount':bsqAmount, u'time':time, u'txType':txType, u'txTypeDisplayString':txTypeDisplayString, u'txId':txid, u'index':str(index)} - if bsq_globals.addr_dict.has_key(addr): - if unspent==True: - bsq_globals.stats_dict['Unspent TXOs']+=1 - if bsq_globals.addr_dict[addr].has_key(u'utxos'): - bsq_globals.addr_dict[addr][u'utxos'].append(txo_entry) - else: - bsq_globals.addr_dict[addr][u'utxos']=[txo_entry] - else: - bsq_globals.stats_dict['Spent TXOs']+=1 - if bsq_globals.addr_dict[addr].has_key(u'stxos'): - bsq_globals.addr_dict[addr][u'stxos'].append(txo_entry) - else: - bsq_globals.addr_dict[addr][u'stxos']=[txo_entry] - - else: - if unspent==True: - bsq_globals.stats_dict['Unspent TXOs']+=1 - bsq_globals.addr_dict[addr]={u'utxos':[txo_entry]} - else: - bsq_globals.stats_dict['Spent TXOs']+=1 - bsq_globals.addr_dict[addr]={u'stxos':[txo_entry]} - - if (o[u'txOutputType'] == 'GENESIS_OUTPUT' or - o[u'txOutputType'] == 'ISSUANCE_CANDIDATE_OUTPUT') and \ - o[u'isVerified'] == True: - # collect minted coins for stats - bsq_globals.stats_dict['Minted amount']+=o[u'bsqAmount'] - - # collect the fee for stats - bsq_globals.stats_dict['Burnt amount']+=float(tx[u'burntFee']) - - line_dict={u'bsqAmount':txBsqAmount, u'txType':txType, u'txTypeDisplayString':txTypeDisplayString, u'txId':txid, u'time':time, u'burntFee':burntFee, u'outputsNum':outputsNum, u'height':last_block} - lines.append(line_dict) - -# divide by 100 Satoshi/BSQ (1 BSQ = 100 Sat) -bsq_globals.stats_dict['Minted amount']/=100 -bsq_globals.stats_dict['Burnt amount']/=100 -bsq_globals.stats_dict['Existing amount']/=100 - -# calculate more stats -# TODO missing issued amount -bsq_globals.stats_dict['Existing amount']=bsq_globals.stats_dict['Minted amount']-bsq_globals.stats_dict['Burnt amount'] - -bsq_globals.stats_dict['Addresses']=len(bsq_globals.addr_dict.keys()) - -# TODO not provided yet in jsons (though in app available) -# bsq_globals.stats_dict['Price']=0.001234 -# bsq_globals.stats_dict['Marketcap']=bsq_globals.stats_dict['Price']*bsq_globals.stats_dict['Existing amount'] - - -stats_json=[] -for k in ["Existing amount", "Minted amount", "Burnt amount", "Addresses", "Unspent TXOs", "Spent TXOs", "Price", "Marketcap"]: - stats_json.append({"name":k, "value":bsq_globals.stats_dict[k]}) - -atomic_json_dump(stats_json,'www/general/stats.json', add_brackets=False) - - -# split recent tx to pages -lines.reverse() -pages=int((len(lines)-1)/lines_per_page)+1 -for i in range(pages): - strnum=str(i+1).zfill(4) - atomic_json_dump(lines[i*lines_per_page:(i+1)*lines_per_page],'www/general/BSQ_'+strnum+'.json', add_brackets=False) - -atomic_json_dump({"currency": "BSQ", "name": "BSQ token", "pages": pages},'www/values.json') - - -# update field in addr -for addr in bsq_globals.addr_dict.keys(): - balance=0 - totalReserved=0 - burntNum=0 - genesisTxNum=0 - receivedOutputsNum=0 - spentOutputsNum=0 - totalGenesis=0 - totalReceived=0 - totalSpent=0 - if bsq_globals.addr_dict[addr].has_key(u'utxos'): - for u in bsq_globals.addr_dict[addr][u'utxos']: - balance+=u[u'bsqAmount'] - if u[u'txType']=='GENESIS': - genesisTxNum+=1 - totalGenesis+=u[u'bsqAmount'] - else: - receivedOutputsNum+=1 - totalReceived+=u[u'bsqAmount'] - if bsq_globals.addr_dict[addr].has_key(u'stxos'): - for s in bsq_globals.addr_dict[addr][u'stxos']: - if s[u'txType']=='GENESIS': - genesisTxNum+=1 - totalGenesis+=s[u'bsqAmount'] - spentOutputsNum+=1 - totalSpent+=s[u'bsqAmount'] - else: - receivedOutputsNum+=1 - totalReceived+=s[u'bsqAmount'] - spentOutputsNum+=1 - totalSpent+=s[u'bsqAmount'] - - totalBurnt=totalGenesis+totalReceived-totalSpent-balance - - bsq_globals.addr_dict[addr][u'address']=addr - bsq_globals.addr_dict[addr][u'balance']=balance - bsq_globals.addr_dict[addr][u'genesisTxNum']=genesisTxNum - bsq_globals.addr_dict[addr][u'totalGenesis']=totalGenesis - bsq_globals.addr_dict[addr][u'receivedOutputsNum']=receivedOutputsNum - bsq_globals.addr_dict[addr][u'totalReceived']=totalReceived - bsq_globals.addr_dict[addr][u'spentOutputsNum']=spentOutputsNum - bsq_globals.addr_dict[addr][u'totalSpent']=totalSpent - bsq_globals.addr_dict[addr][u'burntNum']=burntNum - bsq_globals.addr_dict[addr][u'totalBurnt']=totalBurnt - bsq_globals.addr_dict[addr][u'totalReserved']=totalReserved - - -for addr in bsq_globals.addr_dict.keys(): - atomic_json_dump(bsq_globals.addr_dict[addr],'www/addr/'+addr+'.json', add_brackets=False) - -(commit_hexsha,commit_time)=get_git_details() -now=get_now() -revision_dict={"commit_hexsha":commit_hexsha, "commit_time":commit_time, "last_block":last_block, "last_parsed":now, "url":"https://github.com/bisq-network/bisq"} - -atomic_json_dump(revision_dict,'www/revision.json', add_brackets=False) diff --git a/bsq_utils_general.py b/bsq_utils_general.py deleted file mode 100644 index d2692a7..0000000 --- a/bsq_utils_general.py +++ /dev/null @@ -1,161 +0,0 @@ -#!/usr/bin/python - -########################################### -# # -# Copyright Bisq 2018 # -# # -########################################### - -import subprocess -import inspect -import simplejson -import time -import git -import os -import bsq_globals - -def run_command(command, input_str=None, ignore_stderr=False): - if ignore_stderr: - if input_str!=None: - p = subprocess.Popen(command, shell=True, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE) - return p.communicate(input_str) - else: - p = subprocess.Popen(command, shell=True, - stdout=subprocess.PIPE) - return p.communicate() - else: - if input_str!=None: - p = subprocess.Popen(command, shell=True, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - return p.communicate(input_str) - else: - p = subprocess.Popen(command, shell=True, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - return p.communicate() - -def error(msg): - last_block_msg='' - func_name='unknown' - try: - func_name=inspect.stack()[1][3] - except IndexError: - pass - # on parse: update last block - if func_name.startswith('parse'): - # store last parsed block - try: - last_block_msg='('+str(bsq_globals.last_block)+')' - except IOError: - pass - print '[E] '+func_name+': '+str(msg)+last_block_msg - exit(1) - -def info(msg): - func_name='unknown' - try: - func_name=inspect.stack()[1][3] - except IndexError: - pass - print '[I] '+func_name+': '+str(msg) - -def debug(msg): - if bsq_globals.d == True: - func_name='unknown' - try: - func_name=inspect.stack()[1][3] - except IndexError: - pass - print '[D] '+func_name+': '+str(msg) - -def formatted_decimal(float_number): - s=str("{0:.8f}".format(float_number)) - if s.strip('0.') == '': # only zero and/or decimal point - return '0.0' - else: - trimmed=s.rstrip('0') # remove zeros on the right - if trimmed.endswith('.'): # make sure there is at least one zero on the right - return trimmed+'0' - else: - if trimmed.find('.')==-1: - return trimmed+'.0' - else: - return trimmed - -def format_time_from_struct(st, short=False): - if short: - return time.strftime('%Y%m%d',st) - else: - return time.strftime('%d %b %Y %H:%M:%S GMT',st) - -def format_time_from_epoch(epoch, short=False): - return format_time_from_struct(time.localtime(int(epoch)), short) - -def get_now(): - return format_time_from_struct(time.gmtime()) - -def get_today(): - return format_time_from_struct(time.gmtime(), True) - -def get_string_xor(s1,s2): - result = int(s1, 16) ^ int(s2, 16) - return '{:x}'.format(result) - -def load_dict_from_file(filename, all_list=False, skip_error=False): - tmp_dict={} - try: - f=open(filename,'r') - if all_list == False: - tmp_dict=simplejson.load(f)[0] - else: - tmp_dict=simplejson.load(f) - f.close() - except IOError: # no such file? - if skip_error: - info('dict load failed. missing '+filename) - else: - error('dict load failed. missing '+filename) - return tmp_dict - -# mkdir -p function -def mkdirp(directory): - if not os.path.isdir(directory): - os.makedirs(directory) - -# dump json to a file, and replace it atomically -def atomic_json_dump(tmp_dict, filename, add_brackets=True): - # check if filename already exists - # if exists, write to a tmp file first - # then move atomically - - # make sure path exists - path, only_filename = os.path.split(filename) - mkdirp(path) - - f=open(filename,'w') - if add_brackets: - f.write('[') - f.write(simplejson.dumps(tmp_dict, sort_keys=True, use_decimal=True)) - if add_brackets: - f.write(']') - f.write('\n') - f.close() - -def load_json_file(filename): - f=open(filename,'r') - json_data=f.read() - f.close() - data=simplejson.loads(unicode(json_data)) - return data - -def get_git_details(): - return ("","") - # repo=git.Repo(config.bisqHome); - # assert repo.bare == False - # head_commit=repo.head.commit - # timestamp=format_time_from_epoch(int(head_commit.authored_date), True) - # return(head_commit.hexsha,timestamp) diff --git a/config.py b/config.py deleted file mode 100644 index fdb0768..0000000 --- a/config.py +++ /dev/null @@ -1,4 +0,0 @@ -rpc_user='rpcuser' -rpc_password='enter_password_here' -rpc_host='127.0.0.1' -rpc_port='8332' diff --git a/install_bsq_explorer_debian.sh b/install_bsq_explorer_debian.sh new file mode 100755 index 0000000..0edbf13 --- /dev/null +++ b/install_bsq_explorer_debian.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +set -e + +echo "[*] Bisq Explorer installation script" + +##### change as necessary for your system + +SYSTEMD_SERVICE_HOME=/etc/systemd/system +SYSTEMD_ENV_HOME=/etc/default + +ROOT_USER=root +ROOT_GROUP=root +ROOT_HOME=~root + +BISQ_USER=bisq +BISQ_HOME=~bisq + +EXPLORER_REPO_URL=https://github.com/bisq-network/bisq-explorer +EXPLORER_REPO_NAME=bisq-explorer +EXPLORER_REPO_TAG=master + +EXPLORER_DEBIAN_PKG="python3-pip inotify-tools rsync" +EXPLORER_PYTHON_PKG="simplejson gitpython" +EXPLORER_BIN_PATH="/usr/local/bin" + +##### + +echo "[*] Cloning BSQ Explorer repo" +sudo -H -i -u "${BISQ_USER}" git config --global advice.detachedHead false +sudo -H -i -u "${BISQ_USER}" git clone --branch "${EXPLORER_REPO_TAG}" "${EXPLORER_REPO_URL}" "${BISQ_HOME}/${EXPLORER_REPO_NAME}" + +echo "[*] Installing BSQ Explorer debian packages" +sudo -H -i -u "${ROOT_USER}" DEBIAN_FRONTEND=noninteractive apt-get install -qq -y ${EXPLORER_DEBIAN_PKG} + +echo "[*] Installing BSQ Explorer python packages" +sudo python3 -m pip install ${EXPLORER_PYTHON_PKG} + +echo "[*] Installing BSQ Explorer scripts" +for script in bsq-index bsq-explorer;do + sudo -H -i -u "${ROOT_USER}" install -c -o "${ROOT_USER}" -g "${ROOT_GROUP}" -m 755 "${BISQ_HOME}/${EXPLORER_REPO_NAME}/${script}" "${EXPLORER_BIN_PATH}" +done + +echo "[*] Installing BSQ Explorer systemd service" +sudo -H -i -u "${ROOT_USER}" install -c -o "${ROOT_USER}" -g "${ROOT_GROUP}" -m 644 "${BISQ_HOME}/${EXPLORER_REPO_NAME}/bsq-explorer.service" "${SYSTEMD_SERVICE_HOME}" + +echo "[*] Reloading systemd daemon configuration" +sudo -H -i -u "${ROOT_USER}" systemctl daemon-reload + +echo "[*] Enabling BSQ Explorer service" +sudo -H -i -u "${ROOT_USER}" systemctl enable bsq-explorer.service + +echo "[*] Starting BSQ Explorer service" +sudo -H -i -u "${ROOT_USER}" systemctl start bsq-explorer.service +sudo -H -i -u "${ROOT_USER}" journalctl --no-pager --unit bsq-explorer + +echo '[*] Done!' +exit 0 diff --git a/nginx.conf b/nginx.conf index 750329b..5ba6236 100644 --- a/nginx.conf +++ b/nginx.conf @@ -32,27 +32,40 @@ http { server { listen 80; listen [::]:80; - server_name explorer.bisq.network; + server_name __HOSTNAME__; - if ($host = explorer.bisq.network) { + if ($host = __HOSTNAME__) { return 301 https://$host$request_uri; } # managed by Certbot return 404; # managed by Certbot } + server { + listen 127.0.0.1:81; + server_name __HOSTNAME__; + + index index.html; + root /bisq/bisq-explorer/www; + + location / { + expires 10s; + try_files $uri $uri/ /index.html =404; + } + } + server { listen [::]:443 ssl http2; # managed by Certbot listen 443 ssl http2; # managed by Certbot - ssl_certificate /etc/letsencrypt/live/explorer.bisq.network/fullchain.pem; # managed by Certbot - ssl_certificate_key /etc/letsencrypt/live/explorer.bisq.network/privkey.pem; # managed by Certbot + ssl_certificate /etc/letsencrypt/live/__HOSTNAME__/fullchain.pem; # managed by Certbot + ssl_certificate_key /etc/letsencrypt/live/__HOSTNAME__/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot - server_name explorer.bisq.network; # managed by Certbot + server_name __HOSTNAME__; # managed by Certbot index index.html; - root /home/bisqweb/bisq-explorer/www; + root /bisq/bisq-explorer/www; location / { expires 10s; diff --git a/update_data.sh b/update_data.sh deleted file mode 100755 index a280e39..0000000 --- a/update_data.sh +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash - -DATADIR=$1 -cd `dirname $0` - -function wait_for_change { - inotifywait -r \ - -e modify,move,create,delete \ - $DATADIR/json -} - -while wait_for_change; do - sleep 2 - rsync -rlt --delete $DATADIR/json www/data/ - /usr/bin/python ./bsq_json.py & -done - diff --git a/update_monitor.sh b/update_monitor.sh deleted file mode 100755 index 67a52cb..0000000 --- a/update_monitor.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -DATADIR=/home/explorer/.local/share/seedNode/btc_testnet/db -EXPLORER_HOME=/home/explorer/explorer - -while true -do -echo `date` "(Re)-starting update_data.sh" -$EXPLORER_HOME/update_data.sh $DATADIR > /dev/null 2> errors.log -echo `date` "update_data.sh terminated unexpectedly!!" -sleep 3 -done -