Horizonclient fix merge issues (#1241)

* horizonclient/txnbuild README fixes (#1210) * fix client links in top-level readme * update clients README * mark old client deprecated in godoc short summary * fix code of conduct, standardise example * fix code of conduct/contributing links * txnbuild: enables multiple signatures (#1198) This PR enables multiple signatures on a transaction in the new Go SDK. It also lets every `Operation` type have a different source account than its `Transaction`. These changes are intertwined. Without multiple signatures, every operation in a transaction must share the transaction's source account. Differing source accounts are the most common use case for multiple signatures, and they also test it with greatest completeness. * root repo changelog links to sub-projects (#1214) * keystore: add update-keys endpoints to spec (#1215) We need an endpoint to update the encrypted seed when users forget their passwords. * move keystore to exp to fix build (#1223) * Add minimal files to fix build (#1225) * exp/ticker: Orderbook data support (#1193) * exp/ticker: update partial market query to also output base/counter asset types * exp/ticker: add functions to scrape orderbook data * exp/ticker: update orderbookstats param names to match market standards * exp/ticker: create orderbookstats database model and migrations * exp/ticker: consolidate upsert logic and implement orderbookstats upsert * exp/ticker: create CLI command to ingest orderbooks * exp/ticker: update horizon SDK path * exp/ticker: add orderbook stats to market.json generator * exp/ticker: add separate query to retrieve relevant markets * exp/ticker: add orderbook stats to graphql interface * exp/ticker: add Docker support for orderbook ingestion * exp/ticker: fix partial aggregated market query + add tests * exp/ticker: fix globall aggregated market query + add tests * exp/ticker: update ticker binary link and fix crontab comment * exp/ticker: fix issue that would enable inf values to be stored in db * exp/ticker: ensure only store valid orderbook entries are stored * exp/ticker: format gql/static/bindata.go to prevent CI errors * exp/ticker: add updated bindata with pg 9.5-compatible migrations * exp/ticker: ensure markets with 0 orderbook entries can be handled by tickerdb * exp/ticker: remove unnecessary commas from graphql schema * exp/ticker: consolidate bid and ask positions on field names * exp/ticker: simplify logic for creating orderbook requests * exp/ticker: fix how bid and ask volumes are calculated * exp/ticker: allow negative spread values * exp/ticker: fix close_time fallback on markets query * exp/ticker: add page size limit to orderbook requests * Make PR template a bit more helpful. (#1238) * Make PR template a bit more helpful. Clean up some typos, and point to an example docs folder in the go repo rather than just the external docs hosted on our site. * Fix typo. * Agh, formatting. * horizonclient: update Fund method (#1213) * update client.Fund * update change log * fix space * horizonclient: add more documentation (#1226) * Remove old stream method * more comments * fix format * split comment * horizonclient: Set default HTTP client (#1228) * set default HTTP client * changes from review * changelog for txnbuild 1.1 (#1231) * horizonclient/txnbuild README fixes (#1210) * fix client links in top-level readme * update clients README * mark old client deprecated in godoc short summary * fix code of conduct, standardise example * fix code of conduct/contributing links * txnbuild: enables multiple signatures (#1198) This PR enables multiple signatures on a transaction in the new Go SDK. It also lets every `Operation` type have a different source account than its `Transaction`. These changes are intertwined. Without multiple signatures, every operation in a transaction must share the transaction's source account. Differing source accounts are the most common use case for multiple signatures, and they also test it with greatest completeness. * root repo changelog links to sub-projects (#1214) * keystore: add update-keys endpoints to spec (#1215) We need an endpoint to update the encrypted seed when users forget their passwords. * move keystore to exp to fix build (#1223) * Add minimal files to fix build (#1225) * changelog for txnbuild 1.1 * update change log (#1233) * horizonclient: add version (#1229) * add package version * fix go fmt * clients/horizonclient: support dynamic effects (#1217) * update effects in protocols/horizon * update client interface * update tests * update changelogs * fix typo
stellar · May 3, 2019 · 6543779 · 6543779
1 parent 0f63acf
commit 6543779
Show file tree

Hide file tree

Showing 26 changed files with 997 additions and 89 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,22 +1,28 @@
-If you're making a doc PR or something tiny where the below is irrelevant, just delete this template and use a short description.
+If you're making a doc PR or something tiny where the below is irrelevant, just delete this
+template and use a short description.
 
 ### PR Structure
 
-* [ ] This PR has reasonably narrow scope (if not, break it down into smaller PRs)
-* [ ] This PR avoids mixing refactoring changes with feature changes (split into two PRs otherwise)
-* [ ] This PR's title starts with name of package that is most changed in the PR, ex. `services/friendbot`
-
+* [ ] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
+* [ ] This PR avoids mixing refactoring changes with feature changes (split into two PRs
+  otherwise).
+* [ ] This PR's title starts with name of package that is most changed in the PR, ex.
+  `services/friendbot`
 
 ### Thoroughness
 
 * [ ] This PR adds tests for the most critical parts of the new functionality or fixes.
-* [ ] I've updated any docs ([developer docs](https://www.stellar.org/developers/reference/), `.md` files, etc...) affected by this change
+* [ ] I've updated any docs ([developer docs](https://www.stellar.org/developers/reference/), `.md`
+  files, etc... affected by this change). Take a look in the `docs` folder for a given service,
+  like [this one](https://github.com/stellar/go/tree/master/services/horizon/internal/docs).
 
 ### Release planning
 
-* [ ] I've updated the relevant CHANGELOG ([here](services/horizon/CHANGELOG.md) for Horizon) if needed with deprecations, added features, breaking changes, and DB schema changes
-* [ ] I've decided if this PR requires a new major/minor version according to [semver](https://semver.org/), or if it's monly a patch change. The PR is targeted at the next release branch if it's not a patch change.
-
+* [ ] I've updated the relevant CHANGELOG ([here](services/horizon/CHANGELOG.md) for Horizon) if
+  needed with deprecations, added features, breaking changes, and DB schema changes.
+* [ ] I've decided if this PR requires a new major/minor version according to
+  [semver](https://semver.org/), or if it's mainly a patch change. The PR is targeted at the next
+  release branch if it's not a patch change.
 
 ## Summary
 

diff --git a/exp/ticker/cmd/generate.go b/exp/ticker/cmd/generate.go
@@ -76,7 +76,7 @@ var cmdGenerateAssetData = &cobra.Command{
 		Logger.Infof("Starting asset data generation, outputting to: %s\n", AssetsOutFile)
 		err = ticker.GenerateAssetsFile(&session, Logger, AssetsOutFile)
 		if err != nil {
-			Logger.Fatal("could not generate market data:", err)
+			Logger.Fatal("could not generate asset data:", err)
 		}
 	},
 }
diff --git a/exp/ticker/cmd/ingest.go b/exp/ticker/cmd/ingest.go
@@ -16,6 +16,7 @@ func init() {
 	rootCmd.AddCommand(cmdIngest)
 	cmdIngest.AddCommand(cmdIngestAssets)
 	cmdIngest.AddCommand(cmdIngestTrades)
+	cmdIngest.AddCommand(cmdIngestOrderbooks)
 
 	cmdIngestTrades.Flags().BoolVar(
 		&ShouldStream,
@@ -96,3 +97,26 @@ var cmdIngestTrades = &cobra.Command{
 		}
 	},
 }
+
+var cmdIngestOrderbooks = &cobra.Command{
+	Use:   "orderbooks",
+	Short: "Refreshes the orderbook stats database with new data retrieved from Horizon.",
+	Run: func(cmd *cobra.Command, args []string) {
+		Logger.Info("Refreshing the asset database")
+		dbInfo, err := pq.ParseURL(DatabaseURL)
+		if err != nil {
+			Logger.Fatal("could not parse db-url:", err)
+		}
+
+		session, err := tickerdb.CreateSession("postgres", dbInfo)
+		if err != nil {
+			Logger.Fatal("could not connect to db:", err)
+		}
+		defer session.DB.Close()
+
+		err = ticker.RefreshOrderbookEntries(&session, Client, Logger)
+		if err != nil {
+			Logger.Fatal("could not refresh error database:", err)
+		}
+	},
+}
diff --git a/exp/ticker/docker/conf/crontab.txt b/exp/ticker/docker/conf/crontab.txt
@@ -4,6 +4,9 @@
 # Refresh the database of assets, hourly:
 @hourly /opt/stellar/bin/ticker ingest assets --db-url=postgres://127.0.0.1:5432/ticker > /home/stellar/last-ingest-assets.log 2>&1
 
+# Refresh the database of orderbooks, every 10 minutes:
+*/10 * * * * /opt/stellar/bin/ticker ingest orderbooks --db-url=postgres://127.0.0.1:5432/ticker > /home/stellar/last-ingest-orderbooks.log 2>&1
+
 # Backfill the database of trades (including possible new assets), every 6 hours:
 0 */6 * * * /opt/stellar/bin/ticker ingest trades --db-url=postgres://127.0.0.1:5432/ticker > /home/stellar/last-ingest-trades.log 2>&1
 

diff --git a/exp/ticker/docker/setup b/exp/ticker/docker/setup
@@ -10,7 +10,7 @@ mkdir -p /opt/stellar/www
 chown -R stellar:stellar /opt/stellar/www
 mkdir -p /opt/stellar/postgresql/data
 
-wget -O ticker.tar.gz https://github.com/accordeiro/ticker-releases/releases/download/v0.4-alpha/ticker.tar.gz
+wget -O ticker.tar.gz https://github.com/accordeiro/ticker-releases/releases/download/v0.5.3-alpha/ticker.tar.gz
 tar -xvzf ticker.tar.gz
 mv ticker /opt/stellar/bin/ticker
 chmod +x /opt/stellar/bin/ticker
diff --git a/exp/ticker/docker/start b/exp/ticker/docker/start
@@ -28,6 +28,7 @@ function main() {
 
 	populate_assets
 	populate_trades
+	populate_orderbooks
 	generate_assets_file
 	generate_markets_file
 
@@ -70,6 +71,19 @@ function populate_trades() {
 }
 
 
+function populate_orderbooks() {
+	if [ -f $PGHOME/.orderbooks-populated ]; then
+		echo "ticker: orderbooks already pre-populated"
+		return 0
+	fi
+	echo ""
+	echo "Populating initial orderbook database"
+	echo ""
+	sudo -u stellar $STELLAR_BIN/ticker ingest orderbooks --db-url=$PGURL
+	touch $PGHOME/.orderbooks-populated
+}
+
+
 function generate_assets_file() {
 	if [ -f $STELLAR_HOME/www/assets.json ]; then
 		echo "ticker: assets.json already pre-populated"

diff --git a/exp/ticker/internal/actions_market.go b/exp/ticker/internal/actions_market.go
@@ -59,6 +59,8 @@ func GenerateMarketSummary(s *tickerdb.TickerSession) (ms MarketSummary, err err
 
 func dbMarketToMarketStats(m tickerdb.Market) MarketStats {
 	closeTime := utils.TimeToUnixEpoch(m.LastPriceCloseTime)
+
+	spread, spreadMidPoint := utils.CalcSpread(m.HighestBid, m.LowestAsk)
 	return MarketStats{
 		TradePairName:    m.TradePair,
 		BaseVolume24h:    m.BaseVolume24h,
@@ -77,6 +79,14 @@ func dbMarketToMarketStats(m tickerdb.Market) MarketStats {
 		Change7d:         m.PriceChange7d,
 		Price:            m.LastPrice,
 		Close:            m.LastPrice,
+		BidCount:         m.NumBids,
+		BidVolume:        m.BidVolume,
+		BidMax:           m.HighestBid,
+		AskCount:         m.NumAsks,
+		AskVolume:        m.AskVolume,
+		AskMin:           m.LowestAsk,
+		Spread:           spread,
+		SpreadMidPoint:   spreadMidPoint,
 		CloseTime:        closeTime,
 	}
 }
diff --git a/exp/ticker/internal/actions_orderbook.go b/exp/ticker/internal/actions_orderbook.go
@@ -0,0 +1,65 @@
+package ticker
+
+import (
+	"time"
+
+	horizonclient "github.com/stellar/go/clients/horizonclient"
+	"github.com/stellar/go/exp/ticker/internal/scraper"
+	"github.com/stellar/go/exp/ticker/internal/tickerdb"
+	"github.com/stellar/go/support/errors"
+	hlog "github.com/stellar/go/support/log"
+)
+
+// RefreshOrderbookEntries updates the orderbook entries for the relevant markets that were active
+// in the past 7-day interval
+func RefreshOrderbookEntries(s *tickerdb.TickerSession, c *horizonclient.Client, l *hlog.Entry) error {
+	sc := scraper.ScraperConfig{
+		Client: c,
+		Logger: l,
+	}
+
+	// Retrieve relevant markets for the past 7 days (168 hours):
+	mkts, err := s.Retrieve7DRelevantMarkets()
+	if err != nil {
+		return errors.Wrap(err, "could not retrieve partial markets")
+	}
+
+	for _, mkt := range mkts {
+		ob, err := sc.FetchOrderbookForAssets(
+			mkt.BaseAssetType,
+			mkt.BaseAssetCode,
+			mkt.BaseAssetIssuer,
+			mkt.CounterAssetType,
+			mkt.CounterAssetCode,
+			mkt.CounterAssetIssuer,
+		)
+		if err != nil {
+			l.Error(errors.Wrap(err, "could not fetch orderbook for assets"))
+			continue
+		}
+
+		dbOS := orderbookStatsToDBOrderbookStats(ob, mkt.BaseAssetID, mkt.CounterAssetID)
+		err = s.InsertOrUpdateOrderbookStats(&dbOS, []string{"base_asset_id", "counter_asset_id"})
+		if err != nil {
+			l.Error(errors.Wrap(err, "could not insert orderbook stats into db"))
+		}
+	}
+
+	return nil
+}
+
+func orderbookStatsToDBOrderbookStats(os scraper.OrderbookStats, bID, cID int32) tickerdb.OrderbookStats {
+	return tickerdb.OrderbookStats{
+		BaseAssetID:    bID,
+		CounterAssetID: cID,
+		NumBids:        os.NumBids,
+		BidVolume:      os.BidVolume,
+		HighestBid:     os.HighestBid,
+		NumAsks:        os.NumAsks,
+		AskVolume:      os.AskVolume,
+		LowestAsk:      os.LowestAsk,
+		Spread:         os.Spread,
+		SpreadMidPoint: os.SpreadMidPoint,
+		UpdatedAt:      time.Now(),
+	}
+}
diff --git a/exp/ticker/internal/gql/main.go b/exp/ticker/internal/gql/main.go
@@ -38,6 +38,7 @@ type asset struct {
 	Countries                   string
 	Status                      string
 	IssuerID                    int32
+	OrderbookStats              orderbookStats
 }
 
 // partialMarket represents the aggregated market data for a
@@ -58,6 +59,20 @@ type partialMarket struct {
 	Close                float64
 	IntervalStart        graphql.Time
 	FirstLedgerCloseTime graphql.Time
+	OrderbookStats       orderbookStats
+}
+
+// orderbookStats represents the orderbook stats for a
+// specific pair of assets (aggregated or not)
+type orderbookStats struct {
+	BidCount       BigInt
+	BidVolume      float64
+	BidMax         float64
+	AskCount       BigInt
+	AskVolume      float64
+	AskMin         float64
+	Spread         float64
+	SpreadMidPoint float64
 }
 
 type resolver struct {

diff --git a/exp/ticker/internal/gql/resolvers_market.go b/exp/ticker/internal/gql/resolvers_market.go
@@ -5,6 +5,7 @@ import (
 
 	"github.com/graph-gophers/graphql-go"
 	"github.com/stellar/go/exp/ticker/internal/tickerdb"
+	"github.com/stellar/go/exp/ticker/internal/utils"
 )
 
 // Markets resolves the markets() GraphQL query.
@@ -83,6 +84,18 @@ func validateNumHoursAgo(n *int32) (int, error) {
 
 // dbMarketToPartialMarket converts a tickerdb.PartialMarket to a *partialMarket
 func dbMarketToPartialMarket(dbMarket tickerdb.PartialMarket) *partialMarket {
+	spread, spreadMidPoint := utils.CalcSpread(dbMarket.HighestBid, dbMarket.LowestAsk)
+	os := orderbookStats{
+		BidCount:       BigInt(dbMarket.NumBids),
+		BidVolume:      dbMarket.BidVolume,
+		BidMax:         dbMarket.HighestBid,
+		AskCount:       BigInt(dbMarket.NumAsks),
+		AskVolume:      dbMarket.AskVolume,
+		AskMin:         dbMarket.LowestAsk,
+		Spread:         spread,
+		SpreadMidPoint: spreadMidPoint,
+	}
+
 	return &partialMarket{
 		TradePair:            dbMarket.TradePairName,
 		BaseAssetCode:        dbMarket.BaseAssetCode,
@@ -99,5 +112,6 @@ func dbMarketToPartialMarket(dbMarket tickerdb.PartialMarket) *partialMarket {
 		Close:                dbMarket.Close,
 		IntervalStart:        graphql.Time{Time: dbMarket.IntervalStart},
 		FirstLedgerCloseTime: graphql.Time{Time: dbMarket.FirstLedgerCloseTime},
+		OrderbookStats:       os,
 	}
 }
diff --git a/exp/ticker/internal/gql/static/bindata.go b/exp/ticker/internal/gql/static/bindata.go
diff --git a/exp/ticker/internal/gql/static/schema.gql b/exp/ticker/internal/gql/static/schema.gql
@@ -75,6 +75,7 @@ type Market {
 	close: Float!
 	intervalStart: Time!
 	firstLedgerCloseTime: Time!
+	orderbookStats: OrderbookStats!
 }
 
 type AggregatedMarket {
@@ -89,6 +90,18 @@ type AggregatedMarket {
 	close: Float!
 	intervalStart: Time!
 	firstLedgerCloseTime: Time!
+	orderbookStats: OrderbookStats!
+}
+
+type OrderbookStats {
+ 	bidCount: BigInt!
+	bidVolume: Float!
+	bidMax: Float!
+	askCount: BigInt!
+	askVolume: Float!
+	askMin: Float!
+	spread: Float!
+	spreadMidPoint: Float!
 }
 
 type Issuer {