Coinbase: Update exchange implementation #1480

cranktakular · 2024-02-14T06:07:40Z

PR Description

Updating Coinbase to the Advanced Trade and Sign In With Coinbase APIs, as well as a few tiny fixes of other parts of the codebase found along the way.

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
This change requires a documentation update

How has this been tested

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and
also consider improving test coverage whilst working on a certain feature or package.

go test ./... -race
golangci-lint run

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally and on Github Actions with my changes

The only tests which seem to be failing (exchanges/kucoin, database/models/postgres, config, and common/file) are parts that I haven't substantively changed, and which seem to be failing on master too.

Continual enhance of Coinbase tests The revamp continues Oh jeez the Orderbook part's unfinished don't look Coinbase revamp, Orderbook still unfinished

…otrader into coinbase_api_revamp

V3 done, onto V2 Coinbase revamp nears completion Coinbase revamp nears completion Test commit should fail Coinbase revamp nears completion

shazbert

Thanks heaps for this update, this is looking real good. I have tested the public endpoints, next will be private then websocket testing.

exchanges/asset/asset_test.go

shazbert · 2024-09-18T00:00:51Z

exchanges/coinbasepro/coinbasepro.go

+	startDateString = "start_date"
+	endDateString   = "end_date"
+
+	errIntervalNotSupported = "interval not supported"


this can be an error with errors.New() or use kline package level error.

It can't, since it's used in a function that returns a string, not an error.

return an error then 🚀

engine/currency_state_manager.md

shazbert · 2024-09-18T00:14:30Z

exchanges/coinbasepro/coinbasepro.go

+	errInvalidGranularity     = errors.New("invalid granularity")
+	errOrderFailedToCancel    = errors.New("failed to cancel order")
+	errUnrecognisedStatusType = errors.New("unrecognised status type")
+	errPairEmpty              = errors.New("pair cannot be empty")


This can be a currency package error currency.ErrCurrencyPairEmpty also has error for empty currency Code, there are others for unknown side/type errors in the order package. If you can, just check across these and update to use those package errors please.

exchanges/coinbasepro/coinbasepro.go

exchanges/coinbasepro/coinbasepro_wrapper.go

exchanges/exchange.go

exchanges/stream/buffer/buffer.go

gloriousCode

Nice work cranktakular. Please rebase/merge master. I am happy with the work you have done 🎉

exchanges/coinbasepro/coinbasepro_test.go

exchanges/coinbasepro/coinbasepro_wrapper.go

cmd/exchange_wrapper_standards/exchange_wrapper_standards_test.go

gloriousCode

Happy with your changes, thanks! There's an intermitted exchange_wrapper_standards tests breaking now post-master-merge. I think I'm good once that's sorted

gloriousCode

Just have a question on a comment. Then after that if you could rebase/merge I can approve

exchanges/subscription/template_test.go

gloriousCode

Unfortunately it has been difficult to test the scenario due to CBP websocket not respecting config subscriptions:

 "subscriptions": [
     {
      "enabled": false,
      "channel": "heartbeat"
     },
     {
      "enabled": false,
      "channel": "status"
     },
     {
      "enabled": false,
      "channel": "ticker",
      "asset": "spot"
     },
     {
      "enabled": false,
      "channel": "candles",
      "asset": "spot"
     },
     {
      "enabled": false,
      "channel": "allTrades",
      "asset": "spot"
     },
     {
      "enabled": false,
      "channel": "orderbook",
      "asset": "spot"
     },
     {
      "enabled": true,
      "channel": "account",
      "authenticated": true
     },
     {
      "enabled": false,
      "channel": "ticker_batch",
      "asset": "spot"
     }

[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"candles","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","channel":"heartbeats","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","channel":"user","signature":"POP","api_key":"CAT","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"market_trades","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"level2","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"ticker","timestamp":"1730245901"}

This will need to be fixed

gloriousCode

Not many requests other than the reversion I already commented on. Good stuff

gloriousCode · 2024-11-05T03:06:14Z

exchanges/coinbasepro/coinbasepro_websocket.go

+					Type:            oType,
+					Side:            oSide,
+					Status:          oStatus,
+					AssetType:       asset.Spot,


Given the usage of asset.All for "user" and that this can return productType of either SPOT or FUTURE I think you should be returning the appropriate asset type

gloriousCode · 2024-11-05T03:10:56Z

exchanges/coinbasepro/coinbasepro_websocket.go

+func statusToStandardStatus(stat string) (order.Status, error) {
+	switch stat {
+	case "received":
+		return order.New, nil
+	case "open":
+		return order.Active, nil
+	case "done":
+		return order.Filled, nil
+	case "match":
+		return order.PartiallyFilled, nil
+	case "change", "activate":
+		return order.Active, nil
+	default:
+		return order.UnknownStatus, fmt.Errorf("%w %v", errUnrecognisedStatusType, stat)
+	}
+}


These may have changed since you did this:
https://docs.cdp.coinbase.com/advanced-trade/docs/ws-channels#user-channel
I note that CB seems to be doing things in upper case and I'm not seeing case handling anywhere

gloriousCode · 2024-11-05T03:11:54Z

exchanges/coinbasepro/coinbasepro_websocket.go

+				if err != nil {
+					return warnString, err
+				}
+				var oSide order.Side
+				oSide, err = order.StringToOrderSide(wsUser[i].Orders[j].OrderSide)
+				if err != nil {
+					return warnString, err
+				}
+				var oStatus order.Status
+				oStatus, err = statusToStandardStatus(wsUser[i].Orders[j].Status)
+				if err != nil {
+					return warnString, err
+				}
+				sliToSend = append(sliToSend, order.Detail{


Use the following just for these instances of if err != nil

if err != nil { c.Websocket.DataHandler <- order.ClassificationError{ Exchange: c.Name, Err: err, } }

gloriousCode · 2024-11-05T03:40:37Z

exchanges/protocol/features.go

+	// FullPayloadSubscribe flushes and changes full subscription on websocket
+	// connection by subscribing with full default stream channel list
+	FullPayloadSubscribe              bool `json:"fullPayloadSubscribe,omitempty"`


This appears unused and the only addition

shazbert

Good stuff minor requests.

shazbert · 2024-11-07T00:02:42Z

exchanges/coinbasepro/coinbasepro.go

+	startDateString = "start_date"
+	endDateString   = "end_date"
+
+	errIntervalNotSupported = "interval not supported"


return an error then 🚀

shazbert · 2024-11-07T00:17:57Z

exchanges/coinbasepro/coinbasepro.go

-	if err := c.SendHTTPRequest(ctx, exchange.RestSpot, path, &orderbook); err != nil {
-		return nil, err
+// GetProductBookV3 returns a list of bids/asks for a single product
+func (c *CoinbasePro) GetProductBookV3(ctx context.Context, productID string, limit uint16, authenticated bool) (*ProductBook, error) {


this is not done

shazbert · 2024-11-07T00:27:16Z

exchanges/coinbasepro/coinbasepro.go


-	return fee
+// UnmarshalJSON unmarshals the JSON input into a UnixTimestamp type
+func (t *UnixTimestamp) UnmarshalJSON(b []byte) error {


remove UnixTimestamp type and use types.Time

shazbert · 2024-11-07T00:27:39Z

exchanges/coinbasepro/coinbasepro.go

-	} else if c.Equal(currency.EUR) {
-		fee = 0.15
+// UnmarshalJSON unmarshals the JSON input into a UnixTimestamp type
+func (t *UnixTimestampMilli) UnmarshalJSON(b []byte) error {


remove UnixTimestampMilli and use types.Time

shazbert · 2024-11-07T00:30:36Z

exchanges/coinbasepro/coinbasepro_test.go

 }

 // TestWsAuth dials websocket, sends login request.
 func TestWsAuth(t *testing.T) {
-	if !c.Websocket.IsEnabled() && !c.API.AuthenticatedWebsocketSupport || !sharedtestvalues.AreAPICredentialsSet(c) {
+	// t.Parallel()


Just gonna rm that, since other tests interfere with it.

shazbert · 2024-11-07T00:35:18Z

exchanges/coinbasepro/coinbasepro_types.go

+
+// AmCur is a sub-struct used in ListNotificationsSubData, WalletData, TransactionData,
+// DeposWithdrData, and PaymentMethodData
+type AmCur struct {


AmCur -> AmountWithCurrency

shazbert · 2024-11-07T00:37:32Z

exchanges/coinbasepro/coinbasepro_types.go

+// OrderBookResp holds information on bids and asks for a particular currency pair, used for unmarshalling in
+// GetProductBookV1
+type OrderBookResp struct {
+	Bids        [][3]interface{} `json:"bids"`


please convert interface{} -> any across this PR.

shazbert · 2024-11-07T00:38:11Z

exchanges/coinbasepro/coinbasepro_types.go

+// ValCur is a sub-struct used in the types Account, NativeAndRaw, DetailedPortfolioResponse,
+// FuturesBalanceSummary, ListFuturesSweepsResponse, PerpetualsPortfolioSummary, PerpPositionDetail,
+// FeeStruct, AmScale, and ConvertResponse
+type ValCur struct {


ValCur -> ValueWithCurrency

shazbert · 2024-11-07T00:40:18Z

exchanges/coinbasepro/coinbasepro_websocket.go

+	timestamp, err := time.Parse(time.RFC3339, string(data))
+	if err != nil {
+		return time.Time{}, err
+	}
+	return timestamp, nil


Suggested change

timestamp, err := time.Parse(time.RFC3339, string(data))

if err != nil {

return time.Time{}, err

}

return timestamp, nil

return time.Parse(time.RFC3339, string(data))

shazbert · 2024-11-07T00:41:11Z

exchanges/coinbasepro/coinbasepro_websocket.go

+}
+
+// strategyDecoder is a helper function that converts a Coinbase Pro time in force string to a few standardised bools
+func strategyDecoder(str string) (bool, bool, error) {


linter issue here

shazbert

You are going to have to swap over to using JWT authentication. I cannot test the old key format.

shazbert · 2024-11-25T01:36:10Z

exchanges/coinbasepro/coinbasepro_wrapper.go

 	requestFmt := &currency.PairFormat{Delimiter: currency.DashDelimiter, Uppercase: true}
 	configFmt := &currency.PairFormat{Delimiter: currency.DashDelimiter, Uppercase: true}
-	err := c.SetGlobalPairsManager(requestFmt, configFmt, asset.Spot)
+	err := c.SetGlobalPairsManager(requestFmt, configFmt, asset.Spot, asset.Futures)


Please add:

diff --git a/exchanges/coinbasepro/coinbasepro_wrapper.go b/exchanges/coinbasepro/coinbasepro_wrapper.go index a675e7ba9..8ca87ef5a 100644 --- a/exchanges/coinbasepro/coinbasepro_wrapper.go +++ b/exchanges/coinbasepro/coinbasepro_wrapper.go @@ -44,6 +44,10 @@ func (c *CoinbasePro) SetDefaults() { if err != nil { log.Errorln(log.ExchangeSys, err) } + err = c.DisableAssetWebsocketSupport(asset.Futures) + if err != nil { + log.Errorln(log.ExchangeSys, err) + } c.Features = exchange.Features{ Supports: exchange.FeaturesSupported{ REST: true, } As there are currently no websocket market data support for futures.

You want that done, even though there is futures websocket support, limited to the user's orders?

shazbert · 2024-11-25T01:36:59Z

exchanges/coinbasepro/coinbasepro_wrapper.go

-	sort.Sort(trade.ByDate(resp))
-	return resp, nil
+// GetRecentTrades returns the most recent trades for a currency and asset
+func (c *CoinbasePro) GetRecentTrades(_ context.Context, _ currency.Pair, _ asset.Item) ([]trade.Data, error) {


Suggested change

func (c *CoinbasePro) GetRecentTrades(_ context.Context, _ currency.Pair, _ asset.Item) ([]trade.Data, error) {

func (c *CoinbasePro) GetRecentTrades(context.Context, currency.Pair, asset.Item) ([]trade.Data, error) {

shazbert

Quick review on new changes. Then I can do a thorough review before next week.

shazbert · 2024-12-03T03:49:35Z

exchanges/coinbasepro/coinbasepro.go

+	// The code below mostly works, but seems to lead to bad results on the signature step. Deferring until later
+	// head := map[string]any{"kid": creds.Key, "typ": "JWT", "alg": "ES256", "nonce": nonce}
+	// headJSON, err := json.Marshal(head)
+	// if err != nil {
+	// 	return "", time.Time{}, err
+	// }
+	// headEncode := base64URLEncode(headJSON)
+	// regTime := time.Now()
+	// body := map[string]any{"iss": "cdp", "nbf": regTime.Unix(), "exp": regTime.Add(time.Minute * 2).Unix(), "sub": creds.Key /*, "aud": "retail_rest_api_proxy"*/}
+	// if uri != "" {
+	// 	body["uri"] = uri
+	// }
+	// bodyJSON, err := json.Marshal(body)
+	// if err != nil {
+	// 	return "", time.Time{}, err
+	// }
+	// bodyEncode := base64URLEncode(bodyJSON)
+	// hash := sha256.Sum256([]byte(headEncode + "." + bodyEncode))
+	// sig, err := ecdsa.SignASN1(rand.Reader, key, hash[:])
+	// if err != nil {
+	// 	return "", time.Time{}, err
+	// }
+	// sigEncode := base64URLEncode(sig)
+	// return headEncode + "." + bodyEncode + "." + sigEncode, regTime, nil


What is this for?

As comment at the top of the block says, it's my implementation of the JWT protocol, so we won't have to import a separate package to handle that. Alas, there's some issue I can't identify with generating the signature, which has it create an invalid one, so this can't be used right now.

shazbert · 2024-12-03T03:55:54Z

exchanges/coinbasepro/coinbasepro_websocket.go

-	return headEncode + "." + bodyEncode + "." + sigEncode, nil
+// GetWSJWT returns a JWT, using a stored one of it's provided, and generating a new one otherwise
+func (c *CoinbasePro) GetWSJWT() (string, error) {
+	c.mut.RLock()


suggestion: You can simplify this with a normal mutex and just defer the unlock. Then you don't need to worry about most of whats going on below. We can optimise later if this turns into a bottleneck which it most likely won't be.

Is there a point in doing that now, considering how I've already written this handling?

If so, let me know, and I'll refactor it.

shazbert · 2024-12-03T03:57:00Z

exchanges/coinbasepro/coinbasepro_websocket.go

+// GetWSJWT returns a JWT, using a stored one of it's provided, and generating a new one otherwise
+func (c *CoinbasePro) GetWSJWT() (string, error) {
+	c.mut.RLock()
+	if c.jwtLastRegen.Add(time.Minute * 2).After(time.Now()) {


instead of JWTLasTRegen just assign an expiry value so the addition op doesn't need to happen.

cranktakular added 28 commits October 9, 2023 08:42

Slight enhance of Coinbase tests

a006163

Continual enhance of Coinbase tests The revamp continues Oh jeez the Orderbook part's unfinished don't look Coinbase revamp, Orderbook still unfinished

Merge branch 'master' into coinbase_api_revamp

074f367

Coinbase revamp; CreateReport is still WIP

bd9f2dd

More coinbase improvements; onto sandbox testing

25fcd2e

Coinbase revamp continues

5d3fd99

Coinbase revamp continues

3aca36b

Merge branch 'coinbase_api_revamp' of github.com:cranktakular/gocrypt…

f58b76d

…otrader into coinbase_api_revamp

Coinbasepro revamp is ceaseless

eded160

Coinbase revamp, starting on advanced trade API

5a0ba23

Coinbase Advanced Trade Starts in Ernest

88e6713

V3 done, onto V2 Coinbase revamp nears completion Coinbase revamp nears completion Test commit should fail Coinbase revamp nears completion

Coinbase revamp stage wrapper

aa2efef

Merge branch 'master' into coinbase_api_revamp

3560599

Coinbase wrapper coherence continues

4593828

Coinbase wrapper continues writhing

a8cae40

Coinbase wrapper & codebase cleanup

0564e45

Coinbase updates & wrap progress

4f5ccb5

More Coinbase wrapper progress

9a6c7e6

Wrapper is wrapped, kinda

2896b3e

Test & type checking

ffb43d8

Coinbase REST revamp finished

7dd4589

Merge branch 'master' into coinbase_api_revamp

6d7ac5f

Post-merge fix

2881841

WS revamp begins

2166886

WS Main Revamp Done?

35eda67

CB websocket tidying up

6ac86cd

Coinbase WS wrapperupperer

cfe6e11

Coinbase revamp done??

2f770d5

Merge branch 'master' into coinbase_api_revamp

3e54528

thrasher- changed the title ~~Coinbase api revamp~~ Coinbase: Update exchange implementation Feb 14, 2024

thrasher- requested a review from a team February 14, 2024 23:40

Json formatting

d797e69

shazbert requested changes Sep 18, 2024

View reviewed changes

cranktakular added 3 commits September 24, 2024 16:56

ShazNits

9925c88

(L/N)i(n/)t

c88b057

Merge master and minor fixes

0eeb73c

gloriousCode requested changes Oct 16, 2024

View reviewed changes

cranktakular added 3 commits October 16, 2024 13:44

Merge master and minor fixes

4acb64d

Adding a test

5974421

Tiny test improvement

2a90b5d

gloriousCode requested changes Oct 16, 2024

View reviewed changes

gloriousCode mentioned this pull request Oct 23, 2024

Subscriptions: Relax subscription validation for non-existent pairs #1635

Merged

1 task

gloriousCode reviewed Oct 28, 2024

View reviewed changes

exchanges/subscription/template_test.go Outdated Show resolved Hide resolved

Merge branch 'master' into coinbase_api_revamp

3f6976c

gloriousCode requested changes Oct 29, 2024

View reviewed changes

Template patch testing

3eb15db

gloriousCode requested changes Nov 5, 2024

View reviewed changes

cranktakular added 2 commits November 5, 2024 17:02

Fixes

38f2765

Merge branch 'master' into coinbase_api_revamp

2cc9d0c

shazbert requested changes Nov 7, 2024

View reviewed changes

cranktakular added 3 commits November 12, 2024 11:52

Further shaznits

2fd8bb1

Merge branch 'master' into coinbase_api_revamp

1229376

Lint nit

a4252ec

shazbert requested changes Nov 25, 2024

View reviewed changes

cranktakular added 2 commits November 27, 2024 18:03

JWT move and other fixes

ff6ce5f

Small nits

19fbf09

shazbert requested changes Dec 3, 2024

View reviewed changes

cranktakular added 3 commits December 12, 2024 10:16

Shaznit, singular

e1ba339

Merge branch 'master' into coinbase_api_revamp

59a9d7b

Post-merge fix

80cb272

thrasher- mentioned this pull request Dec 18, 2024

exchanges: Fix GateIO/Coinbase test failures and OKX race #1753

Open

11 tasks

	func (c *CoinbasePro) GetRecentTrades(_ context.Context, _ currency.Pair, _ asset.Item) ([]trade.Data, error) {
	func (c *CoinbasePro) GetRecentTrades(context.Context, currency.Pair, asset.Item) ([]trade.Data, error) {

Coinbase: Update exchange implementation #1480

Are you sure you want to change the base?

Coinbase: Update exchange implementation #1480

Conversation

cranktakular commented Feb 14, 2024 • edited Loading

PR Description

Type of change

How has this been tested

Checklist

shazbert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gloriousCode left a comment

Choose a reason for hiding this comment

gloriousCode left a comment • edited Loading

Choose a reason for hiding this comment

gloriousCode left a comment

Choose a reason for hiding this comment

gloriousCode left a comment

Choose a reason for hiding this comment

gloriousCode left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shazbert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shazbert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shazbert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cranktakular commented Feb 14, 2024 •

edited

Loading

gloriousCode left a comment •

edited

Loading