Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

currency: Adds matching lookup table built from available pairs. #1312

Merged
merged 25 commits into from
Oct 18, 2023

Conversation

shazbert
Copy link
Collaborator

@shazbert shazbert commented Aug 9, 2023

PR Description

@gloriousCode brought up an issue with the old matching system with respective benchmarks here sometimes when using this super fast function can result in incorrect return pair if there is no delimiter. These benchmarks can be run again.

  • Adds matching lookup table that is built from available pairs list for a faster return that doesn't impact currency package NewPairFromString and derivatives which call currency.NewCode() * 2
  • Adds optimisation to NewPairFromString for even speedier usage 💨
  • Adds new exchange base function for the matching of symbols to stored pairs MatchSymbolWithAvailablePairs
  • If type currency.Pair is used for a json field this has become more strict and requires now a delimiter as we have to make sure that we are correctly formatting to match with our stored list.

Note: Keeping this as non-invasive as possible, separate PR's can be opened when needed if MatchSymbolWithAvailablePairs is found to be a better use case for speed considerations (mainly websocket throughput).

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How has this been tested

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and
also consider improving test coverage whilst working on a certain feature or package.

  • go test ./... -race
  • golangci-lint run
  • Test X

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally and on Github Actions/AppVeyor with my changes
  • Any dependent changes have been merged and published in downstream modules

@shazbert shazbert requested a review from a team August 9, 2023 05:21
@shazbert shazbert self-assigned this Aug 9, 2023
@thrasher- thrasher- added the review me This pull request is ready for review label Aug 10, 2023
@shazbert shazbert removed the blocked label Aug 10, 2023
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice 🍐💨

currency/manager.go Outdated Show resolved Hide resolved
currency/manager.go Outdated Show resolved Hide resolved
currency/pair_methods.go Show resolved Hide resolved
// MatchSymbolWithAvailablePairs returns a currency pair based on the supplied
// symbol and asset type. If the string is expected to have a delimiter this
// will attempt to screen it out.
func (b *Base) MatchSymbolWithAvailablePairs(symbol string, a asset.Item, hasDelimiter bool) (currency.Pair, error) {
Copy link
Collaborator

@gloriousCode gloriousCode Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So utilising the same benchmarking as last time for comparison:

func BenchmarkNewPairFromString(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		_, _ = currency.NewPairFromString("ZTG_USDT")
	}
}

func BenchmarkDeriveFrom(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		avail, _ := g.GetAvailablePairs(asset.Spot)
		_, _ = avail.DeriveFrom(strings.Replace("ZTG_USDT", "_", "", 1))
	}
}

func BenchmarkDeriveFromCheating(b *testing.B) {
	b.ReportAllocs()
	avail, _ := g.GetAvailablePairs(asset.Spot)
	for i := 0; i < b.N; i++ {
		_, _ = avail.DeriveFrom(strings.Replace("ZTG_USDT", "_", "", 1))
	}
}

func BenchmarkMatchSymbolWithAvailablePairs(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		_, _ = g.MatchSymbolWithAvailablePairs("ZTG_USDT", asset.Spot, true)
	}
}
Benchmark times tested time taken Bytes per op Allocations
BenchmarkNewPairFromString-10 20982969 57.07 ns/op 4 B/op 1 allocs/op
BenchmarkDeriveFrom-10 17596 67605 ns/op 294928 B/op 4 allocs/op
BenchmarkDeriveFromCheating-10 156580 7163 ns/op 17 B/op 2 allocs/op
BenchmarkMatchSymbolWithAvailablePairs-10 15378223 76.50 ns/op 16 B/op 2 allocs/op

Does show a lovely improvement. Would you be considering the removal of Derivefrom? Because your new feature seems underutilised in this PR

Plus very nice work on the PairFromString improvements!

Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Just need to update Bybit to be filtering on enabled pairs to not change behaviour

currency/manager_types.go Show resolved Hide resolved
currency/manager_types.go Outdated Show resolved Hide resolved
currency/manager.go Outdated Show resolved Hide resolved
exchanges/bybit/bybit_websocket.go Outdated Show resolved Hide resolved
exchanges/bybit/bybit_wrapper.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with this. I'm already wanting to use it
Please update tests:

--- FAIL: TestUpdateTicker (1.85s)
    bitmex_test.go:1103: pair not found for xrpusd perpetualcontract
--- FAIL: TestUpdateTickers (6.73s)
    bitfinex_test.go:504: pair not found for xautf0btcf0 spot
--- FAIL: TestUpdateTicker (0.32s)
    bitfinex_test.go:477: pair not found for btcf0eutf0 spot
--- FAIL: TestGetFundingRates (0.00s)
    rpcserver_test.go:3031: received: 'pair matcher is nil' but expected: '<nil>'
  --- FAIL: TestAllExchangeWrappers/coinut_wrapper_tests (0.43s)
        --- FAIL: TestAllExchangeWrappers/coinut_wrapper_tests/MatchSymbolWithAvailablePairs-- (0.00s)
            exchange_wrapper_standards_test.go:214: COINUT Func 'MatchSymbolWithAvailablePairs' Error: 'pair not found for 1337 '. Inputs: [1337  false].

And any others. Which now fail due from this update

exchanges/bitmex/bitmex_test.go Outdated Show resolved Hide resolved
exchanges/bitmex/bitmex_wrapper.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for that, just finishing up I think. I know this is annoying 😭

exchanges/bybit/bybit_wrapper.go Show resolved Hide resolved
exchanges/exmo/exmo_wrapper.go Outdated Show resolved Hide resolved
exchanges/hitbtc/hitbtc_wrapper.go Outdated Show resolved Hide resolved
@gloriousCode gloriousCode added reconstructing Based on PR feedback, this is currently being reworked and is not to be merged and removed review me This pull request is ready for review labels Sep 21, 2023
@shazbert shazbert added review me This pull request is ready for review and removed reconstructing Based on PR feedback, this is currently being reworked and is not to be merged labels Sep 28, 2023
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one thing I noticed, plus please rebase/merge and check that tests are good 🤙

exchanges/exmo/exmo_wrapper.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK! Nice work Shazbert

return nil
}

// Has Fast path optimisation when int == 64
i, err := strconv.Atoi(s)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is why we try not to use Atoi, thanks for the fix!

@gloriousCode gloriousCode added the gcrc GloriousCode Review Complete label Oct 6, 2023
Copy link
Collaborator

@thrasher- thrasher- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @shazbert !

@thrasher- thrasher- merged commit ceef7a1 into thrasher-corp:master Oct 18, 2023
shazbert added a commit to shazbert/gocryptotrader that referenced this pull request Nov 10, 2023
…sher-corp#1312)

* currency: Add pair matching update (cherry-pick)

* exchange/currency: Add tests and update func

* linter fix, also if using json unmarshal functionality stop usage of string conversion without delimiter

* gemini: fix test

* currency/manager: potential optimisation

* exchanges: purge derive from wrapper cases and add warning comment

* glorious: nits

* glorious: nits

* linter: fix

* glorious: nits

* whoops

* whoops

* glorious: nits continued

* glorious: diff THANKS!

* hitbtc: fix update tradable pairs strings splitting. continue if not enabled tickers update pair.

* glorious: nits

* linter: fix

* Update exchanges/exmo/exmo_wrapper.go

Co-authored-by: Scott <[email protected]>

* bitstamp: fix test when 32 biterinos architecturinos

* capture more strings for speed

* swapsies because whos running 32bit \0/?

---------

Co-authored-by: Ryan O'Hara-Reid <[email protected]>
Co-authored-by: Scott <[email protected]>
@shazbert shazbert deleted the currency_matcher branch May 3, 2024 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gcrc GloriousCode Review Complete review me This pull request is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants