Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pricenode consistency tracking #39

Open
ghost opened this issue Oct 31, 2023 · 8 comments
Open

Pricenode consistency tracking #39

ghost opened this issue Oct 31, 2023 · 8 comments

Comments

@ghost
Copy link

ghost commented Oct 31, 2023 •
edited by ghost
Loading

Overview

Several users have expressed concerns with the XMR rates in Matrix chat, e.g:

  • one pricenode may have a divergent price other pricenodes.
  • sometimes consistently divergent rates have been seen.
  • rates are not updated quickly enough.
  • a trade may be taken at a slightly disadvantageous rate, resulting in their profit margin being wiped out.

There have been some investigations and fixes already (#18, #28, #33). Also, due to unexplained production issues one pricenode contained stale Poloniex rates for 1.5 months (Aug 15 to Oct 8th) and this may be partly the driver for user complaints. However there may be other things that can be improved. The purpose of this issue is to closely monitor the production pricenodes over several weeks, documenting any problems found and brainstorming possible improvements. It is hoped this will lead to gaining an understanding of what the pricenode can deliver in terms of price accuracy.

Background on why some variability can be seen.

Pricenodes individually poll their exchanges once per minute, updating their internal data store with the new rates. Those rates are averaged and outliers removed at the point of responding to Bisq client requests. Pricenodes activity are not synchronized nor are Bisq requests synchronized to any completion point of the Pricenode's backend cycle. Bisq clients A and B will get slightly different rates from Pricenode P if their requests are made just 30 seconds apart. Likewise, Bisq clients A and B connected to Pricenodes P and Q respectively, will get slightly different rates even if they make their queries at exactly the same time (because P and Q are operating independently).

Methodology

A script is being run that logs the XMR rates from each pricenode, the timestamp and percentage variability. This will enable us to see what typical divergences happen, and for how long do they persist.

I'll check the results daily and document any observations under this issue.

Post Mortem

At the beginning of checking consistency, an issue was observed where one node's prices deviated from the others due to a bug. This was alleviated by #36 and since deployment the issue has not manifested in the form of price discrepancies. However it seems likely the root cause has not yet been identified. This example shows the need for some form of error monitoring as requested here, since without it there's only reporting from attentive users.

From observing consistency below, 0.5% is the typical max deviation between pricenodes, with 0.1% to 0.2% being normal. In each of the cases the difference seems to be timing: each pricenode polls their providers once per minute so any variations in price during a minute can reflect as deviations between nodes. There are many chart examples of that below. Issue to consider improving this.
@ghost
Copy link
Author

ghost commented Oct 31, 2023

Here is an example of the largest divergence observed during the first 12 hours of monitoring. It was just over half a percent difference and lasted just one query interval (one minute). Afterwards, results stayed consistent between pricenodes. This shows that a quick price decline is adopted by each pricenode independently during their poll of providers. emzy polled before the decline, ro7nv during, and devin towards the end.

image

Raw data 
1698523566|emzyp|0.0047501|devin|0.0047501|ro7nv|0.0047501|diff|0.000000|
1698523626|emzyp|0.004750766666666666|devin|0.004750766666666666|ro7nv|0.004750766666666666|diff|0.000000|
1698523686|emzyp|0.0047511|devin|0.0047511|ro7nv|0.0047511|diff|0.000000|
1698523746|emzyp|0.0047501|devin|0.0047501|ro7nv|0.0047511|diff|0.000210|
1698523806|emzyp|0.0047501|devin|0.0047501|ro7nv|0.0047501|diff|0.000000|
1698523866|emzyp|0.0047501|devin|0.0047501|ro7nv|0.0047501|diff|0.000000|
1698523926|emzyp|0.004750766666666666|devin|0.004725333333333333|ro7nv|0.004737|diff|0.005380|ALERT!
1698523986|emzyp|0.0047174999999999995|devin|0.0047174999999999995|ro7nv|0.0047174999999999995|diff|0.000000|
1698524046|emzyp|0.004720133333333334|devin|0.004720133333333334|ro7nv|0.0047198|diff|0.000070|
1698524106|emzyp|0.004719333333333333|devin|0.0047206999999999995|ro7nv|0.004719333333333333|diff|0.000280|
1698524166|emzyp|0.0047206999999999995|devin|0.0047206999999999995|ro7nv|0.0047206999999999995|diff|0.000000|

@ghost
Copy link
Author

ghost commented Oct 31, 2023 •
edited by ghost
Loading

image

Here a discrepancy occurred with emzy node that lasted an extended period. At its peak the discrepancy was .634 of a percent. The timestamps range from 1698552546 (GMT: Sunday, October 29, 2023 4:09:06 AM) to 1698567366 (GMT: Sunday, October 29, 2023 8:16:06 AM). The timestamps had to be inferred after obtaining the data, so may not be completely accurate.

I have no explanation for this at the moment, perhaps checking logs may help, if they are available.

[EDIT] Emzy's logs showed that Binance stopped being polled for rates, and thus the pricenode was stuck with stale Binance rates for 20 hours. The stale XMR rate was 0.004757 and sometimes it was filtered as an outlier, sometimes it was not far enough to filter and therefore affected the average calculation. PR #36 (recently adopted) may help, by detecting and erasing stale rates. The root cause of the outage still needs to be addressed.

@ghost ghost mentioned this issue Oct 31, 2023
Open
@ghost
Copy link
Author

ghost commented Nov 1, 2023

File 1: October 28, 2023 4:10:06 AM (1698466206 - 1698678126)

One anomaly detected (middle of chart), reported above.

image

File 2: October 30, 2023 4:52:06 PM (1698684726 - 1698702066)

No issues.

image

File 3: October 30, 2023 9:46:41 PM (1698702401 - 1698753262)

No issues.

image

File 4: October 31, 2023 11:56:31 AM (1698753391 - 1698852686)

No issues.

image

@ghost
Copy link
Author

ghost commented Nov 2, 2023 •
edited by ghost
Loading

File 5: Wednesday, November 1, 2023 3:32:03 PM (1698852723 - 1698932621)

image

No issues, only one alert (0.7%) which was a rapid decline similar to the example already mentioned above.

image

1698891562	emzyp	0.004812	devin	0.004812	ro7nv	0.004812	diff	0	
1698891624	emzyp	0.0048090667	devin	0.004812	ro7nv	0.004812	diff	0.000609	
1698891686	emzyp	0.0047675	devin	0.0048024	ro7nv		        diff	0.00732	 ALERT!
1698891755	emzyp	0.0047685	devin	0.0047675	ro7nv	0.004767	diff	0.000314	
1698891817	emzyp	0.0047747	devin	0.0047758	ro7nv	0.0047751333	diff	0.00023

File 6: Thursday, November 2, 2023 1:44:39 PM (1698932679 - 1699021525)

No sustained discrepancies, max deviation 0.29%

File 7: Friday, November 3, 2023 2:26:28 PM (1699021588 - 1699111604)

No sustained discrepancies, max deviation 0.25%

File 8: Saturday, November 4, 2023 4:29:45 PM (1699115385 - 1699220096)

No sustained discrepancies, max deviation 0.2568%

2 outages observed (6 minutes: emzyp,devin) and (47 minutes: devin), shown below:

image

@ghost
Copy link
Author

ghost commented Nov 7, 2023 •
edited by ghost
Loading

File 9: Sunday, November 5, 2023 10:24:26 PM (1699223066 - 1699293823)

No sustained discrepancies, max deviation 0.4631%

File 10: Monday, November 6, 2023 6:06:04 PM (1699293964 - 1699396379)

No sustained discrepancies, max deviation 0.5786% - a sudden price drop shown below in column 4. Was caught earlier by devin; one minute later by the other two nodes.

image

File 11: Tuesday, November 7, 2023 10:33:30 PM (1699396410 - 1699460255)

No sustained discrepancies, max deviation 0.2385%

File 12 : Wednesday, November 8, 2023 4:18:20 PM (1699460300 - 1699571774)

max deviation 1.7531% - a sudden price drop, emzy node polling the value later than the other two nodes.

image

Below chart: emzy node perhaps displaying symptoms of issue 33. Issue is that a provider has ceased operation, therefore that node's calculated average will likely be off until cleared by node restart.

image

File 13 : Thursday, November 9, 2023 11:17:43 PM (1699571863 - 1699638925)

max deviation 0.5334%

Below chart: emzy still displaying symptoms of issue 33. I'll request a reboot and check the logs.

image

File 14 : Friday, November 10, 2023 5:59:40 PM (1699639180 - 1699743168)

No sustained discrepancies after reboot, max deviation 0.3257%

@ghost
Copy link
Author

ghost commented Nov 13, 2023 •
edited by ghost
Loading

File 15: Saturday, November 11, 2023 10:53:58 PM (1699743238 - 1699814402)

max deviation 0.3321%

File 16: Sunday, November 12, 2023 6:40:59 PM (1699814459 - 1699889405)

ro7nv outage 3 hours: (1699823340 - 1699833429)
max deviation 0.2361%

File 17: Monday, November 13, 2023 3:59:35 PM (1699891175 - 1699981939)

max deviation 0.3459%

File 18: Tuesday, November 14, 2023 5:15:40 PM (1699982140 - 1700062651)

max deviation 0.7029% : an unexplained 1 candle blip from emzy node.
1699988052 0.0045650667 0.0045332 0.0045332 diff 0.007029 ALERT!
Other than that, max deviation was 0.3702%

File 19: Wednesday, November 15, 2023 3:42:41 PM (1700062961 - 1700160274)

A sudden drop of 1.5887% in which one node lagged the others for 1 minute.
Other than that, max deviation was 0.3438%

File 20: Thursday, November 16, 2023 6:51:55 PM (1700160715 - 1700240419)

A sudden increase of 0.5244% in which one node lagged the others for 1 minute.
Other than that, max deviation was 0.4769%

@ghost ghost mentioned this issue Nov 17, 2023
Open
@ghost
Copy link
Author

ghost commented Nov 21, 2023 •
edited by ghost
Loading

File 21: Friday, November 17, 2023 5:13:14 PM (1700241194 - 1700337761)

max deviation 0.2973%

File 22: Saturday, November 18, 2023 8:07:07 PM (1700338027 - 1700436695)

max deviation 0.2494%

File 23: Sunday, November 19, 2023 11:32:18 PM (1700436738 - 1700495993)

max deviation 0.5103%

image

File 24: Monday, November 20, 2023 4:02:06 PM (1700496126 - 1700591384)

max deviation 0.4065%

image

File 25: Tuesday, November 21, 2023 6:47:22 PM (1700592442 - 1700672565)

max deviation 0.4678%

File 26: Wednesday, November 22, 2023 5:05:51 PM (1700672751 - 1700757631)

max deviation 0.5333%

image

@ghost
Copy link
Author

ghost commented Dec 4, 2023

File 27: Thursday, November 23, 2023 4:43:30 PM (1700757810 - 1700861060)

max deviation 0.5463%

image

File 28: Friday, November 24, 2023 9:29:23 PM (1700861363 - 1700948958)

max deviation 0.1948%

File 29: Saturday, November 25, 2023 9:49:54 PM (1700948994 - 1701033041)

max deviation 0.5932%

File 30: Sunday, November 26, 2023 9:11:47 PM (1701033107 - 1701119934)

max deviation 0.3628%

File 31: Monday, November 27, 2023 9:20:49 PM (1701120049 - 1701188659)

max deviation 0.2974%

@ghost ghost mentioned this issue Dec 4, 2023
Open
@ghost ghost mentioned this issue Dec 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
0 participants