docs/utils: add decoy selection implementation guides and tools #9024

jeffro256 · 2023-10-17T20:29:28Z

Documentation to understand/implement non-fingerprinting decoy selection plus a reference code to empirically test correctness. If anyone wants c++ source for an executable that empirically proves the correctness of the Python script, I might create a new branch on Github and release that code.

hinto-janai · 2023-12-13T13:55:29Z

docs/DECOY_SELECTION.md

+
+Finally, when we are doing decoy selection to find the other members of a ring, our result is a list of global output
+indexes, which represent a set of transaction outputs with the same amount as the transaction output we are trying to
+spend. We sample these global output indicies according to a certain distribution, with this distribution hopefully


Suggested change

spend. We sample these global output indicies according to a certain distribution, with this distribution hopefully

spend. We sample these global output indices according to a certain distribution, with this distribution hopefully

hinto-janai · 2023-12-13T13:55:55Z

docs/DECOY_SELECTION.md

+Finally, when we are doing decoy selection to find the other members of a ring, our result is a list of global output
+indexes, which represent a set of transaction outputs with the same amount as the transaction output we are trying to
+spend. We sample these global output indicies according to a certain distribution, with this distribution hopefully
+statistically matching the distribution of the ages of "true spends", so that the ring member we truely wish to spend is


Suggested change

statistically matching the distribution of the ages of "true spends", so that the ring member we truely wish to spend is

statistically matching the distribution of the ages of "true spends", so that the ring member we truly wish to spend is

hinto-janai · 2023-12-13T13:56:05Z

docs/DECOY_SELECTION.md

+statistical dependence for picks within rings more than necessary. When you are trying to build up a set of X unique
+decoy picks, if the first pick has 100 choices, then the next pick has 99 choices, then 98 choices, etc, etc. Since
+these picks are not statistically independent, then the distribution of the picks gets more and more skewed for the later
+picks. You can combat this effect by simply commiting to the order in which you pick the outputs, and try adding them


Suggested change

picks. You can combat this effect by simply commiting to the order in which you pick the outputs, and try adding them

picks. You can combat this effect by simply committing to the order in which you pick the outputs, and try adding them

hinto-janai · 2023-12-13T13:56:17Z

docs/DECOY_SELECTION.md

+### First, Some Numeric Constants
+
+* `GAMMA_SHAPE = 19.28` [source](https://github.com/monero-project/monero/blob/67d190ce7c33602b6a3b804f633ee1ddb7fbb4a1/src/wallet/wallet2.cpp#L141-L142)
+  * Shape paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)


Suggested change

* Shape paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)

* Shape parameter for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)

hinto-janai · 2023-12-13T13:56:27Z

docs/DECOY_SELECTION.md

+* `GAMMA_SHAPE = 19.28` [source](https://github.com/monero-project/monero/blob/67d190ce7c33602b6a3b804f633ee1ddb7fbb4a1/src/wallet/wallet2.cpp#L141-L142)
+  * Shape paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)
+* `GAMMA_RATE = 1.61` [source](https://github.com/monero-project/monero/blob/67d190ce7c33602b6a3b804f633ee1ddb7fbb4a1/src/wallet/wallet2.cpp#L141-L142)
+  * Rate paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)


Suggested change

* Rate paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)

* Rate parameter for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)

hinto-janai · 2023-12-13T13:58:58Z

docs/DECOY_SELECTION.md

+        psuedo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock
+
+        # 7
+        picked_block_index = bisect.bisect_left(crod, psuedo_global_output_index)


Suggested change

picked_block_index = bisect.bisect_left(crod, psuedo_global_output_index)

picked_block_index = bisect.bisect_left(crod, pseudo_global_output_index)

hinto-janai · 2023-12-13T13:59:17Z

docs/DECOY_SELECTION.md

+distribution, we can use a two-sample [Kolmogorov–Smirnov Test](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test)
+to statistically test if a given implementation statistically matches the reference implementation. Running the provided
+Python decoy selection reference script (utils/python-rpc/decoy_selection.py) will generate a TXT file containing
+decoy selection picks (you can specify how many) seperated by newlines. This data can be imported and used to perform


Suggested change

decoy selection picks (you can specify how many) seperated by newlines. This data can be imported and used to perform

decoy selection picks (you can specify how many) separated by newlines. This data can be imported and used to perform

hinto-janai · 2023-12-13T13:59:59Z

utils/python-rpc/decoy_selection.py

+            continue
+
+        # 6
+        psuedo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock


Suggested change

psuedo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock

pseudo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock

hinto-janai · 2023-12-13T14:00:05Z

utils/python-rpc/decoy_selection.py

+        psuedo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock
+
+        # 7
+        picked_block_index = bisect.bisect_left(crod, psuedo_global_output_index)


Suggested change

picked_block_index = bisect.bisect_left(crod, psuedo_global_output_index)

picked_block_index = bisect.bisect_left(crod, pseudo_global_output_index)

hinto-janai · 2023-12-13T14:00:16Z

utils/python-rpc/decoy_selection.py

+def main():
+    # Handle CLI arguments
+    arg_parser = argparse.ArgumentParser(prog='Decoy Selection Python Reference',
+                    description='We provide an easy-to-read non-fingerprinting reference for Monero decoy selecton',


Suggested change

description='We provide an easy-to-read non-fingerprinting reference for Monero decoy selecton',

description='We provide an easy-to-read non-fingerprinting reference for Monero decoy selection',

hinto-janai

Examples, links, python scripts all work.

$ time ./decoy_selection.py -p 18082 -n 1024

Fetching the CROD up to height <top> from daemon at '127.0.0.1:18082'...
The start height of the CROD is 1220516, and the top height is 3039030.
Performing 1024 picks and writing output to 'python_decoy_selections.txt'...
Progress: 100.0%

real	0m0.868s
user	0m0.397s
sys	0m0.072s

python_decoy_selections.txt (1024 lines)

iamamyth · 2023-12-14T19:23:46Z

utils/python-rpc/decoy_selection.py

+import argparse
+import bisect
+try:
+    import numpy as np


Why require numpy? It's an incredibly large dependency for a very small set of features which the built-in random module also provides.

j-berman

Appreciate this :) This is useful documentation that I'm already putting to good use.

One thing: the reference impl doesn't check to make sure selected decoys are unlocked, so coinbase outputs that are <60 blocks old or timelocked outputs can get selected. This skews the distribution a bit compared to a correct implementation.

Suggestion: use the get_outs RPC for selected outputs and check if they're unlocked. Re-pick for any locked outputs.

(I'm currently testing this reference impl against monero-serai)

j-berman · 2023-12-19T08:15:26Z

I implemented the above suggestion (made sure this reference implementation only selects unlocked outputs), compared the result to the DSA I implemented in monero-serai, and found that it passed the Kolmogorov-Smirnov test. It did not pass the KS test before implementing the suggestion (also thank you to @Rucknium for helping me validate the KS test).

I think it's worth making sure the reference implementation does not select locked outputs as well. Here was my code:

    # Do gamma picking and write output
    print("Performing {} picks and writing output to '{}'...".format(args.num_picks, args.output_file))
    print_period = args.num_picks // 1000 if args.num_picks >= 1000 else 1
    batch_size = 10_000

    with open(args.output_file, 'w', newline='') as outf:
        batch_of_picks = []
        for i in range(args.num_picks):
            if (i+1) % print_period == 0:
                progress = (i+1) / args.num_picks * 100
                print("Progress: {:.1f}%".format(progress), end='\r')

            batch_of_picks.append(gamma_pick(crod, average_output_delay, num_usable_rct_outputs))

            if (i+1) % batch_size == 0 or i == (args.num_picks - 1):
                res = daemon.get_outs([{'amount': 0, 'index': pick} for pick in batch_of_picks], get_txid = False)
                assert len(res.outs) == len(batch_of_picks)
                j = 0
                for out in res.outs:
                    pick = batch_of_picks[j]

                    # Pick a decoy until we find one that is unlocked
                    unlocked = out.unlocked
                    while not unlocked:
                        pick = gamma_pick(crod, average_output_delay, num_usable_rct_outputs)
                        backup_res = daemon.get_outs([{'amount': 0, 'index': pick}], get_txid = False)
                        assert len(backup_res.outs) == 1
                        unlocked = backup_res.outs[0].unlocked

                    print(pick, file=outf)

                    j += 1

                batch_of_picks = []

With the above implemented, I also think the reference implementation should remove support for the to_height argument, since the unlocked status of an output from the get_outs RPC depends on the current height. I also think the documentation should clarify that the chain height must match when checking distributions. Perhaps the .txt filename could have the daemon's height appended to it.

In the same vein, I also think it's worth checking that the top block hash at the start of the script is the same at the end, like this:

    # Get daemon height and block hash
    res = daemon.get_info()
    height = res.height
    top_block_hash = res.top_block_hash
    
    
 ...
 
 
    # Make sure height is the same as when we started so that we can be certain
    # output locked status stays consistent
    res = daemon.get_info()
    if res.height != height or res.top_block_hash != top_block_hash:
        print("Error: the chain height advanced while the script was running. This can harm the analysis.")
        exit(1)

When I'm testing, I disconnect my daemon from peers to make sure the height remains constant.

It would be nice if a user could run this script^ even if they can't point to a disconnected daemon, but I don't believe it would be correct to do so without some changes to the get_outs RPC (that makes sure that timelocked outputs are unlocked at a specific height).

Final point, I think it would be helpful if the documentation explained how to run the KS test. It's super simple, just call scipy.stats.kstest(reference_global_output_indexes, implementation_global_output_indexes) and see if the p-value is greater than 0.05. If it's greater than 0.05, that means we cannot reject the null hypothesis (the null hypothesis is that the two distributions are identical).

EDIT: removed extra height param in my sample code above spotted by @jeffro256

jeffro256 · 2023-12-26T23:37:02Z

@iamamyth AFAIK the "random" module doesn't have a way to randomly sample from a gamma distribution Dec 14, 2023, 13:23 by ***@***.***:

…

@iamamyth> commented on this pull request. In > utils/python-rpc/decoy_selection.py <#9024 (comment)>> : > @@ -0,0 +1,150 @@+#!/usr/bin/python3++''' See step-by-step accompaniment in docs/DECOY_SELECTION.md '''++import argparse+import bisect+try:+ import numpy as np Why require numpy? It's an incredibly large dependency for a very small set of features which the built-in random module also provides. — Reply to this email directly, > view it on GitHub <#9024 (review)>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSWLOT5FMAIAZK5TS6MOL3YJNG45AVCNFSM6AAAAAA6ENRC6CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTOOBSGU2TGNBQGY>> . You are receiving this because you authored the thread.> Message ID: > <monero-project/monero/pull/9024/review/1782553406> @> github> .> com>

iamamyth · 2023-12-28T16:07:46Z

https://docs.python.org/3/library/random.html#random.gammavariate

jeffro256 · 2023-12-28T19:40:58Z

Okay the reference code now doesn't select locked outputs and also doesn't use numpy. I'll do some KS tests soon to test if it matches the behavior of wallet2. The reference code also throws an error if the top block hash changes

docs/utils: add decoy selection implementation guides and tools

fa94530

Gingeropolous mentioned this pull request Oct 20, 2023

wallet: mitigate statistical dependence for decoy selection within rings #9023

Merged

change average_output_flow -> average_output_delay for clarity

9103f00

0xFFFC0000 added the pending review label Dec 7, 2023

hinto-janai reviewed Dec 13, 2023

View reviewed changes

fix typos (@hinto-janai)

813ad56

hinto-janai approved these changes Dec 13, 2023

View reviewed changes

0xFFFC0000 added the documentation label Dec 14, 2023

iamamyth reviewed Dec 14, 2023

View reviewed changes

j-berman reviewed Dec 15, 2023

View reviewed changes

j-berman mentioned this pull request Dec 27, 2023

Monero: fix decoy selection algo and add test for latest spendable serai-dex/serai#384

Merged

dont select locked outputs and remove numpy dep

6f41197

Rucknium mentioned this pull request Oct 25, 2024

Rucknium's review of “Uniformly Most Powerful Tests for Ad Hoc Transactions in Monero” cypherstack/churn#2

Open

Rucknium mentioned this pull request Dec 20, 2024

Change "triangular" to "gamma" in ringsignatures.md monero-project/monero-site#2437

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs/utils: add decoy selection implementation guides and tools #9024

docs/utils: add decoy selection implementation guides and tools #9024

jeffro256 commented Oct 17, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai Dec 13, 2023

hinto-janai left a comment

iamamyth Dec 14, 2023

j-berman left a comment

j-berman commented Dec 19, 2023 •

edited

Loading

jeffro256 commented Dec 26, 2023 via email

iamamyth commented Dec 28, 2023

jeffro256 commented Dec 28, 2023

	spend. We sample these global output indicies according to a certain distribution, with this distribution hopefully
	spend. We sample these global output indices according to a certain distribution, with this distribution hopefully

	statistically matching the distribution of the ages of "true spends", so that the ring member we truely wish to spend is
	statistically matching the distribution of the ages of "true spends", so that the ring member we truly wish to spend is

	picks. You can combat this effect by simply commiting to the order in which you pick the outputs, and try adding them
	picks. You can combat this effect by simply committing to the order in which you pick the outputs, and try adding them

	* Shape paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)
	* Shape parameter for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)

	* Rate paramater for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)
	* Rate parameter for a [gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)

	picked_block_index = bisect.bisect_left(crod, psuedo_global_output_index)
	picked_block_index = bisect.bisect_left(crod, pseudo_global_output_index)

	decoy selection picks (you can specify how many) seperated by newlines. This data can be imported and used to perform
	decoy selection picks (you can specify how many) separated by newlines. This data can be imported and used to perform

	psuedo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock
	pseudo_global_output_index = num_usable_rct_outputs - 1 - target_num_outputs_post_unlock

	description='We provide an easy-to-read non-fingerprinting reference for Monero decoy selecton',
	description='We provide an easy-to-read non-fingerprinting reference for Monero decoy selection',

docs/utils: add decoy selection implementation guides and tools #9024

Are you sure you want to change the base?

docs/utils: add decoy selection implementation guides and tools #9024

Conversation

jeffro256 commented Oct 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hinto-janai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

j-berman left a comment

Choose a reason for hiding this comment

j-berman commented Dec 19, 2023 • edited Loading

jeffro256 commented Dec 26, 2023 via email

iamamyth commented Dec 28, 2023

jeffro256 commented Dec 28, 2023

j-berman commented Dec 19, 2023 •

edited

Loading