SeedNodes report duplicate TradeStatistics2 (same OfferId different Hash)

[bodymindarts]

When requesting data from a seednode the GetDataReponse persistable_network_payload_items array contains many TradeStatistics2 messages that are actually duplicates.
ie. Same offer_id but different hash fields. It appears as though this hashing function is infact non-deterministic and truly yields different results in prod in the wild (unlike the other non-deterministic hashing issue that does not appear to manifest itself).

Here is an example of 2 TradingStatistics2 messages received from the seed nodes:

{
        "direction": "SELL",
        "price": "7199.12320000",
        "amount": "0.03000000",
        "volume": "215.97360000",
        "payment_method": "SEPA",
        "trade_id": "DAtKiw-6951da2b-38e0-44ed-9b17-4935cf85aff3-117",
        "trade_date": "1571920381023",
        "hash": "7318687539069de402ca208ce8c8dfe93aa3326e"
      },
      {
        "direction": "SELL",
        "price": "7199.12320000",
        "amount": "0.03000000",
        "volume": "215.97360000",
        "payment_method": "SEPA",
        "trade_id": "DAtKiw-6951da2b-38e0-44ed-9b17-4935cf85aff3-117",
        "trade_date": "1571920381023",
        "hash": "7a229b08c04ded7f2247feb6b25d35f0335dbef6"
      }

[mrosseel]

great find, is it the hash function itself (would be very suprising) or something in the tojson which is non-deterministic?

[bodymindarts]

Likely its tojsonthat yields different results. I'd be very suprised if ripemd160 were broken.

[chimp1984]

Do you get those differences from the rust project or from pure java code (production code)?

TradeStatisics are published by both traders and as tradeDate is set locally at each user it differs and need to be ignored.

    private final long tradeDate;

Hash takes the json (date should be excluded):

this.hash = Hash.getSha256Ripemd160hash(Utilities.objectToJson(this).getBytes(Charsets.UTF_8));

If the tradeDate is in the json the JsonExclude does not work anymore. Migh be related with the recent changes of the json library.

[chimp1984]

The map has different content:

Oct-25 11:17:26.751 [InputHandler-localhost:7777] ERROR b.c.t.s.TradeStatistics2: offerId=58813797-56327afe-7282-4636-b967-1da57f78e5b9-120, json={
  "direction": "BUY",
  "baseCurrency": "BTC",
  "counterCurrency": "CNY",
  "offerPaymentMethod": "ALI_PAY",
  "offerDate": 1572020232589,
  "offerUseMarketBasedPrice": true,
  "offerMarketPriceMargin": -0.2647,
  "offerAmount": 100000,
  "offerMinAmount": 100000,
  "offerId": "58813797-56327afe-7282-4636-b967-1da57f78e5b9-120",
  "tradePrice": 750026161,
  "tradeAmount": 100000,
  "depositTxId": "fa42884de839d836d9b846decb81bfe057adda048cc706e8ad6965e4482fb58c",
  "extraDataMap": {
    "arbAddr": ":0",
    "medAddr": "localhost:9222"
  }
}, hash=d70a3966a804157380febb086ff98e11b3d400ab 
Oct-25 11:17:26.785 [InputHandler-localhost:9222] ERROR b.c.t.s.TradeStatistics2: offerId=58813797-56327afe-7282-4636-b967-1da57f78e5b9-120, json={
  "direction": "BUY",
  "baseCurrency": "BTC",
  "counterCurrency": "CNY",
  "offerPaymentMethod": "ALI_PAY",
  "offerDate": 1572020232589,
  "offerUseMarketBasedPrice": true,
  "offerMarketPriceMargin": -0.2647,
  "offerAmount": 100000,
  "offerMinAmount": 100000,
  "offerId": "58813797-56327afe-7282-4636-b967-1da57f78e5b9-120",
  "tradePrice": 750026161,
  "tradeAmount": 100000,
  "depositTxId": "fa42884de839d836d9b846decb81bfe057adda048cc706e8ad6965e4482fb58c",
  "extraDataMap": {
    "medAddr": "localhost:9222"
  }
}, hash=4b1471b72923a388b5805242fb452712b141023b 

Seems there is a bug that buyer and seller dont add the same entries (arb). I will check, probably introduced at last release or with current release (tested on 1.2. branch).

[chimp1984]

Yes there was a bug with buyer and seller handling old arbitrator different. I will fix and remove arbitrator entry as it is not used anymore in v.1.2.

[chimp1984]

Thanks for finding the issue.

[chimp1984]

Fixed with #3469