solr indexing desynchronized with riak bucket [JIRA: RIAK-2568]

[mogadanez]

have a buckettype "maps": '{"props":{"datatype":"map"}}'

at some point some record was "cleared": by send operation with removeCounter | removeFlag | removeSet | removeMap | removeRegister for all all present fields

as result querying for /types/maps/buckets/my-bucket/datatypes/key
returns

type: "map",
value: {}

but search via solr still returns this record as in before clean.

what is preffered way to prevent;debug such errors?

[mogadanez]

another issue:
5 nodes riak cluster.
records in solr multiplied in some cases:
have a single record at /types/maps/buckets/my-bucket/datatypes/key
but search in solr randomly returns 1 or 3 records

checked via :8093/internal_solr/my-bucket-index/select?q=id:111 on each node
2 nodes return nothing, wich is ok.

but another 3 nodes returns different results:

node1:

{
updated_timestamp_register: 1467046702201,
...+ 4 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*31*4lpJx40qfaq93QReXlBtMj",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},
{
updated_timestamp_register: 1467046702201,
... + 10 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*31*4sPmYKWxAZsTrKIvCNZ6Uk",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},
{
updated_timestamp_register: 1467710039133,
... + 10 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*31",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},

node2:

{
updated_timestamp_register: 1467046702201,
...+ 4 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*30*4lpJx40qfaq93QReXlBtMj",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},
{
updated_timestamp_register: 1467046702201,
... + 10 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*30*4sPmYKWxAZsTrKIvCNZ6Uk",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},
{
updated_timestamp_register: 1467710039133,
... + 10 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*30",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},

node3:

{
updated_timestamp_register: 1467710039133,
... + 10 fields
_yz_id: "1*maps*my-bucket*ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8*32",
_yz_rk: "ae2cfc7a6f464c5299ef3623d7df6824:9rrp6pbq88umgb8",
_yz_rt: "maps",
_yz_rb: "my-bucket"
},

strange difference in _yz_id field

[fadushin]

Hello @mogadanez,

Can you tell us which version of Riak you are using, and whether your have AAE enabled? We have recently made several fixes to the 2.0 branch (resulting in the 2.0.7 release), in addition to some feature improvements. One of the bug fixes was around AAE and detection of differences in hash trees, which it is possible you are seeing here.

One way to see if you are vulnerable to this bug is to rebuild your YZ AAE trees and see whether these mysterious Solr entries get deleted. Ideally, you should do that on 2.0.7, as there are some subtle bugs in other releases that can cause these differences to go undetected.

We are currently working to merge these fixes forward to a later version of Riak.

[mogadanez]

2.1.4