Add support for KV datatypes (counters, sets, maps) #250

seancribbs · 2013-12-11T18:39:04Z

Extracts top-level types into fields as described in yz_dt_extractor.
Defines content types that match those used in riak_kv
Adds fields and dynamic fields to the default schema.

TODO:

riak_test for indexing CRDTs

seancribbs · 2013-12-12T21:55:34Z

@rzezeski I'd love a little help here, for some reason yokozuna with my branch creates an unusable cluster in riak_test. In the original directory, the dev nodes start up just fine. In my rtdev, solr bombs. Here's a console output: https://gist.github.com/7936269

rzezeski · 2013-12-16T17:51:39Z

src/yz_extractor.erl

@@ -32,7 +32,11 @@
                      {"application/json",yz_json_extractor},
                      {"application/xml",yz_xml_extractor},
                      {"text/plain",yz_text_extractor},
-                      {"text/xml",yz_xml_extractor}]).
+                      {"text/xml",yz_xml_extractor},


This list needs to be ordered as the ordsets modules is used to determine extractor.

Addressed in a0fb872

* Extractor registrations have to be sorted. * There's a tiny delay (about 1s) in committing to Solr, so we have to use rt:wait_until/1 for the queries. * URL-escape fields and terms in search queries going over HTTP. * Cleanup whitespace

coderoshi · 2013-12-17T18:20:19Z

src/yz_dt_extractor.erl

+%%     {<<"name_register">>, <<"Ryan Zezeski">>},
+%%     {<<"phones_set">>, <<"555-5555">>},
+%%     {<<"phones_set">>, <<"867-5309">>},
+%%     {<<"page_views_counter">>, <<"1502">>},


I'm curious why *_flag is boolean, but *_counter is binary?

I'm following convention of the JSON extractor. Should it be an integer?

Nevermind, this will work fine. I forgot we converted json numbers to binary too.

lenary · 2013-12-17T20:13:17Z

👍 seems all good to me (though I should really run the tests)

rzezeski · 2013-12-17T20:18:40Z

Yes, please someone make sure they paste the riak_test results output in the PR. All tests but yz_schema_admin should pass.

seancribbs · 2013-12-17T21:13:27Z

@rzezeski Is it that my schema changes broke yz_schema_admin or something else?

rzezeski · 2013-12-17T21:42:52Z

@seancribbs No, caused by upgrade to solr 4.6 (#245).

coderoshi · 2013-12-17T22:02:00Z

src/yz_dt_extractor.erl

+
+-spec field_name(field_path_name(), datatype(), binary()) -> binary().
+field_name(undefined, map, _Sep) ->
+    %% (timeforthat)


Got some time for that in 0a31d51

coderoshi · 2013-12-18T11:09:23Z

+1

seancribbs · 2013-12-18T15:13:20Z

👯 WOO!

Add support for KV datatypes (counters, sets, maps)

rzezeski · 2013-12-18T15:30:25Z

Where's my riak test output?

seancribbs · 2013-12-18T16:14:40Z

@rzezeski Would mine be sufficient? Do you need a full run of verify.sh?

rzezeski · 2013-12-18T16:19:52Z

@seancribbs I just want to make sure someone ran the full suite. If no one did it's not a huge deal as I will run it on develop soon enough. I know I don't always follow this rule myself but moving forward it would be nice if the result output was a prerequisite for a PR being merged.

rzezeski · 2013-12-18T16:55:24Z

Dialyzer had two issues with new extractor. I haven't looked at them closely as I'm trying to do something else ATM.

yz_dt_extractor.erl:93: The variable Err can never match since previous clauses completely covered the type #state{}
yz_dt_extractor.erl:99: The call yz_dt_extractor:extract_map(Name::binary()) breaks the contract (field_path_name()) -> fun(({{binary(),module()},term()},state()) -> state())

seancribbs · 2013-12-18T16:58:02Z

@rzezeski Thanks, will fix.

coderoshi · 2013-12-18T18:02:27Z

@rzezeski Sorry about that. My security failure is unrelated to this

yz_wm_extract_test-bitcask   : pass
yz_stat_test-bitcask         : pass
yz_solr_start_timeout-bitcask: pass
yz_siblings-bitcask          : pass
yz_security-bitcask          : fail
yz_schema_admin-bitcask      : pass
yz_rs_migration_test-bitcask : pass
yz_pb-bitcask                : pass
yz_monitor_solr-bitcask      : pass
yz_mapreduce-bitcask         : pass
yz_languages-bitcask         : pass
yz_index_admin-bitcask       : pass
yz_fallback-bitcask          : pass
yz_errors-bitcask            : pass
yz_dt_test-bitcask           : pass
yokozuna_essential-bitcask   : pass
aae_test-bitcask             : pass
---------------------------------------------
1 Tests Failed
16 Tests Passed

rzezeski · 2013-12-18T18:10:06Z

@coderoshi No problem, thanks for attaching the output.

Sean Cribbs added 2 commits December 10, 2013 22:44

Add extractors for Riak KV datatypes.

3725e7d

Add datatypes fields to the default schema.

43e4a53

seancribbs mentioned this pull request Dec 12, 2013

Additions to Counters for Yokozuna basho/riak_kv#594

Closed

Sean Cribbs added 2 commits December 12, 2013 14:50

Add riak_test for datatype extraction.

f543779

Fix bad format in yz_solr_proc.

bb4c355

rzezeski mentioned this pull request Dec 16, 2013

Fix logging of Solr's stdout/stderr #247

Closed

rzezeski reviewed Dec 16, 2013
View reviewed changes

Fix some bugs.

a0fb872

* Extractor registrations have to be sorted. * There's a tiny delay (about 1s) in committing to Solr, so we have to use rt:wait_until/1 for the queries. * URL-escape fields and terms in search queries going over HTTP. * Cleanup whitespace

ghost assigned coderoshi Dec 17, 2013

coderoshi reviewed Dec 17, 2013
View reviewed changes

Change register fields to stored.

0a31d51

seancribbs pushed a commit that referenced this pull request Dec 18, 2013

Merge pull request #250 from basho/feature/sdc/crdt-extractors

38e37ee

Add support for KV datatypes (counters, sets, maps)

seancribbs merged commit 38e37ee into develop Dec 18, 2013

seancribbs deleted the feature/sdc/crdt-extractors branch December 18, 2013 15:13

seancribbs mentioned this pull request Dec 18, 2013

Fix some dialyzer errors in yz_dt_extractor. #255

Merged

seancribbs unassigned coderoshi Feb 4, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for KV datatypes (counters, sets, maps) #250

Add support for KV datatypes (counters, sets, maps) #250

seancribbs commented Dec 11, 2013

seancribbs commented Dec 12, 2013

rzezeski Dec 16, 2013

seancribbs Dec 16, 2013

coderoshi Dec 17, 2013

seancribbs Dec 17, 2013

coderoshi Dec 17, 2013

lenary commented Dec 17, 2013

rzezeski commented Dec 17, 2013

seancribbs commented Dec 17, 2013

rzezeski commented Dec 17, 2013

coderoshi Dec 17, 2013

seancribbs Dec 18, 2013

coderoshi commented Dec 18, 2013

seancribbs commented Dec 18, 2013

rzezeski commented Dec 18, 2013

seancribbs commented Dec 18, 2013

rzezeski commented Dec 18, 2013

rzezeski commented Dec 18, 2013

seancribbs commented Dec 18, 2013

coderoshi commented Dec 18, 2013

rzezeski commented Dec 18, 2013

Add support for KV datatypes (counters, sets, maps) #250

Add support for KV datatypes (counters, sets, maps) #250

Conversation

seancribbs commented Dec 11, 2013

seancribbs commented Dec 12, 2013

rzezeski Dec 16, 2013

Choose a reason for hiding this comment

seancribbs Dec 16, 2013

Choose a reason for hiding this comment

coderoshi Dec 17, 2013

Choose a reason for hiding this comment

seancribbs Dec 17, 2013

Choose a reason for hiding this comment

coderoshi Dec 17, 2013

Choose a reason for hiding this comment

lenary commented Dec 17, 2013

rzezeski commented Dec 17, 2013

seancribbs commented Dec 17, 2013

rzezeski commented Dec 17, 2013

coderoshi Dec 17, 2013

Choose a reason for hiding this comment

seancribbs Dec 18, 2013

Choose a reason for hiding this comment

coderoshi commented Dec 18, 2013

seancribbs commented Dec 18, 2013

rzezeski commented Dec 18, 2013

seancribbs commented Dec 18, 2013

rzezeski commented Dec 18, 2013

rzezeski commented Dec 18, 2013

seancribbs commented Dec 18, 2013

coderoshi commented Dec 18, 2013

rzezeski commented Dec 18, 2013