Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

facet.pivot 4+ fields pivot can kill cluster [JIRA: RIAK-3051] #708

Open
mogadanez opened this issue Nov 23, 2016 · 0 comments
Open

facet.pivot 4+ fields pivot can kill cluster [JIRA: RIAK-3051] #708

mogadanez opened this issue Nov 23, 2016 · 0 comments

Comments

@mogadanez
Copy link

single node installation ( dev )
docs in index: 2300

query: some-index?q=:&facet=true&rows=0&facet.pivot=field1,field2,field3
queryTime: 8-9sec

query: some-index?q=:&facet=true&rows=0&facet.pivot=field1,field2,field3,filed4

Internal Server Error

The server encountered an error while processing this request:
{error,
    {throw,
        {"Failed to search",
         "http://localhost:8093/internal_solr/some-index/select",
         {error,req_timedout}},
        [{yz_solr,search,3,[{file,"src/yz_solr.erl"},{line,343}]},
         {yz_wm_search,search,3,[{file,"src/yz_wm_search.erl"},{line,134}]},
         {webmachine_resource,resource_call,3,
             [{file,"src/webmachine_resource.erl"},{line,186}]},
         {webmachine_resource,do,3,
             [{file,"src/webmachine_resource.erl"},{line,142}]},
         {webmachine_decision_core,resource_call,1,
             [{file,"src/webmachine_decision_core.erl"},{line,48}]},
         {webmachine_decision_core,decision,1,
             [{file,"src/webmachine_decision_core.erl"},{line,562}]},
         {webmachine_decision_core,handle_request,2,
             [{file,"src/webmachine_decision_core.erl"},{line,33}]},
         {webmachine_mochiweb,loop,2,
             [{file,"src/webmachine_mochiweb.erl"},{line,72}]}]}}

Problem 1.
after that Solr still run something in background with 100% CPU usage

m5

Problem 2.
restart riak not helps, solr still continue running and start attempt failed

2016-11-23 13:07:16.996 [info] <0.19886.0> Administrative stop
2016-11-23 13:07:17.005 [info] <0.603.0>@yz_app:prep_stop:74 Stopping application yokozuna.
2016-11-23 13:07:33.133 [notice] <0.19249.0>@yz_entropy:iterate_entropy_data:43 Can't ping Solr index <<"test-map-index">> to start iterating over entropy data
2016-11-23 13:07:46.600 [error] <0.6519.0>@yz_solrq_helper:send_solr_ops_for_entries:301 Updating a batch of Solr operations failed for index <<"test-index">> with error {error,{othe
r,{error,req_timedout}}}
.....
2016-11-23 13:08:17.214 [notice] <0.14994.0>@yz_exchange_fsm:handle_info:99 YZ Exchange FSM received a DOWN message from a process it was monitoring.  The received message is: {'DOWN',
#Ref<0.0.7.71704>,process,<0.6580.0>,shutdown}
2016-11-23 13:08:22.484 [error] <0.1967.0> Supervisor yz_general_sup had child yz_events started with yz_events:start_link() at <0.6578.0> exit with reason killed in context shutdown_e
rror
2016-11-23 13:08:22.484 [error] <0.171.0> Supervisor riak_core_eventhandler_sup had child {yz_events,yz_events} started with riak_core_eventhandler_guard:start_link(yz_events, yz_event
s, [], undefined) at <0.6593.0> exit with reason killed in context child_terminated
2016-11-23 13:08:22.485 [error] <0.19999.0> CRASH REPORT Process <0.19999.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:init_it/6 line 328
2016-11-23 13:08:22.485 [error] <0.171.0> Supervisor riak_core_eventhandler_sup had child {yz_events,yz_events} started with riak_core_eventhandler_guard:start_link(yz_events, yz_event
s, [], undefined) at <0.6593.0> exit with reason noproc in context start_error
2016-11-23 13:08:22.486 [error] <0.171.0> Supervisor riak_core_eventhandler_sup had child {yz_events,yz_events} started with riak_core_eventhandler_guard:start_link(yz_events, yz_event
s, [], undefined) at {restarting,<0.6593.0>} exit with reason noproc in context start_error
2016-11-23 13:08:22.486 [error] <0.20007.0> CRASH REPORT Process <0.20007.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:init_it/6 line 328
....
2016-11-23 13:08:27.536 [info] <0.603.0>@yz_app:stop:88 Stopped application yokozuna.
2016-11-23 13:08:27.536 [error] <0.1964.0> Supervisor yz_sup had child yz_solr_sup started with yz_solr_sup:start_link() at <0.1965.0> exit with reason killed in context shutdown_error
2016-11-23 13:08:27.543 [info] <0.338.0>@riak_kv_app:prep_stop:228 Stopping application riak_kv - marked service down.
....

2016-11-23 13:08:47.495 [info] <0.6627.0>@yz_solr_proc:init:119 Starting solr: "/usr/bin/java" ["-Djava.awt.headless=true","-Djetty.home=/usr/lib/riak/lib/yokozuna-2.1.7-0-g6cf80ad/pri
2016-11-23 13:08:47.495 [info] <0.208.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_control,member_info_version} = v1
2016-11-23 13:08:47.571 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err: OpenJDK 64-Bit Server VM warning: ignoring option UseStringCache; support was removed in 8.0

2016-11-23 13:08:47.803 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err: Error: Exception thrown by the agent
2016-11-23 13:08:47.803 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err: : java.rmi.server.ExportException: Port already in use: 8985; nested exception is:
        java.net.BindException: Address already in use (Bind failed)
2016-11-23 13:08:47.804 [info] <0.6627.0>@yz_solr_proc:handle_info:184 solr stdout/err:

2016-11-23 13:08:48.122 [warning] <0.6.0> lager_error_logger_h dropped 15 messages in the last second that exceeded the limit of 100 messages/sec
2016-11-23 13:08:48.122 [error] <0.6627.0> gen_server yz_solr_proc terminated with reason: {"solr OS process exited",1}
....
2016-11-23 13:08:48.183 [info] <0.606.0>@yz_app:stop:88 Stopped application yokozuna.
....
2016-11-23 13:08:48.202 [info] <0.340.0>@riak_kv_app:stop:250 Stopped  application riak_kv.
...
2016-11-23 13:08:48.490 [info] <0.166.0>@riak_core_app:stop:116 Stopped  application riak_core.

only way is hard kill yokozuna process

@Basho-JIRA Basho-JIRA changed the title facet.pivot 4+ fields pivot can kill cluster facet.pivot 4+ fields pivot can kill cluster [JIRA: RIAK-3051] Nov 23, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants