Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rhive failing on udf #76

Open
ghost opened this issue Nov 6, 2014 · 7 comments
Open

rhive failing on udf #76

ghost opened this issue Nov 6, 2014 · 7 comments

Comments

@ghost
Copy link

ghost commented Nov 6, 2014

I am running the following code against a table with one column (elements in column are 4 alpha characters long):

myFilter <- function(x){
return(x)
}

rhive.execute(sprintf("CREATE TEMPORARY FUNCTION %s AS "%s"", "R", "com.nexr.rhive.hive.udf.RUDF"))

rhive.query("select R('myfilter',term,0.0) from rhiveterm_bsp")

here is the exception I am seeing in map output:

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"term":"fsrfx"} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at

apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"term":"fsrfx"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.exec.UDFArgumentException: org.rosuda.REngine.Rserve.RserveException: Cannot connect: Connection refused at org.rosuda.REngine.Rserve.RConnection.(RConnection.java:88) at org.rosuda.REngine.Rserve.RConnection.(RConnection.java:60) at org.rosuda.REngine.Rserve.RConnection.(RConnection.java:51) at com.nexr.rhive.hive.udf.RUDF.getConnection(RUDF.java:261) at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:243) at com.nexr.rhive.hive.udf.RUDF.evaluate(RUDF.java:73) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:246) at com.nexr.rhive.hive.udf.RUDF.evaluate(RUDF.java:73) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519) ... 9 more Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: org.rosuda.REngine.Rserve.RserveException: Cannot connect: Connection refused at org.rosuda.REngine.Rserve.RConnection.(RConnection.java:88) at org.rosuda.REngine.Rserve.RConnection.(RConnection.java:60) at org.rosuda.REngine.Rserve.RConnection.(RConnection.java:51) at com.nexr.rhive.hive.udf.RUDF.getConnection(RUDF.java:261) at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:243) at com.nexr.rhive.hive.udf.RUDF.evaluate(RUDF.java:73) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) at com.nexr.rhive.hive.udf.RUDF.getConnection(RUDF.java:265) at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:243) ... 20 more
@ssshow16
Copy link
Contributor

ssshow16 commented Nov 6, 2014

Please, check if Rserve is running at each DataNode.

@ghost
Copy link
Author

ghost commented Nov 6, 2014

thanks - I ran 'R CMD RServe' on each of the data nodes now I get a slightly different error:

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"term":"fsrfx"} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"term":"fsrfx"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127 at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:246) at com.nexr.rhive.hive.udf.RUDF.evaluate(RUDF.java:73) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519) ... 9 more Caused by: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127 at org.rosuda.REngine.Rserve.RConnection.eval(RConnection.java:234) at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:243) ... 20 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143

@ssshow16
Copy link
Contributor

ssshow16 commented Nov 7, 2014

Did you assign and export your function like the following:

myFilter <- function(x){
return(x)
}

rhive.assign("myFilter",myFilter)
rhive.export("myFiler")
rhive.query("select R('myFiler',term,0.0) from rhiveterm_bsp")

And you don't need to call like the following because RHive make a functoin when rhive.connect().
rhive.execute(sprintf("CREATE TEMPORARY FUNCTION %s AS "%s"", "R", "com.nexr.rhive.hive.udf.RUDF"))

Thanks.

@ghost
Copy link
Author

ghost commented Nov 7, 2014

Thanks - I gave it a try:

myFilter <- function(x){

  • return(x)
  • }

rhive.assign("myFilter",myFilter)
[1] TRUE
rhive.export("myFilter")
No encryption was performed by peer.
No encryption was performed by peer.
[1] TRUE
rhive.query("select R('myFilter',term,0.0) from rhiveterm_bps")
Error: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

mapper logs still getting error code: 127

2014-11-07 09:12:35,447 FATAL [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"term":"fsrfx"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127
at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:247)
at com.nexr.rhive.hive.udf.RUDF.evaluate(RUDF.java:73)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more
Caused by: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127
at org.rosuda.REngine.Rserve.RConnection.eval(RConnection.java:234)
at com.nexr.rhive.hive.udf.RUDF.loadRObjects(RUDF.java:243)
... 20 more

@ssshow16
Copy link
Contributor

Please check if there is /rhive/udf/{user}/myFilter.RData in HDFS.

@ghost
Copy link
Author

ghost commented Nov 10, 2014

yes I confirmed that piece early on when I was getting a different exception.

@ssshow16
Copy link
Contributor

RHive try to download myFilter.RData into tmp dir(ex. /tmp/{user}) at each datanode from HDFS.

Please check if there is myFilte.RData in tmp dir and tmp dir have permission to write files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ssshow16