[R-package] crash with direct .Call

[khotilov]

I was trying to add booster attribute setter & getter to the R interface, and have noticed some strange behavior: some .Call statements run fine within the package functions, but crash my R session when called at command line. I've tried in linux and windows with the same results.

It could be illustrated with the following code:

library(xgboost)
data(agaricus.train, package='xgboost')
dtrain <- xgb.DMatrix(agaricus.train$data, label = agaricus.train$label)
watchlist <- list(train = dtrain)
pars=list(eta=0.01)

# this works fine
(b <- xgboost:::xgb.Booster(pars, watchlist))

# go through what xgb.Booster does directly at the command line:
(h <- .Call("XGBoosterCreate_R", watchlist, PACKAGE = "xgboost"))
.Call("XGBoosterSetParam_R", h, names(pars)[1], as.character(pars[[1]]), PACKAGE = "xgboost")

The last command segfaults:

 *** caught segfault ***
address 0x3, cause 'memory not mapped'

When I add some printf's to XGBoosterSetParam_R, I see that it finishes fine.
And the coredump also shows that the segfault happens in R itself:

#0  0x00007fb55500b258 in SET_SYMVALUE () from /usr/lib64/R/lib/libR.so
#1  0x00007fb555006d5f in Rf_ReplIteration () from /usr/lib64/R/lib/libR.so
#2  0x00007fb555007091 in R_ReplConsole () from /usr/lib64/R/lib/libR.so
#3  0x00007fb555007144 in run_Rmainloop () from /usr/lib64/R/lib/libR.so
#4  0x000000000040080b in main ()

I wonder if some of you who have more experience with R's C-interface could shed some light?

[khotilov]

Apparently, proper .Call interface functions need to return a SEXP value.
I'll test the changes during a day and will submit a PR.