error when installing development version

[GeraldCNelson]

With this command in rstudio on a mac.

install.packages("data.table", type = "source", repos = "https://Rdatatable.github.io/data.table")
I get the following error

fwrite.c:8:10: fatal error: 'omp.h' file not found

include <omp.h>

[jangorecki]

@GeraldCNelson Are you able to reproduce it in R? what is your sesionInfo()?

[GeraldCNelson]

Running from R in the shell that Rstudio opens gives the same error.

Session info for Rstudio
R version 3.2.4 (2016-03-10)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.4 (El Capitan)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] tools_3.2.4 Rcpp_0.12.4 openxlsx_3.1.16

[arunsrinivasan]

Will fix. It needs a #ifdef for compilers that either aren't openmp aware or doesn't have it enabled by default.

Since you're on a mac, you can install clang-omp using homebrew:

brew install clang-omp

Then using your favorite text editor, add the lines:

CC=clang-omp -fopenmp
CXX=clang-omp++ -fopenmp
# on mine, I also have:
# CFLAGS=-g -O3 -Wall -pedantic -mtune=native  -std=gnu99
# CXXFLAGS=-g -O3 -Wall -pedantic -mtune=native -std=c++11

to ~/.R/Makevars. If you have gcc, you can substitute clang-omp with gcc and clang-omp++ with g++. If you don't want to go through this, then just wait until I push a fix (not likely today) and you can compile data.table with your current version of llvm-clang, but won't have the parallel stuff disabled.

[GeraldCNelson]

At some point I must have installed homebrew. Running
brew install clang-omp
I got a warning about not providing support for a pre-release version but clang-omp installed in 18.9 minutes.
My Makevars file now looks like

#CC=clang
#CXX=clang++
CC=clang-omp -fopenmp
CXX=clang-omp++ -fopenmp
# on mine, I also have:
CFLAGS=-g -O3 -Wall -pedantic -mtune=native  -std=gnu99
CXXFLAGS=-g -O3 -Wall -pedantic -mtune=native -std=c++11

But I'm still getting an error. Here's what it looks like

In file included from assign.c:1:
In file included from ./data.table.h:1:
/Library/Frameworks/R.framework/Resources/include/R.h:28:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>
^
1 error generated.
make: *** [assign.o] Error 1

[arunsrinivasan]

That's strange.. stdlib.h not found... ?!?

[arunsrinivasan]

Could you paste your command and the entire output here please?

[GeraldCNelson]

Here is the command and output from the Rstudio console

>  install.packages("data.table", type = "source", repos = "https://Rdatatable.github.io/data.table") 
> Error in install.packages : Updating loaded packages
> > install.packages("data.table", type = "source", repos = "https://Rdatatable.github.io/data.table")
> Installing package into ‘/Users/gcn/Library/R/3.2/library’
> (as ‘lib’ is unspecified)
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0 2650k    0 15782    0     0  15539      0  0:02:54  0:00:01  0:02:53 15564100 2650k  100 2650k    0     0  1485k      0  0:00:01  0:00:01 --:--:-- 1486k
> * installing *source* package ‘data.table’ ...
> ** libs
> clang-omp -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG  -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include    -fPIC  -g -O3 -Wall -pedantic -mtune=native  -std=gnu99 -c assign.c -o assign.o
> In file included from assign.c:1:
> In file included from ./data.table.h:1:
> /Library/Frameworks/R.framework/Resources/include/R.h:28:10: fatal error: 'stdlib.h' file not found
> #include <stdlib.h>
>          ^
> 1 error generated.
> make: *** [assign.o] Error 1
> ERROR: compilation failed for package ‘data.table’
> * removing ‘/Users/gcn/Library/R/3.2/library/data.table’
> * restoring previous ‘/Users/gcn/Library/R/3.2/library/data.table’
> Warning in install.packages :
>   installation of package ‘data.table’ had non-zero exit status
> 
> The downloaded source packages are in
>   ‘/private/var/folders/bv/881_rsvd2kv83c13yywwc_xh0000gn/T/RtmpfgeyUh/downloaded_packages’
> > 

[GeraldCNelson]

Re the 'missing' stdlib.h file, here's my path

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/CrossPack-AVR/bin:/usr/local/git/bin:/Library/TeX/texbin

I do have a stdlib.h file on my computer. It's at /usr/local/include/c++/4.9.0/tr1.

Should this be in my path?

[arunsrinivasan]

Do you've Xcode development tools installed? Please install the latest Xcode along with the development tools. Then update homebrew and reinstall clang-omp. That should work I think.. but it's time consuming, sorry about that..

[mattdowle]

I can add the #ifdef _OPENMP and a warning message on startup. Actually, maybe a warning message on running fwrite() since the startup message might be missed/suppressed by user.

Aside, 3 related answers. Don't know if useful.
http://stackoverflow.com/a/32982307/403310
http://stackoverflow.com/a/33223066/403310
http://stackoverflow.com/a/36273914/403310

[arunsrinivasan]

Okay, I'll leave you to it then. If you add the ifdef _OPENMP around all pragmas as well, then it'd compile without openmp when it's not enabled, then there's no need for any warning..?

I'm testing some things on parallelising .CsubsetDT, and I've separate code for with/without OpenMP and seems to work fine. In any case, we'd have to use it if we use constructs/directives from newer versions of OpenMP, I guess..

[mattdowle]

Done. Not around all #pragma: http://bisqwit.iki.fi/story/howto/openmp/#Discussion. We want users to be warned they have an openmp lacking compiler/environment (which seems just OSX clang). Or rather, it should be up to user whether they turn off those warnings by turning off -Wunknown-pragmas
http://stackoverflow.com/questions/23108307/unknown-pragma-when-compiling-openmp-with-g

Either -Wno-unknown-pragmas or #pragma warning (disable : 4068 ) :
http://stackoverflow.com/a/133521/403310

[GeraldCNelson]

Yes, I do have Xcode development tools installed.

Actually, no. I have Xcode installed but not the development tools. I'll do that and see what I progress I can make.

Gerald C. Nelson

Professor Emeritus, UIUC

+1 217-390-7888 (cell)

+1 970-639-2079 (land line)

http://bit.ly/1arho7d

From: Arun [email protected]
Reply-To: "Rdatatable/data.table" [email protected]
Date: Monday, April 18, 2016 at 3:04 PM
To: "Rdatatable/data.table" [email protected]
Cc: Gerald Nelson [email protected]
Subject: Re: [Rdatatable/data.table] error when installing development version (#1652)

Do you've Xcode development tools installed?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[GeraldCNelson]

Updating home-brew seems to have done the trick. At least the dev version of data.table now installs. Thanks for your help!

[arunsrinivasan]

👍

[markdanese]

That seems to have worked for me. I finally found makevars (I was thinking it was a directory and not a text file). Mine looks like this now. I can't recall why I have the FLIBS in there.
It seems to load without a problem now.

#CC=clang
#CXX=clang++
CC=clang-omp -fopenmp
CXX=clang-omp++ -fopenmp
FLIBS=-L/usr/local/Cellar/gcc/4.9.1/lib/gcc/x86_64-apple-darwin13.2.0/4.9.1/

[markdanese]

I will say that I had to run it 3 times because I could not believe the write times (13 sec for a 3.1 GB csv file). For a test script, see this link: https://gist.github.com/markdanese/28b9f5412df55efceba754fee2363444

[kevinushey]

It looks like an inclusion of <omp.h> still needs to be guarded here as well:

data.table/src/reorder.c

Line 3 in 5c5067d

#include <omp.h>

(I get a failure when attempting to install on OS X using Apple clang, which does not yet have OpenMP)

---

clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG  -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include    -fPIC  -g -O2 -Wall -march=native -c shift.c -o shift.o
reorder.c:3:10: fatal error: 'omp.h' file not found
#include <omp.h>

[arunsrinivasan]

Yes, I'm aware of it. Will fix.

[pseemakurthi]

Unable to install R studio On Microsoft Open R, Windows Platform using R studio version 0.99.896.
I installed devtools using below command
install.packages("devtools")

CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to Windows directory.

• installing source package 'data.table' ...
  ** libs
  Warning: running command 'make -f "Makevars" -f "C:/PROGRA1/MIE74D1/MRO/R-321.4/etc/x64/Makeconf" -f "C:/PROGRA1/MIE74D1/MRO/R-321.4/share/make/winshlib.mk" SHLIB="data.table.dll" WIN=64 TCLBIN=64 OBJECTS="assign.o bmerge.o chmatch.o dogroups.o fastmean.o fcast.o fmelt.o forder.o frank.o fread.o fwrite.o gsumm.o ijoin.o init.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o subset.o transpose.o uniqlist.o vecseq.o wrappers.o"' had status 127
  Warning messages:
  1: running command '"C:/PROGRA1/MIE74D1/MRO/R-321.4/bin/x64/R" CMD INSTALL -l "\edftradingna.com\dfs\users\pseemakurthi\R\win-library\3.2" C:\Users\PSEEMA1\AppData\Local\Temp\Rtmpm6YUyN/downloaded_packages/data.table_1.9.7.tar.gz' had status 1
  2: In install.packages("data.table", type = "source", repos = "https://Rdatatable.github.io/data.table") :
  installation of package ‘data.table’ had non-zero exit status

[kevinushey]

Usually a '127' error means you don't have R build tools set up properly (I also think you might have trouble since you're using UNC paths)

[arturlascala]

Hello! I get the following error, even though it seems that I've tried all the aforementioned steps. Here's my output. I'm running on an OS X 10.11.14 El Capitain. Can somebody shed some light on this issue?

`> install.packages("data.table", type = "source",

•              repos = "http://Rdatatable.github.io/data.table")
  

  tentando a URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.9.7.tar.gz'

  Content type 'application/octet-stream' length 2737015 bytes (2.6 MB)

  downloaded 2.6 MB

• installing source package ‘data.table’ ...
  ** libs
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c assign.c -o assign.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c bmerge.c -o bmerge.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c chmatch.c -o chmatch.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c dogroups.c -o dogroups.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c fastmean.c -o fastmean.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c fcast.c -o fcast.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c fmelt.c -o fmelt.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c forder.c -o forder.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c frank.c -o frank.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c fread.c -o fread.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c fsort.c -o fsort.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c fwrite.c -o fwrite.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c gsumm.c -o gsumm.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c ijoin.c -o ijoin.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c init.c -o init.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c openmp-utils.c -o openmp-utils.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c quickselect.c -o quickselect.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c rbindlist.c -o rbindlist.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c reorder.c -o reorder.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c shift.c -o shift.o
  clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -fPIC -Wall -mtune=core2 -g -O2 -c subset.c -o subset.o
  subset.c:94:18: error: array subscript is not an integer
  REAL(ans)[ans++] = (this==NA_INTEGER || this>max) ? naval.d : REAL(x)[this-1];
  ^~~~~~
  1 error generated.
  make: *** [subset.o] Error 1
  ERROR: compilation failed for package ‘data.table’

• removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/data.table’
  Warning in install.packages :
  installation of package ‘data.table’ had non-zero exit status

The downloaded source packages are in
‘/private/var/folders/8_/6csz8h8542b366zfbdvgrysr0000gn/T/Rtmp6iduA1/downloaded_packages’`

[arunsrinivasan]

@arturlascala thanks. It's an oversight on my part. It should be ansi++. Will fix.

[arunsrinivasan]

Fixed in commit d0e63f8

[arturlascala]

Thanks, @arunsrinivasan ! Works fine now!

[smcinerney]

Update: You no longer need to install separate clang-omp; in all recent MacOS versions, clang supports -fopenmp-targets since LLVM 3.8. See this thread or SO: Get Apple clang version and corresponding upstream LLVM version