How can I build this project on Ubuntu?

[xiaoiker]

No description provided.

[Neo-X]

Please see the README --> https://github.com/xbpeng/DeepTerrainRL#linux

[xiaoiker]

Do you build it successfully? I mean there isn't a CMakelist.txt . And in the #8, it said download external(https://github.com/xbpeng/DeepTerrainRL/releases/download/v1.0/TerrainRL-external-Linux.tar.gz) and extract it into DeepTerrainRL, do I also need do this? Thanks a lot!

[Neo-X]

I build it successfully. You will need to download the external files. Then you will need to use premake4 to generste the make files.

[xiaoiker]

Thanks Neo-X, but I met a new problem while I builded it with make command, it's like this:
external/caffe/build/src/caffe/proto/caffe.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is
^
external/caffe/build/src/caffe/proto/caffe.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
#error incompatible with your Protocol Buffer headers. Please
^
external/caffe/build/src/caffe/proto/caffe.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
#error regenerate this file with a newer version of proton.

it seems the problem of the protobuf version, but I have already install the newest one(2.6) by: sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler

[xiaoiker]

@Neo-X can you please tell me the details about the build process,is it :

1. ubuntu 16, cd(change dir to home)
   ~$ git clone https://github.com/xbpeng/DeepTerrainRL.git
2. downloaded external(https://github.com/xbpeng/DeepTerrainRL/releases/download/v1.0/TerrainRL-external-Linux.tar.gz) and extract it into DeepTerrainRL ( should I change the caffe in it by this one: https://github.com/niuzhiheng/caffe.git @ 7b3e6f2341fe7374243ee0126f5cad1fa1e44e14)
   1. replace ~/DeepTerrainRL/external/caffe/src/caffe/proto/caffe.proto with ~/DeepTerrainRL/caffe_mods/caffe.proto
   2. replace ~/DeepTerrainRL/external/caffe/src/caffe/layers/memory_data_layer.cpp with ~/DeepTerrainRL/caffe_mods/memory_data_layer.cpp
   3. replace ~/DeepTerrainRL/external/caffe/include/caffe/layers/memory_data_layer.hpp with ~/DeepTerrainRL/caffe_mods/memory_data_layer.hpp
   4. ~/DeepTerrainRL$ premake4 gmake
   5. make

It that all right please?

[Neo-X]

Yes that looks correct.
make config=debug64
should work better

You might have to use an older version of protobuf which might have been included in the caffe/src/proto/ folder to use.

[Neo-X]

You should not change the version of caffe.

[xiaoiker]

Thanks Neo-X, for the protobuf issue ,I found I should replace the caffe.pb.h and caffe.pb.cc in the
external/caffe/build/src/caffe/proto. And the two files can be produce by cmd while you get into the external/caffe/:

protoc src/caffe/proto/caffe.proto --cpp_out=.

[xiaoiker]

@Neo-X and then I met this one:
==== Building TerrainRL (debug32) ====
NeuralNet.cpp
In file included from learning/NeuralNet.cpp:2:0:
external/caffe/include/caffe/util/hdf5.hpp:6:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
^
compilation terminated.
TerrainRL.make:314: recipe for target 'obj/x32/Debug/NeuralNet.o' failed
make[1]: *** [obj/x32/Debug/NeuralNet.o] Error 1
Makefile:16: recipe for target 'TerrainRL' failed
make: *** [TerrainRL] Error 2

I change the Makefile.confing file in external/caffe/ by add the inclue directory of/usr/include/hdf5/serial/ , but it still doesn't work.

[xiaoiker]

@xbpeng I found some thing wrong, you include the head files of Eigen ,but you didn't include the path of Eigen in premake4.lua, I don't understand!

[xiaoiker]

@Neo-X @xbpeng how about the 4rdparty file in the external? should I change it back to 3 ?

[mthrok]

@xiaoiker I could build this project and run on clean installation of Ubuntu 14.04

The following is the summary of my bash history.
I did not need to touch Eigen or Caffe.
(I might have built jsoncpp and bullet manually, but I don't remember clearly)
https://gist.github.com/mthrok/db4a02afc36f571768df645d3fb0812c

@Neo-X It seems that learning/DMACETrainer - Copy.h and learning/DMACETrainer - Copy.cpp are not meant to be there. Naming is strange and learning/DMACELearner.h is missing. I had to delete them to build. Can you fix it?

[xbpeng]

thanks for catching that, they have been removed.

[Neo-X]

the issue with hdf5.h is a problem with using Ubuntu 16.04. See the solution here from the caffe issue
BVLC/caffe#2347. I could update the premake file but then it would only work on Ubuntu 16.04+.

[Neo-X]

@xiaoiker The thridparty folder is not used on Linux. Only on Windows.

[xiaoiker]

@Neo-X I have successfully installed caffe on my Ubuntu16.04., it seems that the project can not find the hdf5.h. can you please give me the premake file for Ubuntu16.04

@mthrok Thanks.

[Neo-X]

terrainnRL_premake4.zip

Put the premake4.lua file in the optimization folder. These should work...

[xiaoiker]

Thanks a lot！ I have already successfully built this project on another Ubuntu14.04 machine.

Unfortunately, it still doesn't work on Ubuntu16.04 even with the new premake4 file., it says:

can not find the lcaffe from /usr/bin/ld

It is strange, because I found the libcaffe comes from external/caffe/build/ after build the project on the ubuntu14.04.

[Neo-X]

Caffe should be copied into DeepTerrainRL/lib from external/caffe/build/lib

[xiaoiker]

Thanks @Neo-X , I built this project with the new premake4 file, It could not find the lcaffe from/usr/bin/ld

and then I build the caffe in the external,and the I get libcaffe.so. I make the project again,I get this kind of false no matter if can copy the lib into the DeepTerrainRL/lib directory or not.( Actually I didn't get a folder named lib in DeepTerrainRL, so I made one)

animation@Animation-HS:~/DeepTerrainRL$ make config=debug64 -j8
==== Building TerrainRL (debug64) ====
==== Building TerrainRL_Optimizer (debug64) ====
Linking TerrainRL_Optimizer
Linking TerrainRL
obj/x64/Debug/NNSolver.o: In function std::string* google::MakeCheckOpString<int, float>(int const&, float const&, char const*)': /usr/include/glog/logging.h:672: undefined reference togoogle::base::CheckOpMessageBuilder::NewString()'
obj/x64/Debug/NNSolver.o: In function std::string* google::MakeCheckOpString<float, int>(float const&, int const&, char const*)': /usr/include/glog/logging.h:672: undefined reference togoogle::base::CheckOpMessageBuilder::NewString()'
../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::WireFormatLite::WriteStringMaybeAliased(int, std::string const&, google::protobuf::io::CodedOutputStream*)' ../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::string const&, unsigned char_)'
../external/caffe/build/lib/libcaffe.so: undefined reference to cv::imread(cv::String const&, int)' ../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream_, std::string_)'
../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::Message::GetTypeName() const' ../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::StringTypeHandlerBase::Delete(std::string_)'
../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::string const&))' ../external/caffe/build/lib/libcaffe.so: undefined reference tocv::imencode(cv::String const&, cv::InputArray const&, std::vector<unsigned char, std::allocator >&, std::vector<int, std::allocator > const&)'
../external/caffe/build/lib/libcaffe.so: undefined reference to leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)' ../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::Message::DebugString() const'
../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::WireFormatLite::WriteBytesMaybeAliased(int, std::string const&, google::protobuf::io::CodedOutputStream*)' ../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::StringTypeHandlerBase::New()'
../external/caffe/build/lib/libcaffe.so: undefined reference to cv::imdecode(cv::_InputArray const&, int)' ../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::MessageLite::ParseFromString(std::string const&)'
../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::empty_string_' ../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::NameOfEnum(google::protobuf::EnumDescriptor const*, int)'
../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::WireFormatLite::WriteString(int, std::string const&, google::protobuf::io::CodedOutputStream_)' ../external/caffe/build/lib/libcaffe.so: undefined reference to leveldb::Status::ToString() const'
../external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::DescriptorPool::FindFileByName(std::string const&) const' ../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::WireFormatLite::ReadBytes(google::protobuf::io::CodedInputStream, std::string_)'
../external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::Message::InitializationErrorString() const' collect2: error: ld returned 1 exit status Makefile:208: recipe for target '../TerrainRL_Optimizer' failed make[1]: *_\* [../TerrainRL_Optimizer] Error 1 Makefile:20: recipe for target 'TerrainRL_Optimizer' failed make: **\* [TerrainRL_Optimizer] Error 2 make: **\* Waiting for unfinished jobs.... obj/x64/Debug/NNSolver.o: In function std::string* google::MakeCheckOpString<int, float>(int const&, float const&, char const*)':
/usr/include/glog/logging.h:672: undefined reference togoogle::base::CheckOpMessageBuilder::NewString()' obj/x64/Debug/NNSolver.o: In function std::string* google::MakeCheckOpString<float, int>(float const&, int const&, char const*)':
/usr/include/glog/logging.h:672: undefined reference togoogle::base::CheckOpMessageBuilder::NewString()' external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::Message::InitializationErrorString() const'
external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::WireFormatLite::WriteStringMaybeAliased(int, std::string const&, google::protobuf::io::CodedOutputStream_)' external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::string const&, unsigned char_)'
external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::string*)' external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::Message::GetTypeName() const'
external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::Message::DebugString() const' external/caffe/build/lib/libcaffe.so: undefined reference tocv::imdecode(cv::InputArray const&, int)'
external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::empty_string_' external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::MessageLite::ParseFromString(std::string const&)'
external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::NameOfEnum(google::protobuf::EnumDescriptor const_, int)' external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::DescriptorPool::FindFileByName(std::string const&) const'
external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::WireFormatLite::ReadBytes(google::protobuf::io::CodedInputStream_, std::string_)' external/caffe/build/lib/libcaffe.so: undefined reference to cv::imread(cv::String const&, int)'
external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::StringTypeHandlerBase::Delete(std::string_)' external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::string const&))'
external/caffe/build/lib/libcaffe.so: undefined reference toleveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB*_)' external/caffe/build/lib/libcaffe.so: undefined reference to cv::imencode(cv::String const&, cv::InputArray const&, std::vector<unsigned char, std::allocator >&, std::vector<int, std::allocator > const&)'
external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::StringTypeHandlerBase::New()' external/caffe/build/lib/libcaffe.so: undefined reference to google::protobuf::internal::WireFormatLite::WriteBytesMaybeAliased(int, std::string const&, google::protobuf::io::CodedOutputStream)'
external/caffe/build/lib/libcaffe.so: undefined reference to leveldb::Status::ToString() const' external/caffe/build/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::WireFormatLite::WriteString(int, std::string const&, google::protobuf::io::CodedOutputStream)'
collect2: error: ld returned 1 exit status
TerrainRL.make:231: recipe for target 'TerrainRL' failed
make[1]: *_* [TerrainRL] Error 1
Makefile:16: recipe for target 'TerrainRL' failed
make: *** [TerrainRL] Error 2

[Neo-X]

I seems like you don't have the good protobuf library installed. And maybe the leveldb library as well. maybe you should try a

premake4 clean

and then try and rebuild everything.

[ytian81]

Could you please specify which caffe version should be used for Ubuntu 16.04? Because it seems the version referred in the readme.md doesn't work.

[Neo-X]

The updated one in the external library compressed file with the most recent build is the correct version.

[ytian81]

I have tried the external library, and compile them with make conig=debug64. It complains about caffe proto file. I indeed regenerate the file, but it still doesn't work. The stdout and stderr files are as following.

log.txt
log2.txt

Later on I tried the caffe version you mentioned in the readme file. (https://github.com/niuzhiheng/caffe).
After I regenerate proto file for this caffe, proto problems disappeared, but it will complain about other problem like

learning/NeuralNet.h:5:46: fatal error: caffe/layers/memory_data_layer.hpp: No such file or directory
compilation terminated.

After I remedy these problems, it will complain other syntax problems, most of which should be about caffe problem.

BTW, I am using Ubuntu 16.04. I would appreciate it if you guys could give me more help on compiling.

[Neo-X]

The version from the compressed file is definutely the right one but you will need to recompile caffe after you download it. That version was compiled for Ubuntu 14.04. make sure you make clean everything thoroughly before you build.

[Neo-X]

I updated the build instructions for Linux https://github.com/xbpeng/DeepTerrainRL/blob/master/README.md#linux-build-instructions

[ytian81]

@Neo-X Thanks very much. Those new added Linux instructions really help me a lot.

[Neo-X]

@tianyang9310 Great! let me know how it goes!

[xiaoiker]

I reinstall my system and the CUDA, rebulild the project with the external folder after replacing the premake4 files on my Ubuntu16.04, but I met this problem:

obj/x64/Debug/Character.o: In function cCharacter::ReadState(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': /home/animation/mProject/DeepTerrainRL/optimizer/../anim/Character.cpp:247: undefined reference toJson::Value::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/animation/mProject/DeepTerrainRL/optimizer/../anim/Character.cpp:250: undefined reference to Json::Value::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/animation/mProject/DeepTerrainRL/optimizer/../anim/Character.cpp:254: undefined reference toJson::Value::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
obj/x64/Debug/Character.o:/home/animation/mProject/DeepTerrainRL/optimizer/../anim/Character.cpp:257: more undefined references to `Json::Value::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)' follow
collect2: error: ld returned 1 exit status
Makefile:208: recipe for target '../TerrainRL_Optimizer' failed
make[1]: *** [../TerrainRL_Optimizer] Error 1
Makefile:20: recipe for target 'TerrainRL_Optimizer' failed
make: *** [TerrainRL_Optimizer] Error 2

should I build the Jason in the external files?

[Neo-X]

Looks like you should rebuild the JSON library. From your earlier post you had it seemed like you should clean and rebuild from scratch using the command premake4 clean

[jonygli]

@Neo-X @xbpeng @xiaoiker
on ubuntu 14.04, follow instructions above, failed:

1. git clone https://github.com/xbpeng/DeepTerrainRL.git
2. download https://github.com/xbpeng/DeepTerrainRL/releases/download/v1.0/TerrainRL-external-Linux.tar.gz and extract it to ~/DeepTerrainRL
3. download https://github.com/xbpeng/DeepTerrainRL/files/517743/terrainnRL_premake4.zip and extract it to ~/DeepTerrainRL to overwrite the existing ones

cp ~/DeepTerrainRL/caffe_mods/caffe.proto ~/DeepTerrainRL/external/caffe/src/caffe/proto/caffe.proto
cp ~/DeepTerrainRL/caffe_mods/memory_data_layer.cpp ~/DeepTerrainRL/external/caffe/src/caffe/layers/memory_data_layer.cpp
cp ~/DeepTerrainRL/caffe_mods/memory_data_layer.hpp ~/DeepTerrainRL/external/caffe/include/caffe/layers/memory_data_layer.hpp
5. in ~/DeepTerrainRL, premake4 gmake
6. make
fatal error: bits/c++config.h: No such file or directory

the following is the corresponding commands:

# 1
git clone https://github.com/xbpeng/DeepTerrainRL.git
cd DeepTerrainRL/
# 2
wget https://github.com/xbpeng/DeepTerrainRL/releases/download/v1.0/TerrainRL-external-Linux.tar.gz
tar zxvf TerrainRL-external-Linux.tar.gz 
# 3
wget https://github.com/xbpeng/DeepTerrainRL/files/517743/terrainnRL_premake4.zip 
unzip terrainnRL_premake4.zip 
# 4
cp ~/DeepTerrainRL/caffe_mods/caffe.proto ~/DeepTerrainRL/external/caffe/src/caffe/proto/caffe.proto
cp ~/DeepTerrainRL/caffe_mods/memory_data_layer.cpp ~/DeepTerrainRL/external/caffe/src/caffe/layers/memory_data_layer.cpp
cp ~/DeepTerrainRL/caffe_mods/memory_data_layer.hpp ~/DeepTerrainRL/external/caffe/include/caffe/layers/memory_data_layer.hpp 
# 5
premake4 gmake
# 6
make

anything missing? could we add those missing commands to provide a workable guide please?

[Neo-X]

You need to install step 8 pf the dependencies https://github.com/xbpeng/DeepTerrainRL#dependencies.

[jonygli]

@Neo-X ,thanks for your reply.

1. I installed all those deps mentioned in the https://github.com/xbpeng/DeepTerrainRL#dependencies, sorry that I didn't install those.
2. but, it still reported the same error, so I modified premake4.lua by adding "/usr/include/x86_64-linux-gnu/c++/4.9/" to includedir for bits/c++config.h, and it continued till "/usr/include/c++/4.8/bits/random.h:106:26: error: expected unqualified-id before ‘__int128’", then, I realized I should use debug64, luckily it goes on, and in fact I don't need to add includedir for bits/c++config.h
3. till "render/TextureUtil.h:4:21: fatal error: GL/glew.h: No such file or directory", I installed libglew-dev and it goes on
4. then, it reported can't find -lhdf5_serial and hdf5_serial_hl. I modified premake4.lua again to comment hdf5_serial_hl and hdf5_serial, uncomment hdf5 and hdf5_hl, remeber to modify both premake4.lua and optimizer/premake4.lua
5. sudo apt install f2c
6. everything is fine!

so, besides the 6 steps above, the most import changes:

1. change both premake4.lua to use hdf5 and hdf5_hl
2. sudo apt install f2c libglew-dev
3. make config=debug64(please note to specifiy debug64, which is super important).

[Neo-X]

Sounds good. The last step, using debug64 is important.

[nwcora]

a problem
DMACETrainer - Copy.cpp
../learning/DMACETrainer - Copy.cpp:1:26: fatal error: DMACETrainer.h: No such file or directory
#include "DMACETrainer.h"
^
compilation terminated.
make[1]: *** [obj/x64/Debug/DMACETrainer - Copy.o] Error 1
make: *** [TerrainRL_Optimizer] Error 2
i have rebuild the project ,but error occurs ,in the last step make config=debug64

[xbpeng]

sorry about that, DMACETrainer, should not be included in the build process. It is not used anywhere, and should be removed.