Separate dependencies for configurable installation #2523

eelstork · 2015-05-28T11:36:12Z

Addresses #1738.
The PR benefited earlier attempts by @cypof (#2402) and @kloudkl (#1074).
Options provided include USE_OPENCV, USE_LMDB, USE_LEVELDB, USE_HDF5, USE_SNAPPY, with both make and CMake flags available.
This allows selecting which IO libraries to build with. Wherever possible functions that would behave incorrectly as a result of disabling a library are excluded before runtime.

Added two items to the test matrix:
- WITH_CUDA=false WITH_CMAKE=false WITH_IO=false
- WITH_CUDA=false WITH_CMAKE=true WITH_IO=false

Without bloating the test matrix, it is essential to have at least one of these, otherwise correct separation of these dependencies would be unmaintainable.

flx42 · 2015-05-28T20:51:00Z

Instead of adding patches fixing your own mistakes, squash multiple patches in one patch and force push to your branch to update the PR.

bhack · 2015-05-29T06:30:45Z

/cc @Nerei

Nerei · 2015-05-29T21:34:09Z

cmake/Summary.cmake

+  if(USE_LMDB)
+    caffe_status("  lmdb              : " LMDB_FOUND THEN "Yes (ver. ${LMDB_VERSION})" ELSE "No")
+  endif()
+  if(USE-SNAPPY)


shelhamer · 2015-05-29T21:39:18Z

@eelstork do keep hacking at this! It will be great to have separate dependencies in line with #1738.

Nerei · 2015-05-29T21:41:22Z

This PR doesn't generate proper CaffeConfig.cmake. All such definitions should be propagated to client code too.

eelstork · 2015-05-30T04:15:08Z

@Nerei thanks for the many helpful suggestions. Working on this.

eelstork · 2015-06-03T05:16:13Z

@Nerei the forward declaration of cv::Mat would not be helpful since cv:Mat will not be available at runtime?

Nerei · 2015-06-03T08:19:57Z

@eelstork Agree. If user try to call opencv dependent method, it will get link errors. Well this is all up to code organization. In general, It's preferable to hide as much as possible USE_* defines into cpp, but this might require more significant caffe code change. That's IMHO.

Regarding commit c5ebe85, I think it should be reverted. Such config should be included into each header (otherwise that defines are always undefined) but it is not generated by Make. I put my comment above and then disproved it.

eelstork · 2015-06-05T09:20:54Z

@Nerei referring c5ebe85, If add_definitions() is not portable this may need fixing later but for now caffe_config.h appears to be somewhat ineffective. I would suggest that we leave things as-is (consistent with handling of 'CPU_ONLY' flag).

eelstork · 2015-06-05T09:23:12Z

@Nerei please have a look at the generated CaffeConfig.cmake; if this is not appropriate, I welcome suggestions to improve it.

eelstork · 2015-06-05T11:20:48Z

Pending further feedback, standing by.

flx42 · 2015-06-05T15:51:01Z

Please squash the patches, it's difficult to review right now.

Nerei · 2015-06-07T09:29:21Z

Fine with CaffeConfig.cmake

bhack · 2015-06-10T07:44:59Z

@eelstork Can you rebase and squash?

eelstork · 2015-06-10T17:02:17Z

@bhack @flx42 squashed #2523

keenbrowne · 2015-06-17T04:59:41Z

@shelhamer @bhack @flx42 @eelstork
What work is left to get this merged? We would like to depend on these changes in our caffe based application.

eelstork · 2015-06-18T03:44:27Z

@shelhamer @keenbrowne ready for review/merge.

sh1r0 · 2015-06-18T06:18:40Z

Makefile.config.example

+USE_HDF5 := 1    # {IO-flag}
+USE_OPENCV := 1  # {IO-flag}
+USE_SNAPPY := 1  # {IO-flag}
+


The trailing comments (# {IO-flag}) might cause problems for value evaluation.

bhack · 2015-06-18T07:37:59Z

@eelstork Have you tested if with this new modularity can build with androd-cmake: https://github.com/taka-no-me/android-cmake/blob/master/README.md?

bhack · 2015-06-18T07:46:57Z

@sh1r0 This add also apk generation capabilties: https://github.com/Discordia/android-cmake

eelstork · 2015-06-18T10:36:53Z

Removed trailing comments in makefile.config. Thanks to @sh1r0 for the suggestion.

eelstork · 2015-06-18T10:40:29Z

@bhack No, I haven't; nor do I know whether it was possible before? Although we hope that removing the dependencies will help porting Caffe to other platforms, I believe that compatibility with a specific platform/config should be addressed separately.

bhack · 2015-06-18T10:47:36Z

@eelstork Yes of course. I've asked only if somebody tested it with this modularization to know now if could be any other modularization that could improve the process with android NDK.

bhack · 2015-09-04T19:31:47Z

@jeffdonahue @shelhamer What do you think of the related #2619?

eelstork · 2015-09-08T07:05:45Z

@shelhamer and @jeffdonahue thanks for these suggestions, I have made changes accordingly (see relevant commits); squash?
Pending review/merge.

shelhamer · 2015-09-08T16:40:56Z

Thanks @eelstork! This looks good to me, so please squash for merge. However 526372a can stay its own commit to explain Travis CI is CPU-only although the message should change to reflect that.

eelstork · 2015-09-09T12:13:32Z

@shelhamer Thanks! Before letting go, I suggest we drop the option to explicitly enable/disable snappy. This is because caffe does not directly depend on Snappy, instead, it is LevelDB that requires it. Where build scripts relied on associate flag, USE_LEVELDB would be used instead. Intending to push a commit to illustrate proposed solution.

eelstork · 2015-09-09T15:40:54Z

@shelhamer @jeffdonahue @bhack @Nerei, opinions about 58f017e ?

shelhamer · 2015-09-09T20:02:29Z

@eelstork 584f017e is good, but please squash with a5bdc06 and add an explanatory commit message that lists the new flags and explain that leveldb + snappy are coupled since snappy is just a leveldb dependency. Thanks.

eelstork · 2015-09-10T05:29:33Z

@shelhamer - updated; squashed.

eelstork · 2015-09-16T08:48:57Z

@shelhamer Anything missing to get this merged?

jeffdonahue · 2015-09-16T18:17:39Z

@eelstork I took another look -- sorry for the last minute comments. Otherwise looks good for merge to me.

OpenCV, LMDB, LevelDB and Snappy are made optional via switches (USE_OPENCV, USE_LMDB, USE_LEVELDB) available for Make and CMake builds. Since Snappy is a LevelDB dependency, its use is determined by USE_LEVELDB. HDF5 is left bundled because it is used for serializing weights and solverstates.

eelstork · 2015-09-17T07:30:56Z

@jeffdonahue updated. I would suggest leaving 2349c6d as a separate commit.

jeffdonahue · 2015-09-17T07:33:09Z

Thanks @eelstork!

Separate dependencies for configurable installation

bhack · 2015-09-17T07:35:25Z

@jeffdonahue Can you take a look at related #2619?

shelhamer · 2015-09-17T21:00:07Z

@mtngld thanks for pointing this out.

@jeffdonahue thoughts on making all IO on by default for backwards compatibility? The Makefile.config / CMake can then explicitly disable dependencies as desired. That seems reasonable to me.

jeffdonahue · 2015-09-17T21:18:47Z

Oops, I didn't realize this changed the default to no IO dependencies. Yes, I agree the default should be to have them all enabled as before.

d4nst · 2015-10-30T15:47:47Z

Currently it is not possible to disable leveldb and lmdb (at least on Windows) which would be useful to use Caffe for deployment only.

eelstork · 2015-10-31T05:43:12Z

@danst18 I believe that disabling LevelDB and LMDB is possible on Windows (via CMake), granted that building caffe on Windows does require tweaks. What motivates your comment?

d4nst · 2015-10-31T11:06:51Z

@eelstork I mean when both of them are disabled at the same time. In db.cpp I get an error saying that GetDB should return a value. If I remove db.h and db.cpp then I have to add guards to other parts of the code where they are referenced, for example in data_reader.cpp

eelstork · 2015-11-08T10:45:08Z

@danst18 sorry for the late reply. I submitted a fix for this. Thank you for reporting the issue.

shelhamer added compatibility build labels May 28, 2015

Nerei reviewed May 29, 2015
View reviewed changes

cmake/Summary.cmake

if(USE_LMDB)

caffe_status(" lmdb : " LMDB_FOUND THEN "Yes (ver. ${LMDB_VERSION})" ELSE "No")

endif()

if(USE-SNAPPY)

Copy link

Nerei May 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

bhack mentioned this pull request Jun 2, 2015

First attemp to remove boost dependecies. #2537

Closed

This was referenced Jun 12, 2015

Can you rebase the caffe submodule? sh1r0/caffe-android-lib#11

Closed

Provide a Caffe package in Debian #2601

Closed

sh1r0 reviewed Jun 18, 2015
View reviewed changes

eelstork added 3 commits September 17, 2015 15:08

Fix case in CMake notices

2349c6d

Add a comment indicating that Travis CI tests are CPU only

68c9e2b

jeffdonahue added a commit that referenced this pull request Sep 17, 2015

Merge pull request #2523 from BonsaiAI/separate-io-dependencies

5ff3734

Separate dependencies for configurable installation

jeffdonahue merged commit 5ff3734 into BVLC:master Sep 17, 2015

This was referenced Sep 17, 2015

include IO dependencies by default #3081

Merged

Add Optional AudioDataLayer and LibSndFile Dependancy #3065

Closed

lukeyeager mentioned this pull request Sep 25, 2015

Open LMDB files with MDB_NOLOCK if no write access #3088

Merged

shelhamer mentioned this pull request Sep 26, 2015

Make IO dependencies optional (leveldb lmdb snappy hdf5 opencv) #2402

Closed

lukeyeager mentioned this pull request Nov 10, 2015

Adding out of memory handler, making arena bins more detailed NVIDIA/caffe#73

Merged

eelstork deleted the separate-io-dependencies branch November 18, 2015 08:21

shelhamer mentioned this pull request Feb 26, 2016

Separate dependencies for configurable installation #1738

Closed

Separate dependencies for configurable installation #2523

Separate dependencies for configurable installation #2523

Conversation

eelstork commented May 28, 2015

flx42 commented May 28, 2015

bhack commented May 29, 2015

Nerei May 29, 2015

Choose a reason for hiding this comment

shelhamer commented May 29, 2015

Nerei commented May 29, 2015

eelstork commented May 30, 2015

eelstork commented Jun 3, 2015

Nerei commented Jun 3, 2015

eelstork commented Jun 5, 2015

eelstork commented Jun 5, 2015

eelstork commented Jun 5, 2015

flx42 commented Jun 5, 2015

Nerei commented Jun 7, 2015

bhack commented Jun 10, 2015

eelstork commented Jun 10, 2015

keenbrowne commented Jun 17, 2015

eelstork commented Jun 18, 2015

sh1r0 Jun 18, 2015

Choose a reason for hiding this comment

bhack commented Jun 18, 2015

bhack commented Jun 18, 2015

eelstork commented Jun 18, 2015

eelstork commented Jun 18, 2015

bhack commented Jun 18, 2015

bhack commented Sep 4, 2015

eelstork commented Sep 8, 2015

shelhamer commented Sep 8, 2015

eelstork commented Sep 9, 2015

eelstork commented Sep 9, 2015

shelhamer commented Sep 9, 2015

eelstork commented Sep 10, 2015

eelstork commented Sep 16, 2015

jeffdonahue commented Sep 16, 2015

eelstork commented Sep 17, 2015

jeffdonahue commented Sep 17, 2015

bhack commented Sep 17, 2015

shelhamer commented Sep 17, 2015

jeffdonahue commented Sep 17, 2015

d4nst commented Oct 30, 2015

eelstork commented Oct 31, 2015

d4nst commented Oct 31, 2015

eelstork commented Nov 8, 2015