Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nvidia optical flow #2165

Merged
merged 1 commit into from
Jul 7, 2019
Merged

Nvidia optical flow #2165

merged 1 commit into from
Jul 7, 2019

Conversation

vchiluka5
Copy link
Contributor

@vchiluka5 vchiluka5 commented Jun 25, 2019

Main PR: opencv/opencv#14888

Adds source code for OpenCV-OF interface implementation, unit test and performance test.
This change implements a OpenCV interface for the NVIDIA Optical Flow SDK 1.0. The SDK exposes the latest hardware
capability of Turing GPUs dedicated to computing the relative motion of pixels between images.

The hardware uses sophisticated algorithms to yield highly accurate flow vectors, with robust frame-to-frame intensity variations and
tracks the true object motion faster and more accurately. The optical flow hardware accelerator generates block-based optical flow vectors.
The size of the block depends on hardware in use and can be queried using the function getGridSize(). The block-based flow vectors generated
by the hardware can be converted to dense representation (i.e. per-pixel flow vectors) using upSampler() helper function, if needed.
The flow vectors are stored in CV_16SC2 format with x and y components of each flow vector in 16-bit signed fixed point representation S10.5.
This change also adds a unit test and a performance test for the new interface.

force_builders_only=Custom,docs
buildworker:Custom=linux-1,linux-2,linux-4
docker_image:Custom=ubuntu-cuda:16.04

.gitmodules Outdated Show resolved Hide resolved
@vchiluka5
Copy link
Contributor Author

Is it OK if we discuss all necessary changes and then il make all changes at once?
Il move sample app and merge in commit 1 of this PR once we finalize about how headers can be used

Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for update!

modules/cudaoptflow/CMakeLists.txt Outdated Show resolved Hide resolved
modules/cudaoptflow/include/opencv2/cudaoptflow.hpp Outdated Show resolved Hide resolved
modules/cudaoptflow/src/nvidiaOpticalFlow.cpp Outdated Show resolved Hide resolved
modules/cudaoptflow/src/nvidiaOpticalFlow.cpp Outdated Show resolved Hide resolved
@vchiluka5
Copy link
Contributor Author

Hi @alalek
I see this error in some python bindings. What is this about? Can you please check and also suggest possible solutions to this?

Traceback (most recent call last):
File "/build/precommit-contrib_custom_linux64/opencv/modules/python/bindings/..//src2/gen2.py", line 1102, in
generator.gen(srcfiles, dstdir)
File "/build/precommit-contrib_custom_linux64/opencv/modules/python/bindings/..//src2/gen2.py", line 991, in gen
self.add_func(decl)
File "/build/precommit-contrib_custom_linux64/opencv/modules/python/bindings/..//src2/gen2.py", line 909, in add_func
func_map = self.classes[classname].methods
KeyError: 'cuda_NvidiaHWOpticalFlow'

@alalek
Copy link
Member

alalek commented Jul 3, 2019

All base classes must be wrapped into bindings too:

-class CV_EXPORTS NvidiaHWOpticalFlow 
+class CV_EXPORTS_W NvidiaHWOpticalFlow 

@vchiluka5
Copy link
Contributor Author

@alalek Thanks for reviewing till now,
Updated with all above discussed changes. Please Let me know if anymore changes are needed. Il try to complete all changes asap.

@vchiluka5
Copy link
Contributor Author

Hi @alalek

This error here is due to CUDA_CUDA_LIBRARY not linked to opencv_world module for which i submitted a pull request in opencv main branch
https://pullrequest.opencv.org/buildbot/builders/precommit-contrib_custom_linux64/builds/138/steps/compile%20release/logs/stdio

@alalek
Copy link
Member

alalek commented Jul 4, 2019

This build tests both patches together (both patches are applied).
So something in these patches is wrong and should be get fixed/removed.

@vchiluka5
Copy link
Contributor Author

This build tests both patches together (both patches are applied).
So something in these patches is wrong and should be get fixed/removed.

Looks like its not picking properly from opencv main branch. NVIDIA_Optical_Flow branch is not used here:
https://pullrequest.opencv.org/buildbot/builders/precommit-contrib_custom_linux64/builds/140

@alalek
Copy link
Member

alalek commented Jul 4, 2019

Here:

From https://github.com/vchiluka5/opencv
 * branch            NVIDIA_Optical_Flow -> FETCH_HEAD
Removing samples/gpu/optical_flow.cpp
Merge made by the 'recursive' strategy.
 CMakeLists.txt               |   2 +-
 samples/gpu/optical_flow.cpp | 240 -------------------------------------------
 2 files changed, 1 insertion(+), 241 deletions(-)
 delete mode 100644 samples/gpu/optical_flow.cpp

@vchiluka5
Copy link
Contributor Author

Here:

From https://github.com/vchiluka5/opencv
 * branch            NVIDIA_Optical_Flow -> FETCH_HEAD
Removing samples/gpu/optical_flow.cpp
Merge made by the 'recursive' strategy.
 CMakeLists.txt               |   2 +-
 samples/gpu/optical_flow.cpp | 240 -------------------------------------------
 2 files changed, 1 insertion(+), 241 deletions(-)
 delete mode 100644 samples/gpu/optical_flow.cpp

Hi @alalek
I re-built everything. Steps i did are
git clone -b NVIDIA_Optical_Flow https://github.com/vchiluka5/opencv.git
git clone -b NVIDIA_Optical_Flow https://github.com/vchiluka5/opencv_contrib.git
mkdir build && mkdir install
cd build
cmake -D WITH_CUDA=ON -D BUILD_EXAMPLES=ON -D BUILD_TESTS=ON -D BUILD_PERF_TESTS=ON -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/home/kp/Desktop/opencv_master/install -D OPENCV_EXTRA_MODULES_PATH=/home/kp/Desktop/opencv_master/opencv_contrib/modules /home/kp/Desktop/opencv_master/opencv/
make all -j 40 && make install

and i was able to build opencv without any errors.
Error says warning: libcuda.so.1, needed by ../../lib/libopencv_cudaoptflow.so.4.1.1, not found
and hence unresolved references to driver APIs

Is CUDA installed on that machine where the compilation is going on?

@alalek
Copy link
Member

alalek commented Jul 4, 2019

Build is going in Linux docker container (without any CUDA device, just build, no tests).

Dockerfile with CUDA related installations is here.

@alalek
Copy link
Member

alalek commented Jul 4, 2019

These files are available from container:

# find / -name libcuda*
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudadevrt.a
/usr/local/cuda-8.0/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-8.0/extras/Debugger/include/libcudacore.h
/usr/local/cuda-8.0/extras/Debugger/lib64/libcudacore.a

So only "stubs" are available in build container.

Here:

CUDA_CUDA_LIBRARY=/usr/local/cuda/lib64/stubs/libcuda.so

It seems correct. Need to check what linker does wrong.


Update: Looks like stub file is malformed and can't be used with shared libraries.
It re-links on non-existed libcuda.so.1 (from libcuda.so)

$ readelf -d /usr/local/cuda-8.0/targets/x86_64-linux/lib/stubs/libcuda.so

Dynamic section at offset 0x9418 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libcuda.so.1]
 0x0000000000000010 (SYMBOLIC)           0x0
 0x000000000000000c (INIT)               0x4f78
 0x000000000000000d (FINI)               0x6708
 0x000000006ffffef5 (GNU_HASH)           0x158
 0x0000000000000005 (STRTAB)             0x2eb0
 0x0000000000000006 (SYMTAB)             0xc30
 0x000000000000000a (STRSZ)              7497 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x209600
 0x0000000000000002 (PLTRELSZ)           24 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4f60
 0x0000000000000007 (RELA)               0x4f00
 0x0000000000000008 (RELASZ)             96 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x4ee0
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x4bfa
 0x000000006ffffff9 (RELACOUNT)          1
 0x0000000000000000 (NULL)               0x0

$ readelf -d ./lib/libopencv_cudaoptflow.so

Dynamic section at offset 0x47ecc0 contains 37 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libopencv_cudawarping.so.4.1]
 0x0000000000000001 (NEEDED)             Shared library: [libopencv_cudalegacy.so.4.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libcudart.so.8.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcuda.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libopencv_cudaarithm.so.4.1]
 0x0000000000000001 (NEEDED)             Shared library: [libopencv_imgproc.so.4.1]
 0x0000000000000001 (NEEDED)             Shared library: [libopencv_core.so.4.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libopencv_cudaoptflow.so.4.1]
 0x000000000000000f (RPATH)              Library rpath: [/usr/local/cuda/lib64:/build/precommit-contrib_custom_linux64/build/lib::]
 0x000000000000000c (INIT)               0x7d30
 0x000000000000000d (FINI)               0xa630c
 0x0000000000000019 (INIT_ARRAY)         0x67d600
 0x000000000000001b (INIT_ARRAYSZ)       48 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x67d630
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x1f0
 0x0000000000000005 (STRTAB)             0x1568
 0x0000000000000006 (SYMTAB)             0x368
 0x000000000000000a (STRSZ)              7185 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x67f000
 0x0000000000000002 (PLTRELSZ)           3072 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x7130
 0x0000000000000007 (RELA)               0x3410
 0x0000000000000008 (RELASZ)             15648 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x3300
 0x000000006fffffff (VERNEEDNUM)         5
 0x000000006ffffff0 (VERSYM)             0x317a
 0x000000006ffffff9 (RELACOUNT)          521
 0x0000000000000000 (NULL)               0x0

@alalek
Copy link
Member

alalek commented Jul 4, 2019

Related issue: NVIDIA/nvidia-container-toolkit#185

@vchiluka5
Copy link
Contributor Author

Update: Looks like stub file is malformed and can't be used with shared libraries.
I re-links on non-existed libcuda.so.1 (from libcuda.so)

Does re-linking solved the build issue? I still see failure with same error.

@alalek
Copy link
Member

alalek commented Jul 5, 2019

There is typo in my previous comment (I -> it).

I tried building with extra symbolic link (from here), it seems able to build.

However it is not robust solution in general (due patching content in CUDA SDK directories).


Changes are not added into CI infrastructure yet.

@vchiluka5
Copy link
Contributor Author

vchiluka5 commented Jul 5, 2019

There is typo in my previous comment (I -> it).

I tried building with extra symbolic link (from here), it seems able to build.

However it is not robust solution in general (due patching content in CUDA SDK directories).

Changes are not added into CI infrastructure yet.

What should be our next step then? Since you were able to build that indicates everything is fine in this patch right?
I tried with this docker image : https://gitlab.com/nvidia/cuda/blob/ubuntu16.04/8.0/runtime/Dockerfile
It builds fine.

@param enableCostBuffer Optional Parameter. Flag to enable cost buffer output from calc(). Defaults to false.
@param gpuId Optional parameter to select the GPU ID on which the optical flow should be computed. Useful in multi-GPU systems. Defaults to 0.
*/
CV_WRAP static Ptr<NvidiaOpticalFlow_1_0> create(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets remove CV_WRAP here to workaround bindings issue (look like there is bug in bindings generator).
This would disable Python bindings for this functionality (not critical for now - can be enabled later with corresponding Python tests)

Copy link
Contributor Author

@vchiluka5 vchiluka5 Jul 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I got python bindings error repro locally and i was able to fix it. Hopefully it compiles fine now.
Thanks @alalek for all suggestions.. Appreciated :-)
And your test commit works so will you merge it in opencv main branch and then il update my PR or how should we take this integration?

Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thank you 👍

@opencv-pushbot opencv-pushbot merged commit 5e0783e into opencv:master Jul 7, 2019
@vchiluka5
Copy link
Contributor Author

Looks good to me! Thank you 👍

Thanks alot @alalek

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants