Colored Kinect Fusion #2878

DumDereDum · 2021-02-24T16:42:33Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

DumDereDum · 2021-03-17T06:49:45Z

modules/rgbd/include/opencv2/rgbd/colored_kinfu.hpp

+    /** @brief camera intrinsics */
+    CV_PROP_RW Matx33f intr;
+    CV_PROP_RW Matx33f rgb_intr;
+


add info about values

DumDereDum · 2021-03-17T06:53:54Z

modules/rgbd/include/opencv2/rgbd/kinfu.hpp

@@ -81,7 +86,7 @@ struct CV_EXPORTS_W Params

    /** @brief camera intrinsics */
    CV_PROP_RW Matx33f intr;
-
+    CV_PROP_RW Matx33f rgb_intr;


add info about value

DumDereDum · 2021-03-17T06:54:23Z

modules/rgbd/include/opencv2/rgbd/large_kinfu.hpp

@@ -40,7 +40,7 @@ struct CV_EXPORTS_W Params

    /** @brief camera intrinsics */
    CV_PROP_RW Matx33f intr;
-
+    CV_PROP_RW Matx33f rgb_intr;


add info about value

DumDereDum · 2021-03-17T07:08:42Z

modules/rgbd/samples/io_utils.hpp

@@ -87,6 +88,15 @@ static const float cy    = 204.f;
 static const float k1    = 0.12f;


rename like depth_focal and etc

DumDereDum · 2021-03-17T07:11:28Z

modules/rgbd/samples/io_utils.hpp

+        dir = fileList.substr(0, slashIdx);
+
+        if (!file.is_open())
+            throw std::runtime_error("Failed to write depth list");


rgb list, not depth list

DumDereDum · 2021-03-17T07:12:56Z

modules/rgbd/samples/io_utils.hpp

+        std::string depthFname = cv::format("%04d.png", count);
+        std::string fullDepthFname = dir + '/' + depthFname;
+        if (!imwrite(fullDepthFname, depth))
+            throw std::runtime_error("Failed to write depth to file " + fullDepthFname);


DumDereDum · 2021-03-17T07:30:55Z

modules/rgbd/src/colored_tsdf.cpp

+    virtual void integrate(InputArray, float, const Matx44f&, const kinfu::Intr&, const int) override
+        { CV_Error(Error::StsNotImplemented, "Not implemented"); };
+    virtual void integrate(InputArray _depth, InputArray _rgb, float depthFactor, const Matx44f& cameraPose,
+        const kinfu::Intr& intrinsics, const Intr& rgb_intrinsics, const int frameId = 0) override;


rename intrinsics as depth_intrinsics

DumDereDum · 2021-03-17T07:31:42Z

modules/rgbd/src/colored_tsdf.cpp

+
+// use depth instead of distance (optimization)
+void ColoredTSDFVolumeCPU::integrate(InputArray _depth, InputArray _rgb, float depthFactor, const Matx44f& cameraPose,
+                              const Intr& intrinsics, const Intr& rgb_intrinsics, const int frameId)


rename intrinsics as depth_intrinsics

DumDereDum · 2021-03-17T07:39:43Z

modules/rgbd/src/colored_tsdf.cpp

+                        {
+                            v_float32x4 pv = (orig + dir * v_setall_f32(ts));
+                            v_float32x4 nv = volume.getNormalVoxel(pv);
+                            v_float32x4 cl = volume.getColorVoxel(pv);


rename cl as cv

…o colored_kinfu

savuor · 2021-04-03T00:26:39Z

modules/rgbd/src/tsdf_functions.cpp

+                    int _u = (int)projected.get0();
+                    int _v = (int)v_rotate_right<1>(projected).get0();
+                    int rgb_u = (int)projected.get0();
+                    int rgb_v = (int)v_rotate_right<1>(projected).get0();


Vectorized code diverges with scalar one in what coordinate to use for RGB

savuor · 2021-04-03T00:41:22Z

modules/rgbd/src/tsdf_functions.cpp

+                        ColorType& b = voxel.b;
+
+                        // update RGB
+                        if (abs(((float)(r + g + b)) - (colorRGB[0] + colorRGB[1] + colorRGB[2])) < 1000 || weight < 1)


What's the purpose of this extra comparison and why 1000?

it is the previous version of code
will be removed

savuor · 2021-04-07T17:19:42Z

While there are things to do (speed up using OpenCL, rewrite raycasting to get better color filtering, fix demo to allow unregistered depth-rgb pairs) they have more sense to be implemented in upcoming PRs than in this one.
Here we have demo working and tests passing which is going to be enough for now.

savuor · 2021-04-07T17:19:51Z

👍

arsaratovtsev added 30 commits February 24, 2021 19:37

create dummy unworked classes

5ed1b73

dummy class works

14df9f4

colored kinfu demo minor changes

4abffa4

add simple rgb reader and writer

5346fb6

add rgb info to tsdf

9eab94e

add new raycast

b2d2e68

replace TsdfVoxel with RGBTsdfVoxel; integrate done

60347a7

add colors info to raycast

db691f5

just render colors

7744de2

color processing fix

b9f4f85

intergrate color fix

a800eb8

add simple shift; minor fixes

3a4934d

add cv_errors for not implemented functions

aea0de9

minor fixes

e04a6ef

made calibration

e7b70d5

makeColoredFrameFromDepth works

a1290c7

remove comments

0479f0e

makeColoredFrameFromDepth fix

bd6bb45

RGBSoure remove extra code

748d8c3

RGBSoure remove extra code 1

a47372b

simple fix of bug with rgb size

a2c223d

minor fix

7e55474

docs fix

baaeb77

docs fix

07edbc5

unused parameter fix

bcaefe5

warnings fix

68e9c42

fix tsdf errors; rgbd_perf warnings

4997b87

fix errors

71056f2

minor fix

d2832fa

debug print

c8f9659

arsaratovtsev added 7 commits March 12, 2021 14:54

docs fix

e74a154

warnings fix

67658db

minor fix

cf20c8e

function warning fix

1cf0643

add vectorized code

5eeeef2

minor fixes

59e60c2

minor speed up

21666b7

DumDereDum commented Mar 17, 2021

View reviewed changes

arsaratovtsev and others added 6 commits March 17, 2021 11:56

minor fixes; renaming

53eeb47

new minor fixes

ecd8bd5

Merge branch 'master' into colored_kinfu

2988841

bug fix

8b24c41

Merge branch 'master' of https://github.com/opencv/opencv_contrib int…

7eb69d5

…o colored_kinfu

minor fix

20443bd

savuor reviewed Apr 3, 2021

View reviewed changes

arsaratovtsev added 8 commits April 5, 2021 10:25

test update

5a2f485

extra code removed

d1e5f3b

bugfix; warning fix

32cae64

simple interpolation

0fadae2

minor fix

9b030ea

color work minor fix

bc4be41

vectorized interpolation

ba9e2fd

minor fix

8f40b48

savuor self-assigned this Apr 7, 2021

alalek merged commit da6a95e into opencv:master Apr 7, 2021

alalek mentioned this pull request Apr 9, 2021

(5.x) Merge 4.x #2919

Merged

DumDereDum deleted the colored_kinfu branch April 19, 2021 07:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Colored Kinect Fusion #2878

Colored Kinect Fusion #2878

DumDereDum commented Feb 24, 2021 •

edited

Loading

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

DumDereDum Mar 17, 2021

savuor Apr 3, 2021

savuor Apr 3, 2021

DumDereDum Apr 5, 2021

savuor commented Apr 7, 2021

savuor commented Apr 7, 2021

		@@ -87,6 +88,15 @@ static const float cy = 204.f;
		static const float k1 = 0.12f;

Colored Kinect Fusion #2878

Colored Kinect Fusion #2878

Conversation

DumDereDum commented Feb 24, 2021 • edited Loading

Pull Request Readiness Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

savuor commented Apr 7, 2021

savuor commented Apr 7, 2021

DumDereDum commented Feb 24, 2021 •

edited

Loading