Opencv-librealsense not saving video at 60 fps.

[Toonkirby]

---

Required Info
Camera Model	{ D435 }
Firmware Version	(05.12.07.100)
Operating System & Version	{Win (10)
Platform	Visual Studio 2017
SDK Version	{ 2.16.2.304}
Language	{C++ }
Segment	{others }

Issue Description

Hello, I cannot save video at 60 fps through opencv-libsense with an intel realsense D435. It simply saves the video at twice the speed. I originally tried only using opencv's videocapture method, but now I use the librealsense sdk to pass through opencv, but the videocapture method does not save video at 60 fps. Below is my code. Please ignore the several includes and comments, I have been working on this issue for a few weeks and have tried a variety of different methods:

`// cvtest.cpp : This file contains the 'main' function. Program execution begins and ends there.
//

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/imgproc/imgproc_c.h>
#include "opencv2/videoio.hpp"
#include <opencv2/video.hpp>
#include "opencv2/imgcodecs.hpp"
#include "opencv2/opencv.hpp"
#include
#include <string.h>
#include <limits.h>
#include
#include <opencv2/core/utils/logger.hpp>
#include <librealsense2/rs.hpp>
#include
//#include <opencv2/videoio/registry.hpp>

using namespace cv;
using namespace std;
using namespace utils;
using namespace logging;
using namespace rs2;

LogLevel setLogLevel(LogLevel LOG_LEVEL_SILENT);
//LogLevel(LOG_LEVEL_SILENT);

chrono::high_resolution_clock::time_point start_time = chrono::high_resolution_clock::now();
int main(int argc, char** argv)
{
//rs2::log_to_console(RS2_LOG_SEVERITY_ERROR);

rs2::context ctx;
rs2::device_list devices_list = ctx.query_devices();
rs2::device device;

for (auto&& dev : devices_list)
{
	device = dev;
}
rs2::sensor sensor = device.query_sensors().front();
//auto sensor = prof.get_device().first<device_list>();
//cout << sensor.get_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY) << endl;

//sensor.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0);



config cfg;
cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_RGB8, 60);
rs2::pipeline pipe;


auto prof = pipe.start(cfg);

rs2::pipeline_profile profile = pipe.get_active_profile();
auto colorSensor = profile.get_device().query_sensors()[1];
colorSensor.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0);
cout << colorSensor.get_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY) << endl;
const auto window_name = "Display Image";
namedWindow(window_name, WINDOW_AUTOSIZE);



//VideoCapture cap(1);
VideoWriter video;

std::cout << "test for pre camera load" << endl;

string vidname = "trial.avi";
int codec = VideoWriter::fourcc('M', 'J', 'P', 'G');
//cap.set(CAP_PROP_FOURCC, VideoWriter::fourcc('M', 'J', 'P', 'G'));

std::chrono::high_resolution_clock::time_point current_time;
float timediff;
int64_t time_diff;
video.open(vidname, codec, 60, Size(640, 480));



std::cout << " test for post camera load" << endl;

//make sure it opens, if not exit
/*if (!cap.isOpened()) {
	std::cout << "COULDN'T OPEN CAMERA.\n" << endl;
	return -1;
}

int fps = cap.get(CAP_PROP_FPS);
int frameheight = cap.get(CAP_PROP_FRAME_HEIGHT);
int framewidth = cap.get(CAP_PROP_FRAME_WIDTH);
cap.set(CAP_PROP_FPS, fps);
cap.set(CAP_PROP_FRAME_HEIGHT, frameheight);
cap.set(CAP_PROP_FRAME_WIDTH, framewidth);

video.open(vidname, codec, 60, Size(framewidth, frameheight));
//int framewidth = 640;
//int frameheight = 480;

std::cout << "Width: " << framewidth << endl;
std::cout << "Height: " << frameheight << endl;

//double fps = cap.get(CAP_PROP_FPS);
std::cout << "Frames per second: " << cap.get(CAP_PROP_FPS) << endl; //records at 60 fps

String window_name = "Arthroscopy Video";
Mat frame;
int framecount = 0;
*/
Mat imagetemp2;
while (true) {

	rs2::frameset data = pipe.wait_for_frames();
	rs2::frame color = data.get_color_frame();

	const int w = color.as<rs2::video_frame>().get_width();
	const int h = color.as<rs2::video_frame>().get_height();
	Mat imagetemp(Size(w, h), CV_8UC3, (void*)color.get_data(), Mat::AUTO_STEP);
	cvtColor(imagetemp, imagetemp2, cv::COLOR_BGR2RGB);
	Mat image = imagetemp2.clone();

	//video.write(frame);
	video.write(image);
	//imshow(window_name, frame);
	imshow(window_name, image);
	
	//framecount++;

	//27 is ASCII for esc key

	if (waitKey(1) == 27)
	{
		std::cout << "Esc key was pressed. Video is being stopped." << endl;
		break;
	}
}

//cap.release();
video.release();
return 0;

}
`

Any help would be immensely appreciated. I feel as if I have exhausted all avenues to achieve 60 fps.

[MartyG-RealSense]

Hi @Toonkirby The video speed issue that you have encountered is a widely reported problem with OpenCV video recording with VideoWriter in general, not just with RealSense.

In the link below, a comment at the bottom of the page seems to use a method that is close to your own code to capture 60 FPS. They set VideoWriter.fourcc('M', 'J', 'P', 'G'); like you did in your commented-out line, but followed immediately after that line with a call to fourcc:

capture.set(Videoio.CAP_PROP_FOURCC, fourcc);

https://answers.opencv.org/question/160142/streaming-a-webcam-at-60-fps/

[Toonkirby]

Thanks for the labels! The commented out line was when I was attempting to use solely opencv, in which that line was there when I previously attempted that person's method, which, unfortunately, did not work. The current code uses librealsense with opencv, so I do not have a VideoCapture object. Is there perhaps a better way to save the video other than the VideoWriter and not as a bag file?

[MartyG-RealSense]

You could capture a video from outside of the RealSense SDK by using live screen-capture software and putting the RealSense Viewer into fullscreen mode by toggling the F8 key, as described in #6841 (comment)

If you need to use C++ within the SDK, there was a recent discussion in #9541 about saving to .avi using that language, though it again uses VideoWriter.

Others have made use of gstreamer to obtain a video feed from their RealSense camera. One example of this is in the link below.

https://support.intelrealsense.com/hc/en-us/community/posts/360052840153/comments/360013915474

[MartyG-RealSense]

H @Toonkirby Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.