-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Depth image is green and wrong resolution #11
Comments
One thing that I noticed is that the first 15 frames are green while the camera and sensors initialize . If you are saving the first frame that is incoming I would suggest you skip a few and then capture and see if rgb is still green |
VLC only understands how to render common formats. Since the camera incorrectly identifies the first depth format as a 16 bit color format it converts that format to RGB, and you get an ugly green video. Workarounds include patching VLC to treat the realsense formats as grey, or changing the kernel patch to identify one of it's available formats as grey. I haven't looked into how to do the first, but the second I did during development of the patch. I didn't keep it because linux can only select one combination of bit depth and grey per camera, but the camera outputs two 8bit, and two 16bit formats so only one would be usable. I gave them special names so that they would both be accessible. You probably only need one of them so it's probably reasonable to use the kernel workaround. So for example you could change this.
to
to get a depth format to identify as 16bit grey Similarly you could change a infrared format to 8bit grey
to
This doesn't change anything about the raw data it just gives VLC a clue about how to render it. As far as resolution the depth camera has a maximum resolution of 640x480. This is better than the Kinect2, and I think the best currently on the market in a consumer device. The color camera is basically an off the shelf consumer color camera and has a bunch of selectable modes. 1080p cameras are cheap so that is what they gave you. You can reduce the color resolution to match if you wish. It's important to note though that I think the only thing special about it is that they made sure it has time sync with the depth camera and optimized optics choices for computer vision applications. It is on the same device, but is not at exactly the same perspective, and a pixel position on the depth camera probably won't align with a pixel at the same position on the color camera. Finding the color of a given pixel from the depth camera in order to create a colorized point cloud requires a computer vision technique called "registration". I haven't created any tools to do this yet. |
Hi teknotus, I encounter some issue as blew
Thank you so much your support |
My patch is for the F200. I found a patch that looks almost like it for the R200 in code released directly from Intel. Try working with that instead.
From a USB header someone sent me it looks like the three video feeds are "Left / Right", Depth, and RGB. I've sometimes had issues with the camera stopping working from some comments in the source where I found that patch it seems to be an issue with an ambiguity in the USB video standard, and they went with a solution that worked on Windows, but not Linux. They'll probably fix it in a firmware update. I started to write a tool to do a camera reset via software, but haven't finished it. The reset is easy it's reliably finding the device file for the USB port the camera is connected to that is a bit tricky. In the rstools branch is code for finding cameras. It has most of what is needed for finding the USB port to reset. Also another example of producing a live video feed. https://github.com/teknotus/depthview/tree/rstools The getControl is for reading the current value of various controls on the camera. It's for setting the slider positions in the app. I don't know the controls for the R200 yet so I couldn't tell you how to use them anyways. The controls on the F200 are very useful for getting good data optimized for different conditions. For example at the default settings you can't get depth super close to the camera because the laser projector washes out the image, but if you lower the brightness of the laser then you can. |
Hi Teknotus,
|
I've followed the guidance from this link to reset the camera via software before. Fully automating the process would take the extra step of finding the /dev/bus/usb/002/003 or whatever it happens to be connected to via software. The find_cameras.c has most of the code to find it, but needs some more work. So I added the second line here to find_cameras.c, and I think that may be printing out the device node you need now. Just need to figure out how to take that string, and combine it with the code from the link above to reset the camera.
So the video4Linux2 utility can list all the formats a camera supports through Linux. The --help-all option gives the full list of options. More than the man page on my system. Basically... From the patch I would expect to see the following 4 formats that aren't standard for Linux, and possibly some others that Linux didn't need a patch to support. Y8I Greyscale 8-bit Left/Right interleaved I expect the Y8I, and Z16 would be the two easiest to display since they align nicely. Just treat them as 8 or 16 bit grayscale images, and you'll probably have something relatively close to useful. You shouldn't need to getControl to get useful data from the camera. The default settings should be good for many applications, but there are probably useful tweaks available. |
Oh I think Y16, and GREY are also format options. |
Hi Teknotus, According to the converting depth image to gray level, why do you know "depth = int(depth/31.25 + 0.5); // convert to mm", could you please tell me where you get the information ? |
I believe @BlazingForests figured it out for his ROS module. Unfortunately it's a number that changes. It seems to be temperature dependent. I made a tool to try to read the temperature, and other data from the camera which is in rstools. I haven't built a tool to do the calibration based on those numbers though. The temperature stabilizes after a while so you could take some measurements, and get a regression fit to find the constant that you can use as long as the camera is warm. There is a good chance that the interface for collecting this information has changed with the R200 so it might not work at all. It's also possible that the R200 takes the load off of the drivers, and delivers the data in something ready to use like millimeters which would make the job a lot easier. |
Thanks @teknotus @Wesley-Lin I don't have to figure out the calibration of depth and temperature. But i can update code to add @teknotus 's get temperature code, show the debug info temperature and center depth value at the same time. thx |
Hi Teknotus, Second, the definition of depth is 16 bits and 1 channel, however, you use 2 channels to assign value, Finally, I want to compare r200's depth format with f200's. I find the r200's fomat is "Z16_2_1" and it s description is "16 bits, 2 pixels packed into 1 32 bit word", do you know the f200's depth format ? Thank you so much |
I that might be a bug that got fixed in the remote_control branch.
Intel almost exclusively uses little endian. I needed the low, and high bytes separate for something I was doing in an older version of the code base for a feature I removed.
I think that is specifying the X and Y location for a single channel. "depth_cv" is defined as 16 bit unsigned pixels so I don't think it would compile if I tried to assign two channels.
I think Linux will give you a byte buffer, and so the fact that it's defined as two pixels in 32 bits of data may not matter if you just cast the pointer to 16 bits. It might do something weird like indicating that the width is half of the real number of pixels. Since most display systems can only handle 8 bits per color I keep the 16 bit version for point clouds, but drop down to 8 bits for direct display. I'm not totally sure about the F200 depth format. It actually has at least two distinct depth formats. I'm betting one is a 16 bit format that has a linear transformation into real world units like meters, and the other is something like the 10 bit raw format mentioned in the R200 kernel patch padded to 16 bits. I don't have many guesses beyond that. |
I was update my code to show center Z value and realsense's temperature at the same time. Thx |
Every 3d camera I've ever worked with has some amount of "holes" in the 3d data. When the camera can't calculate the distance for whatever reason it will return black instead. Shiny objects are notoriously difficult. The shiny spot may wash out the infrared projector, or if 3d is being generated from disparity images the shiny spot will because of physics be in an inconsistent location in the two images throwing off feature matching. Pure black is also bad because there just isn't anything to work with. Now since the 3d is extracted from infrared it might not be black as you see it, but be black in the infrared spectrum, and still have the problem. As I understand it the R200 can generate 3d in two ways. It can use a laser projector like the F200, or it can use two infrared images from different angles, and match features to extract 3d. I'm not sure if it does the feature extraction, and matching in hardware on the device or is designed to just send both video streams to the computer to do the work. Depending on how it is generating the 3d there are different possible lighting problems. If it is using the laser projector sources of infrared light in the environment might cause interference by washing out the laser light. If it is generating the 3d from infrared disparity images lack of infrared light could cause problems. The F200 has multiple special controls on it that are most useful as a means of optimizing for different lighting conditions, and distance between the camera and the scanning target. The USB header indicates that the R200 has several special controls. They have different specifications from the F200 so I cannot easily guess what they are but they are probably for the same types of tuning.
This might be an integer overflow. Most display systems only handle values between 0-255, but the data from the camera may be as many as 16 bits or 0-65535. Depending on how you convert you may get a sweep from black to white every 255 units of distance.
This is an artifact of how the image is generated. If a part of the image has line of sight from the camera, but not the infrared projector it won't be able to image it. In most application this is worked around by remembering what is there, or stitching pieces together. So for example if the camera isn't moving you could get an image of the background before entering the scene, and fill in this type of hole with the previously recorded background. In many applications you don't actually care about the background, and might simply throw away data further than a given distance from the camera. This might be as simple as.
If on the other hand you are trying to scan things you can't get an image of both the front an back of a head at the same time so you'll have to move the camera around anyways, and splice partial 3d models together. There are strategies that use more cameras to fill in holes, but more cameras is more expensive, and if you are active scanning with an infrared projector there is some interference between them. The technology isn't perfect yet. |
Verbose lsusb will give you most of the USB header stuff. The part that declares special controls for the R200 is this.
In that the bNumControl is the number of controls (21), but the bmControls lines are a bitmask indicating which ones are actually enabled. Not all of the bits are ones so I expect there are slightly less than 21 controls. The F200 has bNumControl of 6 but the 4th bit is a zero, and thus there is no control 4. The USB video class driver for Linux knows how to read it though so you just need to know how to use it to probe for available controls. Some of the information about the controls isn't available in the header, but can be probed for such at the range, and default values. Those numbers were enough for me to match the F200 controls as found on the device with the controls defined in the SDK documentation. Hopefully that will at least partially work with the R200, but it looks like there are a lot more controls. Look at the kernel documentation for UVCIOC_CTRL_QUERY I used this method before I figured out what the controls were, and how to define names for them. The current version of the code has that code removed so look at an old version of. You might need to run the code as root, or give yourself special permissions on the camera. You need to know more about the nature of the controls on the R200 than I currently do to use the method I switched to in the current code base so this is probably the best method to get at the controls at the moment. |
Have you tried swapping the high, and low bytes? That's my best guess at the moment. |
Hi Daniel, yes, the high and low bits don't need to swap, I had tested it, and the depth image don't show wave at all. Thanks |
I've had inexplicable problems with the depth camera, and epoll that doesn't exist with other cameras. For some reason the epoll will just never get the message that things are ready to go, and it will just wait forever. I would try swapping it with some kind of timer like usleep. depthview uses a timer instead of epoll. I think it checks for a new frame once a millisecond which is more latency, and resources than epoll, but still plenty fast for video. I haven't had a chance to try two F200 cameras at once. |
Hi Daniel, Now, I try to read the RGB & depth data from single R200 simultaneously, but it still cannot work. The way I use is the same with before I said, two same app in different folder read the video1 & video2. However, If I read RGB first then read depth, depth image would not be getted but RGB still can work; If I read depth first then read RGB, depth would be stopped and RGB cannot be get neither. Thanks a lot |
Hmm. My best guess is that maybe you are reaching the bandwidth limit of USB. You could try lowering the frame rate, and or resolution. Are there any dmesg errors? |
Hi Daniel, The resolution Depth is 628 x 468, RGB is 640 x 480, and I will check the frame rate. Thanks |
Hi teknotus,
I try to use VLC application to read raw data from realsense, and I also use the patch to add UVC format. However, the depth image is green and resolution is not the same with RGB image.
Could you please teach me how to solve them ?
Thanks
Regards
The text was updated successfully, but these errors were encountered: