Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

D455 infrared RGB streaming #26

Closed
bmegli opened this issue Oct 29, 2020 · 5 comments
Closed

D455 infrared RGB streaming #26

bmegli opened this issue Oct 29, 2020 · 5 comments
Labels
enhancement New feature or request

Comments

@bmegli
Copy link
Owner

bmegli commented Oct 29, 2020

librealsense 2.39.0 with D4xx firmware v5.12.8.200 adds possibility to stream synthetic RGB from left imager.

The quality compared to RGB is probably bad.
At the same time this stream is pixel perfect aligned with the depth without any additional computations.

Extending RNHVE to support infrared RGB should be rather straightforward.
@bmegli bmegli added the enhancement New feature or request label Oct 29, 2020
@bmegli
Copy link
Owner Author

bmegli commented Oct 29, 2020

From realsense datasheet Table 4-3. Image Formats (USB 3.1 Gen1) – D455 for left imager supports pixel formats:

  • Y8 (iuminance only)
  • UYVY (infrared RGB)

In realsense-viewer we also see (working) formats like RGB8, BGR8.

Examing librealsense code it is difficult to say which format is native so let's stick with UYVY from datasheet for now.

@bmegli
Copy link
Owner Author

bmegli commented Oct 29, 2020

From HVE supported pixel formats issue we see that VAAPI supports uyvy422 pixel format. We should be able to match it directly with librealsense UYVY.

@bmegli
Copy link
Owner Author

bmegli commented Oct 30, 2020

Current Realsense datasheet (June 2020) is missing information about IR imager sensor for D455.

It is clearly neither OV2740 (D415 which is rolling shutter and not wide) nor OV9282 (D435 which is monochrome).

librealsense#7260 states that D455 uses OV9782.

librealsense#7155 states that D455 uses the same sensor for color and IR.

Omnivision OV9782 page and product guide. Probably the raw data is some Bayer. This is processed by realsense asic/firmware. From our point of view native format is whatever realsense uses internally so that we avoid unnecessary conversion on CPU.

bmegli added a commit that referenced this issue Oct 30, 2020
@bmegli
infrared rgb H.264 streaming
c17c16d 
- add possibility to stream infrared rgb to H.264 example

Visually reasonable results require:
- disabling auto exposure and setting it manually
- decreasing IR projector power

Related to #26
bmegli added a commit that referenced this issue Oct 30, 2020
@bmegli
infrared rgb HEVC streaming
8ac7dde 
Related to #26
bmegli added a commit that referenced this issue Oct 30, 2020
@bmegli
depth + infrared rgb HEVC Main 10 + HEVC streaming
c1a0a3b 
Related to #26
@bmegli
Copy link
Owner Author

bmegli commented Oct 30, 2020

Ready for merge.

Reasonable results with infrared rgb require:

  • disabling auto exposure
  • lowering IR projector power
  • setting higher manual exposure

Otherwise the image is too dark and IR projector pattern too strong.

@bmegli bmegli mentioned this issue Oct 30, 2020
Merged
bmegli added a commit that referenced this issue Oct 30, 2020
@bmegli
infrared rgb streaming (D415/D455)
091f785 
- infrared rgb H.264 streaming
- infrared rgb HEVC streaming
- depth + infrared rgb HEVC Main 10 + HEVC  streaming

Related to #26
@bmegli
Copy link
Owner Author

bmegli commented Oct 30, 2020

The last thing to do is updating wiki how-it-works section.

@bmegli bmegli closed this as completed Oct 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bmegli