How to get rgb data from depth map

[d12889811]

Hi. Thanks for providing such useful utilities for MR development. Currently I am able to calculate world position with a linear depth map given. I'm wondering that for each pixel of depth data is there any way to directly get the corresponding rgb value in passthrough camera frame data?

[MartinSherburn]

@d12889811 this is actually more difficult than it appears on the surface for a few reasons:

• The sensors/cameras on the Quest to compute depth are different from those that hold the RGB values. Therefore there is an offset between them and although we do our best to align them it's difficult to get a pixel perfect alignment.
• The depth/rgb images also need to be synchronized in time, and currently there isn't an easy way to do that (although we are working on it).

Could you share your higher level goal and we can do our best to advise you on an alternate solution?

[d12889811]

I would like to generate some objects in unity world and make them look exactly how they are in real world. Currently I can spawn them in right place and next thing I need to do is to set correct texture which comes from real world for them. Apprecited if you could give me some advice on how to achieve the goal

[TudorJude]

Hey,

What do you mean by "set the correct texture" here? What kind of texture do you want to apply to your virtual objects?

[d12889811]

For example I’ll first walk around the cup and use the depth map to reconstruct a cup-shaped mesh in the game. Then I want the mesh object to look the same as the real cup—by using the corresponding RGB images at each surface position to set vertex colors, or with some other method. Is this feasible?

[lszmer]

Currently working on exactly the same issue. Got the raw depth data and RGB frames, captured at synchronised ISO timestamps with time lag of around 20ms at 10hz, which I hope is fine. I'd now continue calibrating the camera extrinsics between the two sensors to align the images and get RGB-D data. Then use object reconstruction methods on the row-captured images. Will start with objects not occluded by hands and check if 320x320 and depth sensitivity is actually enough.

Question: what unity does the depth sensor measure in? I have RGB16 data and the relevant stuff is in the red channel, but closer objects have higher value and values are bounded between 0 and 1, i.e. I need to inverse and then undistort. Is there a known inverse function to transform depth R-channel into meters?

[TudorJude]

Question: what unity does the depth sensor measure in? I have RGB16 data and the relevant stuff is in the red channel, but closer objects have higher value and values are bounded between 0 and 1, i.e. I need to inverse and then undistort. Is there a known inverse function to transform depth R-channel into meters?

Yes, the depth data is stored in the red channel and has a non-linear value scaling from 1 down to 0 (as you move further away). If you look in the core sdk for a file called "EnvironmentOcclusion.cginc" you can see how we calculate linearSceneDepth in relation to a world position value in the CalculateEnvironmentDepthOcclusion function, that should put you on the right track.

[lszmer]

Thanks for the heads-up, I actually struggled to get the formula to work;

linearDepth = (1.0 / (inputDepthNdc + _EnvironmentDepthZBufferParams.y)) * _EnvironmentDepthZBufferParams.x

where
inputDepthNdc = rawDepthValue * 2.0 - 1.0
_EnvironmentDepthZBufferParams.y = -1.0 (infinite far plane) //DepthOffset fixed at -1.0
_EnvironmentDepthZBufferParams.x = = -2.0 * near //For near=0.1: invDepthFactor = -0.2

For example, for rawDepth of 0.3 (roughly 20-30cm) this yields 0.28m, but for rawDepth of 0.1 (roughly 50cm-80cm) this yields 0.22m. Maybe I am missing something here though.

However, I proxied with near_clip of 0.1 and currently calculate depth = 0.1/rawSensorValue, which yields useful results. Will calibrate the conversion to better detail soon.

Currently working on the calibration/alignment of RGB with depth sensor to get a fused RGB-D image, do you have any intrinsics (focal lengths, principal points, skew) on the depth sensor? I can access RGB intrinsics via API, couldn't find such data for the depth sensor. I assume you do not know the extrinsics transform matrix between the two sensors?

For the interested reader:
(RGB_INTRINSICS = {
'fx': 868.87, # Focal length X (pixels)
'fy': 868.87, # Focal length Y (pixels)
'cx': 636.96, # Principal point X (pixels)
'cy': 477.66, # Principal point Y (pixels)
'skew': 0.0, # Skew coefficient
'width': 1280, # Image width
'height': 960 # Image height
}

[lszmer]

Please disregard my comment; all relevant work on live data capture and synchronisation of depth and RGB files can be found in repos
https://github.com/t-34400/QuestRealityCapture/
for capture and

https://github.com/t-34400/metaquest-3d-reconstruction
for reconstruction