# Merge 2D image and depth map

**URL:** <https://forum.opencv.org/t/merge-2d-image-and-depth-map/19144>\
**Category:** Python\
**Tags:** calib3d\
**Created:** [October 30, 2024, 6:33am UTC](https://forum.opencv.org/t/merge-2d-image-and-depth-map/19144 "2024-10-30T06:33:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![david1](https://avatars.discourse-cdn.com/v4/letter/d/73ab20/32.png) [@david1](https://forum.opencv.org/u/david1)\
**Post date:** [October 30, 2024, 6:33am UTC](https://forum.opencv.org/t/merge-2d-image-and-depth-map/19144/1 "2024-10-30T06:33:10Z")

</div>

Hi,  
I have a mecheye deep camera. It gives me a colour image and a depth map. It also has a function that returns the parameter arrays for each camera.  
My project consists of finding boxes on a pallet at different heights. Using the API provided by mechmind and OpenCV I am trying to superimpose the deepmap and the 2D image, but I am having some problems.  
At a distance where I make a homography between both images everything is fine. But when I decrease that distance I start to see a horizontal sliding between both images.  
Should I adjust the homography I have applied depending on the distance I am working at, or should that homography be one for any distance? That distance is provided by the 3D image.

As you can see in the image there is a horizontal misalignment in the top box marked in red. The one marked in blue is the distance at which I perform the homography.

 ![img_sliding](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/5/5f9735fb8739af406d8be8e642b46ff1b32d12a1.jpeg)

Could you help me with my problem?  
Best regards and thanks

---

<div class="post-metadata">

**Author:** ![crackwitz](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/crackwitz/32/14_2.png) [@crackwitz](https://forum.opencv.org/u/crackwitz)\
**Post date:** [October 30, 2024, 8:32am UTC](https://forum.opencv.org/t/merge-2d-image-and-depth-map/19144/2 "2024-10-30T08:32:20Z")

</div>

homography is the wrong approach.

the depth view is taken from a different position, not the same as the visual view.

you need to _project_ the visual view _onto_ the point cloud/mesh, taking the transform between views into account.

their website looks like they have examples for that.

---

<div class="post-metadata">

**Author:** ![david1](https://avatars.discourse-cdn.com/v4/letter/d/73ab20/32.png) [@david1](https://forum.opencv.org/u/david1)\
**Post date:** [October 31, 2024, 6:24am UTC](https://forum.opencv.org/t/merge-2d-image-and-depth-map/19144/3 "2024-10-31T06:24:08Z")

</div>

Thank you for your response. I have seen an example on the mechmind page for projecting a 2d image onto a point cloud. What I want to do is to create a mask with everything at a certain distance from the camera to filter the 2D image, but at the moment I don’t know how to do it.  
The camera returns me this parameters, but I don’t understand the translation that the camera is giving to me.

Rotation: From Depth Camera to Texture Camera:  
[0.9994753140272203, -0.004729284653347138, -0.03204263592242101]  
[0.004422137710777892, 0.9999436627330821, -0.009649666212622343]  
[0.032086466746217575, 0.00950290621945331, 0.9994399198676868]

Translation From Depth Camera to Texture Camera:  
X: -20.705958880687646mm, Y: 2.605356548486989mm, Z: 20.070005911936793mm