# Understanding the functions projectPoints() and undistortPoints()

**URL:** <https://forum.opencv.org/t/understanding-the-functions-projectpoints-and-undistortpoints/10847>\
**Category:** Python\
**Tags:** calib3d\
**Created:** [November 6, 2022, 7:20pm UTC](https://forum.opencv.org/t/understanding-the-functions-projectpoints-and-undistortpoints/10847 "2022-11-06T19:20:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![javier](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/javier/32/7284_2.png) [@javier](https://forum.opencv.org/u/javier)\
**Post date:** [November 6, 2022, 7:20pm UTC](https://forum.opencv.org/t/understanding-the-functions-projectpoints-and-undistortpoints/10847/1 "2022-11-06T19:20:56Z")

</div>

Trying to check if I was using correctly both [`projectPoints()`](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga1019495a2c8d1743ed5cc23fa0daff8c) and [`undistortPoints()`](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga55c716492470bfe86b0ee9bf3a1f0f7e), I am obtaining results which I find hard to understand.

Having defined the following intrinsic parameters:

```python
# calibration matrix
K = np.array([
    [500.0, 0.0, 300.0], 
    [0.0, 500.0, 250.0],
    [0.0, 0.0, 1.0]
    ])

# distortion coefficients (k1, k2, p1, p2, k3)
distCoeffs = np.array([1.5, -0.95, -0.005, 0.0025, 1.16])

```

and the following pointcloud in the camera reference (with all the points being in the same plane):

```python
# Coordinates of points in plane, representing a pointcloud in camera reference (c)
# plane
H, W = 1, 2
X, Y = np.meshgrid(np.arange(-W, W, 0.2), np.arange(-H, H, 0.2))
X, Y = X.reshape(1, -1), Y.reshape(1, -1)
# add depth. Pointcloud of n points represented as a (3, n) array:
Z = 5
P_c = np.concatenate((X, Y, Z * np.ones_like(X)), axis=0)

```

I was expecting that the following process would yield the original pointcloud:

1. **Projecting** the points, while accounting for the distortion i.e. obtaining the **distorted** coordinates:

```python
# project points, including with lens distortion 
U_dist, _ = cv2.projectPoints(P_c, np.zeros((3,)), np.zeros((3,)), K, distCoeffs)
# projections as (2, n) array.
U_dist = U_dist[:, 0].T

```

1. **Undistort** them to get the normalized coordinates:

```python
# get normalized coordinates, in camera reference, as a (2, n) array
xn_u = cv2.undistortPoints(U_dist.T, K, distCoeffs, None, None)[:, 0].T

```

1. and multiplying the previous normalized coordinates with the depth of the plane in the camera reference to get the original pointcloud:

```python
# add depth.
P_c2 = Z * np.concatenate((xn_u, np.ones_like(X)))

```

> **\[Complete code\]**
>
> ```python
> import numpy as np
> import cv2
> 
> # calibration matrix
> K = np.array([
> [500.0, 0.0, 300.0], 
> [0.0, 500.0, 250.0],
> [0.0, 0.0, 1.0]
> ])
> 
> # distortion coefficients (k1, k2, p1, p2, k3)
> distCoeffs = np.array([1.5, -0.95, -0.005, 0.0025, 1.16])
> 
> # Coordinates of points in plane, representing a pointcloud in camera reference (c)
> # plane
> H, W = 1, 2
> X, Y = np.meshgrid(np.arange(-W, W, 0.2), np.arange(-H, H, 0.2))
> X, Y = X.reshape(1, -1), Y.reshape(1, -1)
> # add depth. Pointcloud of n points represented as a (3, n) array:
> Z = 5
> P_c = np.concatenate((X, Y, Z * np.ones_like(X)), axis=0)
> 
> # ---------------------------------------------
> # PROJECTION WITH DISTORTION
> # project points, including with lens distortion 
> U_dist, _ = cv2.projectPoints(P_c, np.zeros((3,)), np.zeros((3,)), K, distCoeffs)
> # projections as (2, n) array.
> U_dist = U_dist[:, 0].T
> 
> # UNPROJECTION accounting for distortion
> # get unnormalized coordinates, in camera reference, as a (2, n) array
> xn_u = cv2.undistortPoints(U_dist.T, K, distCoeffs, None, None)[:, 0].T
> # add depth.
> P_c2 = Z * np.concatenate((xn_u, np.ones_like(X)))
> 
> # check equality (raises error)
> assert np.allclose(P_c, P_c2), f'max difference: {np.abs(P_c - P_c2).max()}'
> 
> ```

However, this is not true and the resulting pointcloud is significantly different than the original one.

I feel this may be due to a misunderstanding in the use of the previous functions.

Any help to understand where I’m doing the wrong step(s) is highly appreciated.

* * *

**EDIT**  
After some more experimentation, I believe the issue is with `undistortPoints()` rather than with `projectPoints()`. The latter is deterministic, while the former needs to solve a non-linear optimization problem. Emprically, by increasing the distortion, `undistortPoints()` tends to give worse results. However at lower levels, it correctly fixes it.

---

<div class="post-metadata">

**Author:** ![crackwitz](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/crackwitz/32/14_2.png) [@crackwitz](https://forum.opencv.org/u/crackwitz)\
**Post date:** [November 6, 2022, 7:57pm UTC](https://forum.opencv.org/t/understanding-the-functions-projectpoints-and-undistortpoints/10847/2 "2022-11-06T19:57:36Z")

</div>

undeclared crosspost:

> <https://stackoverflow.com/questions/74339089/understanding-the-functions-projectpoints-and-undistortpoints-of-opencv>