Thin plate spline warpImage produces only zeros

Python
, ,
1

Dear all,

I am trying to warp an image using the thinplatespline but after warping, the image contains only 0 values. I am attaching a working code example and the datasets as well. I hope for some help to show me the right path again…

image
image3334×3334 2.06 MB 

import cv2
import numpy as np
from PIL import Image


pts1 = np.array([[[1000., 1000.], [1000., 2800.], [1000., 4600.], [1000., 6400.],
                  [1000., 7667.], [2800., 1000.], [2800., 2800.], [2800., 4600.],
                  [2800., 6400.], [2800., 7667.], [4600., 1000.], [4600., 2800.],
                  [4600., 4600.], [4600., 6400.], [4600., 7667.], [6400., 1000.],
                  [6400., 2800.], [6400., 4600.], [6400., 6400.], [6400., 7667.],
                  [7667., 1000.], [7667., 2800.], [7667., 4600.], [7667., 6400.],
                  [7667., 7667.]]])
pts2 = np.array([[[1007.,  989.], [1006., 2789.], [1005., 4586.], [1005., 6386.],
                  [1002., 7653.], [2808.,  989.], [2808., 2785.], [2806., 4586.],
                  [2802., 6387.], [2801., 7654.], [4605.,  992.], [4598., 2792.],
                  [4595., 4590.], [4593., 6390.], [4594., 7656.], [6390., 1000.],
                  [6392., 2797.], [6388., 4597.], [6391., 6392.], [6390., 7659.],
                  [7655., 1006.], [7654., 2802.], [7654., 4600.], [7654., 6395.],
                  [7654., 7662.]]])

image = np.array(Image.open('image.jpg'))

bufferzise = 2000

imbuffer = np.zeros((image.shape[0] + bufferzise,
                    image.shape[1] + bufferzise))

imbuffer[int(1000):int(1000 + image.shape[0]),
         int(1000):int(1000 + image.shape[1])] = image[:, :]

matches = list()
for ipoint in range(0, pts1.shape[1]):
    matches.append(cv2.DMatch(ipoint, ipoint, 0))
tps = cv2.createThinPlateSplineShapeTransformer()
tps.estimateTransformation(pts2, pts1, matches)
out_img = tps.warpImage(imbuffer)
2

welcome.

could you point that out precisely in the picture? what do you expect to be there instead, and why should it be there?

3

Thank you for your reply, and my apologies for not being precise. The expected result is the same image but slightly moved (including the small black part of the image which is part of it). Instead the produced array (out_img) contains only zeros no actual data. I upload also the erroneous image I get.

black
black572×580 258 Bytes

4

ah, literally a black image that is black everywhere! it sounded like the result was somewhat more complex.

ok well without trying your code, I’d suggest instantiating the transformer using the static create method:

tps = cv2.createThinPlateSplineShapeTransformer.create() # or ...Transformer_create

see if that helps?

by the way: opencv has imread. spares you from using PIL, unless you want to use it. PIL loads images as RGB channel order. OpenCV loads them as BGR order (and relevant functions assume BGR order).

5

Hi crackwitz and goodday, and thank you for the welcoming! I was too fast to dive in my problem i think :slight_smile:

Some more information:
I am using openCV version cv2.version : ‘3.4.1-dev’
The .create() method does not exist, hence I have to instantiate they way I show above.
I’ve tried the buildin function cv2.imread() as well. The behavior is the same, converts all values to a constant value of 152. I am really confused here.

6

oh, my mistake!

the python API in opencv v3.x has identifiers like createThinPlateSplineShapeTransformer
the v4.x api has ThinPlateSplineShapeTransformer.create

you seem to have used the right identifier already and I missed that.

hm well, now it gets trickier… I hope someone else can get involved. I’m currently not set up to run most opencv modules.

it may be that that code/APIs has never been tested for use with python, and some things in the bindings generation don’t work right. the docs don’t even mention the static create method, which makes me suspicious.

or it might all be working correctly and the issue is something else.

7

Hey thanks for your feedback. Crossing fingers that someone has faced and overcame the same issue…

8

shot in the dark: try passing pts1 and pts2 as np.float32. np.array on python floats will result in np.float64, which some OpenCV APIs don’t expect, so this might be one of those.

same goes for your imbuffer. make sure it’s some “tame” type like float32 or uint8. specify dtype=... in the creation of it.

9

I did tried float32 but same result… everything is black.
I did notice though the following.

  1. Use the whole image which contains a corner with 0s. Put it within this buffered area (imbuffer) which is also 0s. Then the result is this black image. So the array contains only 0s.

  2. I resampled the initial image in order to avoid the small black area at the corner, hence not to contain 0s. Following i added the buffer area, so I have a zero area around the image. The warping image is normal! Meaning that it is not black!

  3. I cheated by using the initial image but replacing all 0s with 1s. Added the buffer. The results again the same; just a totally black image.

Not sure if that leads to somewhere but definitely doesn’t make sense to me.

10

if you believe it could be a bug, feel free to assemble a minimal reproducible example including all necessary data (images) and open an issue on github. you should make sure that issue is reproducible with master branch (latest 4.x) or 3.4 branch, i.e. no old versions of opencv (pre v4.5, pre v3.4)

11

That’s probably a good idea, will try to prepare something as soon as I find the time. Thanks again for your time :slight_smile:

12

I ran into the same issue and the reason was due to choice of points. Just try shrinking it pts2[:,:4,:], pts2[:,:4,:] and check if it’s still black – if yes, choose more spatially distanced points. With certain set of points you’ll get a black screen even if you pass same set of points twice, i.e.:
tps.estimateTransformation(pts, pts1, matches)
^ i’m not sure why it’s the case though. OpenCV 4.5.2