Derive affine transformation from dense optical flow

C++
, ,
1

I’m about to determine the 2D affine transformation between two versions of an image.

My idea was to compute optical flow between the images
and then use that flow to compute the 2D affine transformation.
I want to use Farnback’s alg for dense optical flow.

The problem:
estimateAffine2D always delivers no inliers, and so an empty affine transformation matrix.

For demonstration a simple consistency test:

  • Take an image
  • Generate a second one by applying a defined affine transformation.
  • Compute the dense flow between them.
  • Try to retrieve the original transformation by applying estimateAffine2D at the flow:
    Mat image = imread("input.png"),
        translated_image;

    float tx = 8;
    float ty = 15;
    float warp_values[] = { 1.0, 0.0, tx, 0.0, 1.0, ty };   // translation (tx,ty)
    Mat translation_matrix = Mat(2, 3, CV_32F, warp_values);
    warpAffine(image, translated_image, translation_matrix, image.size());

    // compute dense optical flow
    Mat matGrayImage, matGrayTranslated_image;
    cvtColor (image, matGrayImage, COLOR_BGR2GRAY);
    cvtColor (translated_image, matGrayTranslated_image, COLOR_BGR2GRAY);

    Mat theFlow (image.size(), CV_32FC2, Scalar (0,0));
    calcOpticalFlowFarneback (matGrayImage, matGrayTranslated_image, theFlow, 0.5, 3, 15, 3, 5, 1.2, 0);

    cout << "mean (flow): " << mean (theFlow) << endl;

    // build point vectors from flow
    std::vector< Point2f > pts1;
    std::vector< Point2f > pts2;
    for (int i=0; i < theFlow.rows; i++)
        for (int j=0; j < theFlow.cols; j++) {
            pts1.push_back (Point2f (0,0));
            pts2.push_back (Point2f (theFlow.at<Vec2f>(i, j)[0], theFlow.at<Vec2f>(i, j)[1]));
    }

    std::vector<uchar> inliers (pts1.size(), 0);
    Mat warp_matrix = estimateAffine2D (pts1, pts2, inliers);

    cout << "Number of inliers: " << countNonZero (inliers) << endl;
    cout << "Warp matrix: " << warp_matrix;

The output:

mean (flow): [4.06175, 7.76149, 0, 0]
Number of inliers: 0
Warp matrix: []

Where is the bug?

ADDITION:

I tried to encircle the cause of the effect by reducing due to points:

  • Their number (maybe too much for estimateAffine2D?)
  • Their variance (maybe too large?)

First (reduction on 3x3 points):

for (int i=0; i < 3; i++)
    for (int j=0; j < 3; j++) {
...
}

the output:

mean (pts2): [9.35273, 12.2285, 0, 0]
Number of inliers: 0
Warp matrix: []

Now additional reduction: all points identical value (exact value of trans.)
(Hopefully the solution will still be unambiguous)

...
pts1.push_back (Point2f (0,0));
pts2.push_back (Point2f (tx, ty));
...

The output:

mean (pts2): [8, 15, 0, 0]
Number of inliers: 0
Warp matrix: []