# Derive affine transformation from dense optical flow

I’m about to determine the 2D affine transformation between two versions of an image.

My idea was to compute optical flow between the images
and then use that flow to compute the 2D affine transformation.
I want to use Farnback’s alg for dense optical flow.

The problem:
`estimateAffine2D` always delivers no inliers, and so an empty affine transformation matrix.

For demonstration a simple consistency test:

• Take an image
• Generate a second one by applying a defined affine transformation.
• Compute the dense flow between them.
• Try to retrieve the original transformation by applying `estimateAffine2D` at the flow:
``````    Mat image = imread("input.png"),
translated_image;

float tx = 8;
float ty = 15;
float warp_values[] = { 1.0, 0.0, tx, 0.0, 1.0, ty };   // translation (tx,ty)
Mat translation_matrix = Mat(2, 3, CV_32F, warp_values);
warpAffine(image, translated_image, translation_matrix, image.size());

// compute dense optical flow
Mat matGrayImage, matGrayTranslated_image;
cvtColor (image, matGrayImage, COLOR_BGR2GRAY);
cvtColor (translated_image, matGrayTranslated_image, COLOR_BGR2GRAY);

Mat theFlow (image.size(), CV_32FC2, Scalar (0,0));
calcOpticalFlowFarneback (matGrayImage, matGrayTranslated_image, theFlow, 0.5, 3, 15, 3, 5, 1.2, 0);

cout << "mean (flow): " << mean (theFlow) << endl;

// build point vectors from flow
std::vector< Point2f > pts1;
std::vector< Point2f > pts2;
for (int i=0; i < theFlow.rows; i++)
for (int j=0; j < theFlow.cols; j++) {
pts1.push_back (Point2f (0,0));
pts2.push_back (Point2f (theFlow.at<Vec2f>(i, j)[0], theFlow.at<Vec2f>(i, j)[1]));
}

std::vector<uchar> inliers (pts1.size(), 0);
Mat warp_matrix = estimateAffine2D (pts1, pts2, inliers);

cout << "Number of inliers: " << countNonZero (inliers) << endl;
cout << "Warp matrix: " << warp_matrix;
``````

The output:

``````mean (flow): [4.06175, 7.76149, 0, 0]
Number of inliers: 0
Warp matrix: []
``````

Where is the bug?

I tried to encircle the cause of the effect by reducing due to points:

• Their number (maybe too much for `estimateAffine2D`?)
• Their variance (maybe too large?)

First (reduction on 3x3 points):

``````for (int i=0; i < 3; i++)
for (int j=0; j < 3; j++) {
...
}
``````

the output:

``````mean (pts2): [9.35273, 12.2285, 0, 0]
Number of inliers: 0
Warp matrix: []
``````

Now additional reduction: all points identical value (exact value of trans.)
(Hopefully the solution will still be unambiguous)

``````...
pts1.push_back (Point2f (0,0));
pts2.push_back (Point2f (tx, ty));
...
``````

The output:

``````mean (pts2): [8, 15, 0, 0]
Number of inliers: 0
Warp matrix: []
``````