How to understand optical flow's corner detection and calcOpticalFlowPyrLK() p0 and p1

C++
1

Hello guys this is my first question at all,

I am a student and beginner in C++ I have been dealing exclusively with the Lucas-Kanade Alg. for the last few days and I am reaching the limits of my cognitive capacity. I am not aware of anyone who could help me therefore I ask you. In the video I want to use, the ROI is placed on a person’s chest to capture upward and downward motion at the sagital plane. From the obtained information of the pixel arrangement of each frame, the duration of the inspiration time can be determined.

The following problem:

In a short test video, an ROI in the first frame is selected by hand to determine the optical flow vectors [p0_prev] of the goodFeaturesToTrack() function there, which serves as a parameter for determining the pixels location of the next frame in calcOpticalFlowPyrLK[p1_next] - got this after reading the opencv doc, check. p0 and p1 are velocity vectors, fine. But that is nearly all i understand in terms of p0 and p1.

My thoughts:

goodFeaturesToTrack(old_gray, p0, 100 , 0.1, 3, dist, 3, false, 0.04);

  • I use “100” because the ROI is big and size(p1) will be same size as p0 the more values i get, the more detail my movement detection, right ? maybe i can use even more ?

  • **Are all relevant values of the ROI preserved in p0? How can I ensure this ? **

  • Is it possible to improve the quality of p0 in the parameters ?.
    i want to compare p0,p1 where status i == 0 or 1 statisticly in each framepair via correlatio n[Matlab]after the algo finished its work.

  • i want to add forward backword error detection, but i am not able to implement it :frowning: I think it goes with 2x calcopticalflow.

` calcOpticalFlowPyrLK(old_gray, frame_gray, p0, p1, status, err, Size(5,5), 2, criteria);

  • `How can I adjust Size(5,5) in a meaningful way or what effects does it have on the quality of p1 - is it a good idea to make the size bigger?
  • i want to get the “good corners” from status == 1 to improve the algo and select the good ones
  • What is the relation between max_corners from goodFeaturesToTrack and Size(a*a) from calcOpticalFlowPyrLK ?
  • **How can I ensure that my features are properly defined to be sufficient and compatible for the set ROI?
    **

Problems in the code:

  • p0 decrements in time (and so does p1) - WHY?
  • every member of p1 gets the status == 1 - WHY? i think this is wrong,

In case of the Universtiys lecture i need to export the y-values and plot them in Matlab to compare them with a signal given from a different system - thats why i need to have valuable informations…

Full Code

void Termin_2_3_3(string video, string filename) {


	// load video
	VideoCapture own(video);

	int frame_width = own.get(CAP_PROP_FRAME_WIDTH);
	int frame_height = own.get(CAP_PROP_FRAME_HEIGHT);
	double fps = own.get(CAP_PROP_FPS);

	VideoWriter video_written("Tracking" + video, VideoWriter::fourcc('M', 'J', 'P', 'G'), 10, Size(frame_width, frame_height));

	cout << "Anzahl der Frames im Video =  " << fps << endl;

	ofstream frame_ofs;

	frame_ofs.open(filename + ".dat"); // save the video 


	if (!own.isOpened()) {
		//error in opening the video 
		cerr << "Unable to open video!" << endl;

	}

	Mat first_frame, old_gray;

	vector<Point2f> p0, p1;

	// Take first frame and find corners in it
	for (int i = 0; i <= 2; i++)
	{
		own >> first_frame;

	}
	   	 

	Rect ROI = selectROI("ROI", first_frame);
	destroyWindow("ROI");

	// Center of the ROI pixel x/y axis
	cout << "ROI.x MP =   " << ROI.x << " ROI.y MP =  " << ROI.y << endl;

	cvtColor(first_frame, old_gray, COLOR_BGR2GRAY);
	
	// Create source for further processing and copy in old_gray incl ROI
	Mat src;
	old_gray.copyTo(src);

	// create empty masks
	Mat m1 = Mat::zeros(old_gray.size(), CV_8UC1);
	Mat dist = Mat::zeros(src.size(), CV_8UC1);
	Mat output;


	/*Circe->(current image being edited on, center, radius, color, thickness, linetype, shift)
	Circle is then on black m1 image*/
	circle(m1, Point(ROI.x + 30, ROI.y + 30), 50, Scalar(255, 0, 0), -1, 8, 0);

	// src = Place gray image with ROI on still black images or empty masks m1 and dist
	src.copyTo(dist, m1);

	/*find strong corner by optical flow
	-> (input_image, outputVektor der Kanten, MaximumKanten, QualityLevel, MinDistancePixel, Maske)*/
	goodFeaturesToTrack(old_gray, p0, 100 , 0.1, 3, dist, 3, false, 0.04);
	// goodFeaturesToTrack(image_old, corner_old, 5, 0.1, 7, mask, 3, false, 0.04);

	cout << "Anzahl der Werte nach goodFeatures " << p0.size() << endl;
	cout << "Werte für p0 nach goodFeatures " << p0 << endl;

	// Create a mask image for drawing purposes
	Mat mask = Mat::zeros(old_gray.size(), old_gray.type());

	
	//Variables for the loops	int a = 30;
	int b = 1;
	int i = 0;

	while (i <= 4000)
	{
		Mat frame, frame_gray;

		//Shift next frame (once completely in frame)
		own >> frame;

			if (frame.empty())
			break;

		//convert
		cvtColor(frame, frame_gray, COLOR_BGR2GRAY);

		// calculate optical flow
		vector<uchar> status;
		vector<float> err;


		TermCriteria criteria = TermCriteria((TermCriteria::COUNT) + (TermCriteria::EPS), 20, 0.03);
		try
		{
			// 
			// Edge detection incl. optical flow vectors
			calcOpticalFlowPyrLK(old_gray, frame_gray, p0, p1, status, err, Size(5,5), 2, criteria);
			/* https://docs.opencv.org/4.6.0/dc/d6b/group__video__track.html#ga473e4b886d0bcc6b65831eb88ed93323 */
		}
		catch (Exception e)
		{
			e.formatMessage();
		}
		cout << "Number/Size of p0 after  calcOptical " << p0.size() << endl << endl;
		cout << "Number/Size of p1 after  calcOptical " << p1.size() << endl << endl;
		cout << "Number/Size ROI  " << ROI.size() << endl << endl;
	

		vector<Point2f> good_new;
		float x = 0, y = 0; // Set x and y for new position of ROI
		float anzahl_zuverl = 0;

		for (uint i = 0; i < p1.size(); i++)
		{
			// Select good points of flow vectos from ROI
			if (status[i] == 1) {

				// append p1[i] in good_new 
				good_new.push_back(p1[i]);


				x += p1[i].x - p0[i].x;
				y += p1[i].y - p0[i].y;

				anzahl_zuverl++;

			}

		}

		cout << "x =  " << x << " y =  " << y << endl << endl;
		
		cout << "Number of good_new " << good_new.size() << endl << endl;

		// y s	ave values for export
		frame_ofs << y << endl;


		// Make ROI move direction
		x = x / p1.size();
		y = y / p1.size();
		ROI.x += round(x);
		ROI.y += round(y);
 
		rectangle(frame, ROI, Scalar(255, 0, 255), 2);

		i++;

		imshow("Frame", frame);
		int keyboard = waitKey(30);
		if (keyboard == 'q' || keyboard == 27)
			break;

		// Now update the previous frame and previous points
		old_gray = frame_gray.clone();
		p0 = good_new;

		video_written.write(frame);


	}


}

enter image description here

I dont have any experience in Python, but i found a lot example codes which i am not able to read propperly.

Problem is the opencv doc is just definitions. I changed the values of the parameters a lot but could not rly see a related difference.

Rookie problems guys, sry :stuck_out_tongue:

1 Like