Ok, let me give this a try. I don’t see an “Insert picture” so I assume that I need to use the link insert.
Here is a screen capture of the 2nd monitor with the chessboard displayed. Monitor2 is 1280 X 270 and the Mat mToScreenChessboard has been resized to this same resolution and show FullScreen.
Mat of chessboard shown on 2nd monitor
Here is what that 2nd monitor with the chessboard looks like in the cameraWindow Camera view of chessboard
Once the Homography is done, for testing purposes, I put a spreadsheet on the 2nd monitor with the cells sized at 1/4" X 1/4" and maximized. The dark cell is, give or take, the center of the spreadsheet. What the program will be doing is sending each frame from the camera stream to SimpleBlobDetector to look for a laser hit on the screen. When detected, that will be the keypoints. The program will then take the keypoint on the cameraWindow and, using the Homograph created, determine where this same point is with respect to the Mat mToScreenChessboard and then simulate a mouse click at that point.
At this stage of testing the Mat of the chessboard is on a monitor so there is no complication of a projector screen being skewed. Since the Mat is 1280 X 720 and shown full screen on the 2nd monitor which is also 1280 X 720, then I would assume that what is on the 2nd monitor and the actual Mat should match up more or less 1 to 1. Here is the spreadsheet on the 2nd monitor Spreadsheet shown maximized on 2nd monitor
Now I shoot a laser onto the 2nd monitor more or less in the center. Here is the camera view of that hit. The actual laser hit is surrounded by a black circle drawn around the keypoint detected. So you can see that the keypoint is accurate. What you can also see in this camera view is the result of the simulated mouse click on the 2nd screen. The Homographed keypoint will be the tip of the Arrow Cursor and the spreadsheet cell under that point will activate with a black box around it. In this hit in the center of the screen you will see that the mouse arrow cursor point is right behind the laser hit showing that the Homograph translated accurately to the 2nd monitor. Camera view of laser hit center of 2nd monitor (Camera has been dimmed to help detect laser hits.)
Here is how that Homographed point on the 2nd monitor is ued to simulate a mouse click. For testing purposes, the 2nd monitor has the same resolution as the primary monitor and, in this case, is an extension of the primary and to the right. Sx and Sy are the Homographed keypoint for the screen (2nd monitor).
INPUT Inputs[3] = { 0 };
Inputs[0].type = INPUT_MOUSE;
Inputs[0].mi.time = 0;
Inputs[0].mi.mouseData = 0; // Mouse Wheel movement is 0
// The Sx and Sy have been moved from the Mat 0,0,1280,720 to the corresponding point on the the 2nd monitor which as a starting x of 1280
// Get the sysmetrics of this point
Inputs[0].mi.dx = (float)Sx * (65536.0f / GetSystemMetrics(SM_CXVIRTUALSCREEN)); // desired X coordinate on 2nd monitor
Inputs[0].mi.dy = (float)Sy * (65536.0f / GetSystemMetrics(SM_CYVIRTUALSCREEN)); // desired Y coordinate on 2nd monitor
// Move the mouse to the desired point on the 2nd monitor
Inputs[0].mi.dwFlags = MOUSEEVENTF_ABSOLUTE |
MOUSEEVENTF_VIRTUALDESK | MOUSEEVENTF_MOVE;
// Now left mouse down and then up to make a click
Inputs[1].type = INPUT_MOUSE;
Inputs[1].mi.dwFlags = MOUSEEVENTF_LEFTDOWN;
Inputs[2].type = INPUT_MOUSE;
Inputs[2].mi.dwFlags = MOUSEEVENTF_LEFTUP;
// Now send all 3 inputs
SendInput(3, Inputs, sizeof(INPUT));
Here now are camera views of the next shot more to the left and then another even more to the left. You will notice that as the shots go left the tip of the arrow cursor is more and more to the right of the actual laser hit
Laser hit left of center
Laser hit at left of screen
Now here is a hit on the right of screen to show how the inaccuracy is not a matter of being off center but being to the left.
Laser hit on screen right.
You will see that this particular camera has a slight fish eye effect. I have assumed that the Homography will compensate for this but as soon as I finish with this post I will run the same test with a different camera that does not have the slight fish eye effect. I will let you know how that turn out.
Any opinions or guesses of what I can modify to see if it fixes this would be appreciated.
Ed