Are you always able to get the rotated rect to work as well as it did in the first picture in your your example? What I mean by that is that the rotated rect seems to enclose the desired area / doesn’t clip anything. It also seems that two of the edges are pretty close to coinciding with edges from the ideal quad, which might be helpful but maybe not necessary for what I’m thinking:
Can you start with an enclosing quad (rotated rectangle is fine, maybe a plain old rect is fine) and adjust the corners in a way that ends up with the quad you want? At least for your example image, the minimum enclosing quad (what I think you were going for with approxPolyDP?) would give you a pretty good transform. If so maybe something like:
- Find enclosing rect.
- Iterate Over the vertices and try to reduce the length of the edges with the constraint that the resulting quad still encloses all the pixels you care about.
- Repeat until you can’t reduce any edge lengths.
This is just off the cuff - I’m not sure if it will work at all, much less in the general case, but it might have legs?
To clarify, I’m thinking you’d start with vertex V0 and try to move it toward V3 (the last vertex in the quad) until doing so would cause the edge (V0,V1) to clip your image. Then you would try you move V0 toward V1 until edge (V3,V0) clips your image. Then you would consider V1, first moving it toward V0, then moving it toward V2, and so on.
I’m sure it’s more complicated than this and there are corner cases I’m not seeing, but maybe this would at least give you a path to try?
Good luck
OK, now that I have had a chance to look at this with paper and pencil, that approach clearly won’t work as stated for the general case (starting with any enclosing quad) , but it might work if the starting quad is constrained somehow. I hold out some hope that a minimum area rotated rect might work, or at least be a better starting point. My intuition is based on the belief that a rotated rect will at have (at least) one side parallel/coincident with one side of the desired quad. It’s just a hunch, but that’s where I’d start if I were working on this problem.