Resizing/scale rectangle

Hello,

I am doing something that I almost seem to have correct but its slightly off enough that I’m missing something easy. I’ve got these very large frames from a 4K video that I am resizing to perform operations on and I end up with a rectangle from the resized/smaller frames. My thought process is that scaling up and down the image is probably damaging to the image in that its adds distortions/noise/whatever, so I would like to take the rectangle from the resized smaller frame and expand it to the original frame.

So, for instance imagine I take a frame from a video and resize it to 50% of the original size and then perform face detection upon it, then I’d like to take the resulting rectangle from face detection and expand and reposition it to the original frame, how do I go about doing that?

Currently, I am resizing it by multiplying the ROI height/width by 0.5 and then modifying the ROI x,y coordinates by taking the difference between the original width/height with the resized width/height divided in half-- which is almost right, but leaves me a bit off-- what am I doing wrong here?

Thanks

Nathan

you divided by two, right? then multiply by two.

show what’s going on. narration isn’t debuggable.

Shortly after I wrote the post I realised what my problem was and solved it. Thanks none the less however.

For anyone else that faces similar what I did was take the ROI from the smaller resized image and calculate the percent that the x,y coordinates are into the image, then calculated the numeric value the same percent is into the original larger frame and adjusted the x,y coordinates for the ROI accordingly.

e.g.

// roi.y/x is what percent of frame.rows/cols
// offset_to_percent = 100 * offset / total
std::size_t y_percent(offset_to_percent(roi.y, frame.rows));
std::size_t x_percent(offset_to_percent(roi.x, frame.cols));

// y_percent/x_percent is what number of new_frame.rows/cols
// percent / 100 * total
y_percent = percent_to_offset(y_percent, old_frame.rows);
x_percent = percent_to_offset(x_percent, old_frame.cols);

roi.y = y_percent;
roi.x = x_percent;

edit actually this seems to be still slightly off, but it appears to be an issue with the height/width and the positioning is correct. I’m just not thinking correctly-- the frame size is ~50% smaller, so increasing the ROI height/width by 50% seems like it would work but its not. I’ll figure it out and post a new comment when I do for anyone else trying to do the same thing.

Solved the problem, apparently when increasing the height/width of the rectangle I need to find the percentage difference, convert it to pixels size and add the squared size to height/width.

For anyone else doing similar, the code looks roughly like this-- “f” is the original 4K video frame and “frame” is the resized smaller version. “ROI” is the rectangle for the smaller video frame.

/* percent_difference = 100 * (larger - smaller) / larger
* number_to_percent = 100 * offset / total
* percent_to_number = percent / 100 * total
*/
		if (frame.size() != f->size()) {
			std::size_t	height_diff(percent_difference(f->size().height, frame.size().height));
			std::size_t	width_diff(percent_difference(f->size().width, frame.size().width));
			std::size_t	y_percent(number_to_percent(roi.y, frame.rows));
			std::size_t	x_percent(number_to_percent(roi.x, frame.cols));

			height_diff   = percent_to_number(height_diff, roi.height);
			width_diff	= percent_to_number(width_diff, roi.width);
			y_percent     = percent_to_number(y_percent, frame.rows);
			x_percent     = percent_to_number(x_percent, frame.cols);

			roi.y		= y_percent;
			roi.x		= x_percent;
			roi.height	+= (height_diff * height_diff);
			roi.width	+= (width_diff * width_diff);

			/* There is an assertion inside of opencv which will crash the program 
			 * if we don't reproduce handle it here. 
			 */
			if (0 > roi.x || 0 > roi.width || roi.x + roi.width > f->cols ||
				0 > roi.y || 0 > roi.height || roi.y + roi.height > f->rows) {
				qDebug() << "ASSERTION FAILED";
				return new cv::Mat(f->clone());
			}
			
			frame = (*f)(roi).clone();

that isn’t helpful without the source to those utility functions you use, like height_diff and so on.

I also doubt there needs to be any squaring, if all you do is scale the image by a factor.

that isn’t helpful without the source to those utility functions you use, like height_diff and so on.

They’re simple inline one-liners for the most part whose functionality I defined in the comments:

/* percent_difference = 100 * (larger - smaller) / larger
* number_to_percent = 100 * offset / total
* percent_to_number = percent / 100 * total
*/

I think anyone having the same problem would pretty quickly figure out what was intended, but they’re included below:

		inline std::size_t 
		number_to_percent(const std::size_t offset, const std::size_t total)
		{
			return std::size_t(std::round(100.0f * offset / total));
		}

		inline std::size_t
		percent_to_number(const std::size_t percent, const std::size_t total)
		{
			return std::size_t(std::round(percent / 100.0f * total));
		}

		inline std::size_t
		percent_difference(const std::size_t larger, const std::size_t smaller)
		{
			const std::size_t difference(larger - smaller);
			return std::size_t(std::round(100.0f * difference / larger));
		}

I also doubt there needs to be any squaring, if all you do is scale the image by a factor.

I haven’t overly thought about it, but simply increasing the height/width by the same size as the reduction yielded a ROI that was too small, squaring the increments matched the ROI perfectly.

At any rate, I solved my problem, thanks!

Excuse me, apparently I shouldn’t talk when I’m tired-- I tested it, then cleaned up the code some and pasted the cleaned up code without testing it.

In the original tested/working variant I typed:

			roi.height	+= (height_diff * 2);
			roi.width	+= (width_diff * 2);

which my brain translated to “squared”, which is incorrect. I’m increasing the perimeter of the rectangle which is 2(l+w) and not l^2+w^2. So the lines above are the actual correct solution not the lines that square the new height/width.

Additionally, I removed a redundant copy of a cv::Mat without testing it and it broke the positioning because of the second assignment to y_percent/x_percent operating on the wrong rows/cols. They should read:

			y_percent     = percent_to_number(y_percent, f->rows);
			x_percent     = percent_to_number(x_percent, f->cols);

And just for clarity, the entire section of code that is tested and working is:

		std::size_t	height_diff(percent_difference(f->size().height, frame.size().height));
		std::size_t	width_diff(percent_difference(f->size().width, frame.size().width));
		std::size_t	y_percent(number_to_percent(roi.y, frame.rows));
		std::size_t	x_percent(number_to_percent(roi.x, frame.cols));

		height_diff = percent_to_number(height_diff, roi.height);
		width_diff	= percent_to_number(width_diff, roi.width);

		y_percent	= percent_to_number(y_percent, f->rows);
		x_percent	= percent_to_number(x_percent, f->cols);

		roi.y		= y_percent;
		roi.x		= x_percent;
		roi.height	+= (height_diff * 2);
		roi.width	+= (width_diff * 2);

		/* There is an assertion inside of opencv which will crash the program 
		 * if we don't handle it here. 
		 */
		if (0 > roi.x || 0 > roi.width || roi.x + roi.width > f->cols ||
			0 > roi.y || 0 > roi.height || roi.y + roi.height > f->rows) {
			qDebug() << "ASSERTION FAILED";
			return new cv::Mat(f->clone());
		}
			
		frame = (*f)(roi).clone();

Basically, you need to reposition the xy coordinates in absolute terms and not relative to its current position (although there is probably a way to do it relative to its current xy) and then you need to increase the perimeter of the rectangle, which is 2(length+width).

If you copy/paste, be mindful that there are no sanity checks (e.g. that larger is actually larger) there as in my exact circumstances was impossible.

Excuse me, apparently I shouldn’t type/explain myself while tired. I wrote the code and tested it, then cleaned it up before posting but didn’t test that. As a result a couple of errors creeped in.

Firstly, you’re right, there is no need to square-- in the original code I typed:

			roi.height	+= (height_diff * 2);
			roi.width	+= (width_diff * 2);

But my brain translated that as “squared” when that is the wrong answer. The perimeter of a rectangle is defined as 2(length+width), so that’s what we’re doing here and that (not squared) yields the desired outcome.

Secondly, when I cleaned up the code I removed a superfluous copy of a cv::Mat that I was operating on for testing and when I removed it I broke the positioning code. The second assignment to y_percent/x_percent should operate on the larger video frame, not the smaller one thus this is correct:

			y_percent     = percent_to_number(y_percent, f->rows);
			x_percent     = percent_to_number(x_percent, f->cols);

And while the problem is simplistic enough that I wouldn’t copy/paste, the entire tested/working solution is as follows:

std::size_t	height_diff(percent_difference(f->size().height, frame.size().height));
std::size_t	width_diff(percent_difference(f->size().width, frame.size().width));
std::size_t	y_percent(number_to_percent(roi.y, frame.rows));
std::size_t	x_percent(number_to_percent(roi.x, frame.cols));

height_diff = percent_to_number(height_diff, roi.height);
width_diff	= percent_to_number(width_diff, roi.width);

y_percent	= percent_to_number(y_percent, f->rows);
x_percent	= percent_to_number(x_percent, f->cols);

roi.y		= y_percent;
roi.x		= x_percent;
roi.height	+= (height_diff * 2);
roi.width	+= (width_diff * 2);

/* There is an assertion inside of opencv which will crash the program 
 * if we don't handle it here. 
 */
if (0 > roi.x || 0 > roi.width || roi.x + roi.width > f->cols ||
	0 > roi.y || 0 > roi.height || roi.y + roi.height > f->rows) {
		qDebug() << "ASSERTION FAILED";
		return new cv::Mat(f->clone());
}
			
frame = (*f)(roi).clone();

Thanks!

Actually, apparently I shouldn’t explain myself/code while tired. I wrote the code, tested it and then cleaned it up and posted the cleaned up variant without testing it. This resulted in a couple of bugs.

Firstly, you’re right, it doesn’t need to be squared but rather each additive to height/width needs to be multiplied by 2 (perimeter of a rectangle is 2(length+width)), which my brain translated into “squared” and is incorrect. Secondly, I had been working on a temporary cv::Mat for testing which I removed and then fixed up the code incorrectly resulting in the positioning being off. The second instance of assignments to x_percent/y_percent should operate on f->rows/cols and not frame.rows/cols.

Thus, the entire fixed up and tested solution to my problem was:

std::size_t	height_diff(percent_difference(f->size().height, frame.size().height));
std::size_t	width_diff(percent_difference(f->size().width, frame.size().width));
std::size_t	y_percent(number_to_percent(roi.y, frame.rows));
std::size_t	x_percent(number_to_percent(roi.x, frame.cols));

height_diff = percent_to_number(height_diff, roi.height);
width_diff	= percent_to_number(width_diff, roi.width);

y_percent	= percent_to_number(y_percent, f->rows);
x_percent	= percent_to_number(x_percent, f->cols);

roi.y		= y_percent;
roi.x		= x_percent;
roi.height	+= (height_diff * 2);
roi.width	+= (width_diff * 2);

/* There is an assertion inside of opencv which will crash the program 
 * if we don't handle it here. 
 */
if (0 > roi.x || 0 > roi.width || roi.x + roi.width > f->cols ||
	0 > roi.y || 0 > roi.height || roi.y + roi.height > f->rows) {
		qDebug() << "ASSERTION FAILED";
		return new cv::Mat(f->clone());
}
			
frame = (*f)(roi).clone();

Finally, this is my 3rd time typing up this response and the other two have disappeared for unclear reasons (I deleted one because of buggy syntax highlighting, so technically 4 times). Apparently the thread has been disappeared temporarily under suspicion of spam-- there appears to be some sort of bug relating to editing comments and possibly syntax highlighting.

At any rate, thanks, problem solved.

the factor of 2 is right, if you scaled the picture down by half.

your explanation of why it’s 2 is complete bogus. it has nothing to do with perimeter.

the factor of 2 is right, if you scaled the picture down by half.

your explanation of why it’s 2 is complete bogus. it has nothing to do with perimeter.

While I appreciate your assistance after the fact, can you explain why in geometric terms the multiplication by 2 is requisite? The only reason I can think of relates to the perimeter.

Just for anyone reading this later-- he is correct in that the 2*x is related to the reduction by 50% but rather than sit and wait for this person that has not once been constructive or helpful to find out why, I’ll flip through a geometry book.

For anyone facing a similar problem; the issue was in my comprehension of the geometric aspects of things. If you scale down by 50% and then try to scale back up, you’re not trying to scale by 50% but rather by 200%. Moreover (xw,yh) ends up being the same as (xW,yH)-- this actually makes calculation simpler.

The following I’ve tested on a variety of scales and seems to work (for scaling up). YMMV.

inline double
percent_difference(const std::size_t larger, const std::size_t smaller)
{
	const std::size_t	difference(larger - smaller);
	const double		retval(double(difference) / double(larger));

	return retval;
}

inline double
scale_factor(const double percent)
{
	return (1.0 / percent);
}

[...]
const double	height_percent(1 - percent_difference(f->size().height, frame.size().height));
const double	width_percent(1 - percent_difference(f->size().width, frame.size().width));

	roi.y		*= scale_factor(height_percent);
	roi.x		*= scale_factor(width_percent); 
	roi.height	*= scale_factor(height_percent);  
	roi.width	*= scale_factor(width_percent); 

	/* There is an assertion inside of opencv which will crash the program 
	 * if we don't handle it here. 
	 */
	if (0 > roi.x || 0 > roi.width || roi.x + roi.width > f->cols ||
		0 > roi.y || 0 > roi.height || roi.y + roi.height > f->rows) {
		qDebug() << "ASSERTION FAILED";
	return new cv::Mat(f->clone());
}
			
frame = (*f)(roi).clone();

I’d also suggest if you have this sort of complication trying out one of the many various math forums where the people trying to help don’t have an inherent conflict of interest in that they’re using their status to try to make money, but again YMMV.