I have a video stream I’m reading with the help of cv::VideoCapture. I then send each frame individually to a network client by doing this:
const size_t bytes_len = frame.rows * frame.cols * 3;
const uint8_t * bytes_ptr = frame.data;
socket.send(boost::asio::buffer(bytes_ptr, bytes_len), 0, ec);
Then on the receiving end of that socket, I rebuild the frame by creating a cv::Mat of the correct dimensions and type (CV_8UC3), and then reading from the socket directly into mat.data.
This almost works. I’m getting strange results where the video is split into…tiles, and the colours are off. The way all parts are there but not in the right place makes me think there is some subtle or important piece of information I’m missing about how data is stored in a cv::Mat.
Example frame, where the original video on the left is after reading individual frames, and the reconstructed video is on the right:
I understand this is not a OpenCV problem but a bug in my code, I’m just hoping someone might have some additional insight into cv::Mat that will shed some light onto what could be causing this problem.
that’s a shift. you must be missing some data, or unrelated data was introduced into the stream.
remember, reading from a socket may return less data than requested. you have to handle that. my bet is that’s the issue.
I came to the same conclusion (shift) since I originally wrote the post. Thanks for the confirmation. Boost ASIO is nice, but async communication can quickly turn into a mess.
The way I confirmed it was a shift issue is just before sending the frame, I wiped out the entire mat and replaced it all with a single digit:
static double colour = 0.0;
frame = cv::Scalar(colour, colour, colour);
colour += 1.0;
I increment the colour value for every frame. So I can see the outgoing mat is [0, 0, 0, 0, …] followed by [1, 1, 1, 1, …] but on the receiving end, I’m not seeing the same thing. Somehow I’m consuming more bytes than expected which results in slightly lower frame rate and the image getting shifted.
As suspected, this had nothing to do with my misunderstanding of OpenCV cv::Mat, but was caused by Boost ASIO. Turns out boost::asio::ip::tcp::socket::send() has this notice in the documentation:
“The send operation may not transmit all of the data to the peer. Consider using the write function if you need to ensure that all data is written before the blocking operation completes.”
I swapped the send() for write() and the OpenCV shift problem has gone away.
1 Like