Question on creating a cv::Mat from SL::Screen_Capture::Image

I have a library that takes screen capture data GitHub - smasherprog/screen_capture_lite: cross platform screen/window capturing library

I want to take Screen Capture Lite’s Screen_Capture::Image and translate it to a cv::Mat or std::vector<std::uint8_t> buffer

it sounds like I could use cv::imencode() for this… but having trouble understanding how to switch from encoding files via imread() to this RGB array… ScreenCapture Lite gives an extraction example… but I am unclear how to translate what they’re doing… to what OpenCV wants / needs … so that the client can decode on the other side

void ExtractAndConvertToJPGBuffer(const SL::Screen_Capture::Image &img, std::vector<std::uint8_t> *dst, size_t dst_size)
  assert(dst_size >= static_cast<size_t>(SL::Screen_Capture::Width(img) * SL::Screen_Capture::Height(img) * sizeof(SL::Screen_Capture::ImageBGRA)));
  auto imgsrc = StartSrc(img);
  auto size = Width(img) * Height(img) * sizeof(SL::Screen_Capture::ImageBGRA);
  auto imgbuffer(std::make_unique<unsigned char[]>(size));
  auto imgdist = unsigned char *dst;
  if (img.isContiguous) {
      memcpy(imgdist, imgsrc, dst_size);
  else { 
    for (auto h = 0; h < Height(img); h++) {
        auto startimgsrc = imgsrc;
        for (auto w = 0; w < Width(img); w++) {
            *imgdist++ = imgsrc->R;
            *imgdist++ = imgsrc->G;
            *imgdist++ = imgsrc->B;
        imgsrc = SL::Screen_Capture::GotoNextRow(img, startimgsrc);
  cv::Mat image;
  std::vector<int> param(2);
  cv::imencode(".jpg", imgsrc, &dst, param);

For those curious, the Server Client relationship looks like this. I’ve already gotten this to work with file based JPEGS… just now needing to switch from that… to this SL::Screen_Capture::Image. … biggest clue I have so far is it sounds quite close to InputArray … but one seems to be an unsigned char the other is unclear, their example uses auto which hides what data type this is



std::vector<std::uint8_t> buff(header_data.image_size_bytes);
boost::asio::read(socket, boost::asio::buffer(buff), boost::asio::transfer_exactly(header_data.image_size_bytes), ignored_error);

cv::Mat img(cv::imdecode(buff, cv::IMREAD_ANYCOLOR));


void transmitsocketdata(tcp::socket & socket, const SL::Screen_Capture::Image & screen_cap_img){

  boost::system::error_code ignored_error;
  std::vector<std::uint8_t> buff;

  ExtractAndConvertToJPGBuffer(screen_cap_img, &buff)
  // now we send the header message
  std::string image_dimensions = std::to_string(Width(screen_cap_img)) + "Wx" + std::to_string(Height(screen_cap_img)) + "H";
  std::string image_buff_bytes = std::to_string(buff.size());
  std::string message_header = image_dimensions + "," +  image_buff_bytes;

  message_header.append(63 - message_header.length(), ' ');
  message_header = message_header + '\0';
  std::cout << "sending message header == " << message_header << " ...." << std::endl;
  std::cout << "sending message header of " << std::to_string(message_header.length()) << " bytes...." << std::endl;
  socket.write_some(boost::asio::buffer(message_header), ignored_error);
  std::cout << "sending image message of " << image_buff_bytes << " bytes...." << std::endl;
  socket.write_some(boost::asio::buffer(buff), ignored_error);

no. don’t do any of that. totally wrong idea. no.

what you need to do is use the cv::Mat constructor that takes a data pointer.

I repeat, DO NOT!!! call imencode. that’s for compressing an image and keeping the encoded binary data, which belongs in a file, in memory instead.

one would call imencode to compress the image and maybe send it across the network.

for just turning some random bitmap data from that library into a cv::Mat, you DO NOT call that.

I hope you understand the difference between a bitmap image that can be worked with, and compressed data. it’s the difference between a text file and a zip containing a text file.

Hi Crackwitz, thank you…

I am sending data across the network, maybe that wasn’t apparent in the post… I didn’t want to inundate readers but… I am indeed sending the data over a Boost backed network socket… so IMEncode does serve a purpose here.

My plan was first to , as you said

" use the cv::Mat constructor that takes a data pointer."

Once a cv::Mat is instantiated, I want to run it through imencode… it is here I can apply not just the switch to JPEG format… but apply compression %'s if/when network traffic requires it.

I hope this clears up why imencode is involved… I will next attempt to translate the bitmap of this library … and add it into the cv::Mat … then send it without imencode over the wire… but then I will need to modify the client slightly to stop using imdecode on the other side.

seems i need to read up and lead how to place the data into this area

or for the version I use

i’d say: throw away your “broken beyond repair” ExtractAndConvertToJPGBuffer()

and use the supplied conversion here:

as in:

int W = Width(img);
int H = Height(img);
Mat rgba(H,W,CV_8UC4); // preallocate pixels
Mat bgr; cvtColor(rgba, bgr, COLOR_RGBA2BGR);

vector<uchar> encoded;
imencode(".jpg", bgr, encoded);

@berak I won’t be offended by your assessment of my code… and I just wanted to convey a big thanks to you and the mods… I now have something !

I hope to button it up a bit but… will hope to share this out in a talk coming in April… big thanks

1 Like