Grab raw (2 byte per pixel data)

C++
,
1

Hello!

I have a custom USB camera, which sending 16 bit/pixel 400x400pixel image. I try to make a visible version of the image. I already found a python script which(with minimal modification) is able to show the proper image in grayscale. When i try to recrate the code in c++(for experimenting with the recoloring), After i grab the image check the channels and elem number, I try to read the image as a 2 byte/pixel image for that a read it as a 400x400x2 8bitimage. But I always got the message that the “raw” frame also 400*400pixel.
I currently using this code snippet for grabbing:

int deviceID = 0;             // 0 = open default camera
int apiID = cv::CAP_ANY;      // 0 = autodetect default API
cap.set(cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('Y','1','6',' '));
cap.set(cv::CAP_PROP_CONVERT_RGB, 0);
cap.set(cv::CAP_PROP_FORMAT, -1);
cap.open(deviceID, apiID);
cv::Mat raw(400,400*2,CV_8UC3);

cap.read(raw);
cap.read(frame);

cv::Mat temp_image(szSize.height, szSize.width*2,CV_8UC1,frame.data);
cv::Mat depth_image(szSize.height, szSize.width,CV_16UC1,raw.data);
cout<<"frame o dim: "<<frame.dims<<" type: "<<frame.type()<<" byte elements: "<<frame.elemSize()<<" elements: "<<frame.total()<<" rows: "<<frame.rows<<" cols: "<<frame.cols<<endl;
cout<<"frame r dim: "<<raw.dims<<" type: "<<raw.type()<<" byte elements: "<<raw.elemSize()<<" elements: "<<raw.total()<<" rows: "<<raw.rows<<" cols: "<<raw.cols<<endl;
cout<<"frame t dim: "<<temp_image.dims<<" type: "<<temp_image.type()<<" byte elements: "<<temp_image.elemSize()<<" elements: "<<temp_image.total()<<" rows: "<<temp_image.rows<<" cols: "<<temp_image.cols<<endl;
cout<<"frame m dim: "<<depth_image.dims<<" type: "<<depth_image.type()<<" byte elements: "<<depth_image.elemSize()<<" elements: "<<depth_image.total()<<" rows: "<<depth_image.rows<<" cols: "<<depth_image.cols<<endl;

I am new to OpenCV and the pixel formats, but I want to learn how to handle them. I read multiple post about the topic, but now I am a bit confused. I would be really greatful any kind of help.

2
  • os ? video backend ?
  • please show the console output from this
  • check the return values from cap.get()
    (it might not accept setting the fourcc)
  • preallocating Mat raw is useless, it will get overwritten
    (so dont assume the type to be CV_8UC3)

should be the same

3

Hi! Thanks for the response. Currently i experimenting under windows, and with the DSHOW Api backend. yes you are right preallocating has no effect.
Code input:

cv::Mat raw(400,400*2,CV_8UC3);
cv::Mat rw(400,400,CV_16UC1);
cap.read(rw);
cap.read(raw);
cap.read(frame);
cv::Mat temp_image(szSize.height, szSize.width*2,CV_8UC1,frame.data);
cv::Mat depth_image(szSize.height, szSize.width,CV_16UC1,raw.data);
cv::Mat raw_image(szSize.height, szSize.width,CV_16UC1,rw.data);
cout<<"frame o dim: "<<frame.dims<<" type: "<<frame.type()<<" byte elements: "<<frame.elemSize()<<" elements: "<<frame.total()<<" rows: "<<frame.rows<<" cols: "<<frame.cols<<endl;
cout<<"frame r dim: "<<raw.dims<<" type: "<<raw.type()<<" byte elements: "<<raw.elemSize()<<" elements: "<<raw.total()<<" rows: "<<raw.rows<<" cols: "<<raw.cols<<endl;
cout<<"frame rrdim: "<<rw.dims<<" type: "<<rw.type()<<" byte elements: "<<rw.elemSize()<<" elements: "<<rw.total()<<" rows: "<<rw.rows<<" cols: "<<rw.cols<<endl;
cout<<"frame t dim: "<<temp_image.dims<<" type: "<<temp_image.type()<<" byte elements: "<<temp_image.elemSize()<<" elements: "<<temp_image.total()<<" rows: "<<temp_image.rows<<" cols: "<<temp_image.cols<<endl;
cout<<"frame m dim: "<<depth_image.dims<<" type: "<<depth_image.type()<<" byte elements: "<<depth_image.elemSize()<<" elements: "<<depth_image.total()<<" rows: "<<depth_image.rows<<" cols: "<<depth_image.cols<<endl;

Console output
frame o dim: 2 type: 16 byte elements: 3 elements: 160000 rows: 400 cols: 400
frame r dim: 2 type: 16 byte elements: 3 elements: 160000 rows: 400 cols: 400
frame rrdim: 2 type: 16 byte elements: 3 elements: 160000 rows: 400 cols: 400
frame t dim: 2 type: 0 byte elements: 1 elements: 320000 rows: 400 cols: 800
frame m dim: 2 type: 2 byte elements: 2 elements: 160000 rows: 400 cols: 400

4

means, that none of the flags you tried to set() was accepted,
and that your signal got converted to 3channel 8bit

I already found a python script which(with minimal modification) is able to show the proper image in grayscale.

show us, what that thing does, please

5

The only modification was the bit shifting increased from 3 to 6. And added image saving and text saving lines for understanding what happening. Also tried to recreate the colors at the end of the script, but not much luck.

cap = cv2.VideoCapture(1, cv2.CAP_MSMF)
# set width and height
cols, rows = 400, 400,
cap.set(cv2.CAP_PROP_FRAME_WIDTH, cols)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, rows)
cap.set(cv2.CAP_PROP_FPS, 30)
cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

# Fetch undecoded RAW video streams
cap.set(cv2.CAP_PROP_FORMAT, -1)  # Format of the Mat objects. Set value -1 to fetch 
undecoded RAW video streams (as Mat 8UC1)
while True:
    # Capture frame-by-frame
    ret, frame = cap.read()#read into np array with [1,320000] h*w*2 byte
    #print(frame.shape)
    if not ret:
        break
    # Convert the frame from uint8 elements to big-endian signed int16 format.
    frame = frame.reshape(rows, cols*2) # Reshape to 800*400
    #cv2.imwrite('framereshaped.png', frame)
    frame = frame.astype(np.uint16) # Convert uint8 elements to uint16 elements
    #cv2.imwrite('framecasted.png', frame)
    np.savetxt('test.txt', frame, fmt="%d")
    frame = (frame[:, 0::2] << 8) + frame[:, 1::2]  # Convert from little endian to big endian (apply byte swap), the result is 340x240.
    #cv2.imwrite('framebigendian.png', frame)
    np.savetxt('testendian.txt', frame, fmt="%d")
    frame = frame.view(np.int16)
    np.savetxt('testint16.txt', frame, fmt="%d")
    #cv2.imwrite('frameview16.png', frame)
    # Apply some processing for disapply (this part is just "cosmetics"):
    frame_roi = frame[:, 10:-10]  # Crop 320x240 (the left and right parts are not meant to be displayed).
    # frame_roi = cv2.medianBlur(frame_roi, 3)  # Clean the dead pixels (just for better viewing the image).
    frame_roi = frame_roi << 6  # shift the 6 most left bits
    #cv2.imwrite('frameshifted.png', frame_roi)
    normed = cv2.normalize(frame_roi, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8UC3)  # Convert to uint8 with normalizing (just for viewing the image).
    #cv2.imwrite('framenormed.png', normed)
    gray = cv2.cvtColor(normed, cv2.COLOR_BAYER_RG2RGB)