Camera2 byteArray to BGR uint8 mat conversion

Haider_Saleem · August 12, 2022, 6:46am

I’m trying to convert byteArray from camera2 onImageAvailable listener to Mat object, and then passing it to an algorithm for dehazing in c++. I have tried different methods available for converting byteArray to Mat channel 3 object but whenever I split the mat object in 3 channels then all of them get filled with garbage data which further cause crash.

Here’re the different methods used for converting byte array to mat

    val bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
    val orig = Mat(bmp.height, bmp.width, CvType.CV_8UC3)
    val myBitmap32 = bmp.copy(Bitmap.Config.ARGB_8888, true)
    Utils.bitmapToMat(myBitmap32,orig)

Using imread

        val matImage = Imgcodecs.imread(file!!.absolutePath,IMREAD_COLOR)

using decodebyte array

val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
Utils.bitmapToMat(bitmap,matImage)

here’s c code for the JNICALL

dehazing(JNIEnv *env, jobject, jlong input, jlong output)
{
    cv::Mat& mInput = *((cv::Mat*) input);
    cv::Mat& mOutput = *((cv::Mat*) output);
    dehaze(mInput, mOutput);
}

and finally here’s a little chunk of c++ code

Mat dehazedSplit[3];
Mat ABGR[3];
Mat tBGR[3];
Mat imageBGR[3];
Mat blurredImageBGR[3];
// Normalize, blur image and extract dimensions
W = image.cols;
H = image.rows;

image.convertTo(image, CV_64FC3);
image = image / Scalar(255, 255, 255);

GaussianBlur(image, blurredImage, Size(41, 41), 30, 30);
split(blurredImage, blurredImageBGR);

// Estimate the A matrix
A = Mat(H, W, CV_64FC3, Scalar(1, 1, 1));

split(A, ABGR);

minMaxLoc(blurredImageBGR[0], &minVal, &maxVal, &minLoc, &maxLoc);
ABGR[0] = Scalar(maxVal);
minMaxLoc(blurredImageBGR[1], &minVal, &maxVal, &minLoc, &maxLoc);
ABGR[1] = Scalar(maxVal);
minMaxLoc(blurredImageBGR[2], &minVal, &maxVal, &minLoc, &maxLoc);
ABGR[2] = Scalar(maxVal);

// Estimate the t matrix
t = Mat(H, W, CV_64FC3, Scalar(0, 0, 0));
split(t, tBGR);

tBGR[0] = (Scalar(1) - blurredImageBGR[0].mul(Scalar(twBlue)) / ABGR[0]);
tBGR[1] = (Scalar(1) - blurredImageBGR[1].mul(Scalar(twGreen)) / ABGR[1]);
tBGR[2] = (Scalar(1) - blurredImageBGR[2].mul(Scalar(twRed)) / ABGR[2]);

Any suggestion/help will be appreciated.

berak · August 12, 2022, 8:24am

i see various problems here:

ARGB has 4 channels, it wont fit into CV_8U3, so (i guess), it will ignore your allocation, and return a 4 channel Mat !

then, later:

again, input is ARGB, but blurredImageBGR has only 3 elems.
undefined behaviour, nasal demons, segfaults, anything can happen

blurredImageBGR[0] is A, not B or R, right ? you have to adapt the indexing

Haider_Saleem · August 12, 2022, 10:29am

So issue starts from here

val orig = Mat(bmp.height, bmp.width, CvType.CV_8UC3)
    val myBitmap32 = bmp.copy(Bitmap.Config.ARGB_8888, true)

what could be an alternate solution for converting it to BRG then?

and for remaining c++, its actually written by c++ dev of our team and according to him it works fine as expected.

berak · August 13, 2022, 8:16am

so, problem is, you have an argb image on the java side and an expected bgr Mat on the c++ side.

maybe you can use mixChannels():


Mat argb2bgr(const Mat &m) {
	Mat _r(m.rows, m.cols, CV_8UC3);
	mixChannels({m}, {_r}, {1,2, 2,1, 3,0}); // drop a, swap r,b
    return _r;
}

you also dont need to split anything into channels:


Mat dehaze(const Mat &argb, Scalar tw=Scalar(0.8, 0.8, 0.5)) {
    // convert to rgb, [0...1]
	Mat image = argb2bgr(argb);
	image.convertTo(image, CV_64F, 1.0/255);

	GaussianBlur(image, blurred, Size(41, 41), 30, 30);
    
    // each channel on its own row
    Mat channels = blurred.reshape(1,blurred.total()); 

    // check for max along x axis
    Mat maxVals;
    reduce(channels, maxVals, 0, REDUCE_MAX, image.depth());

    Mat tbgr = (Scalar::all(1.0) - blurred.mul(tw)) / maxVals;
}

btw, do you have a link to the algorithm you try to implement here ?

Haider_Saleem · August 15, 2022, 6:18am

I’ll try to implement your solution, and I guess the algo was written by developer himself.

Here’s complete source code.

int dehaze(Mat& image, Mat& result, double twBlue = 0.5, double twGreen = 0.5, double twRed = 0.1) {
    // MinMax function input placeholders
    double minVal;
    double maxVal;
    Point minLoc;
    Point maxLoc;
    // Image size placeholders
    int W;
    int H;
    // tresholding/rounding/clipping loop placeholders
    double valueBlue;
    double valueGreen;
    double valueRed;
    // intemediate array structure placehodlders
    Mat blurredImage;
    Mat A;
    Mat t;
    Mat dehazed;
    Mat dehazedSplit[3];
    Mat ABGR[3];
    Mat tBGR[3];
    Mat imageBGR[3];
    Mat blurredImageBGR[3];
    // Normalize, blur image and extract dimensions
    W = image.cols;
    H = image.rows;
    image.convertTo(image, CV_64FC3);
    image = image / Scalar(255, 255, 255);
    GaussianBlur(image, blurredImage, Size(41, 41), 30, 30);
    split(blurredImage, blurredImageBGR);

    // Estimate the A matrix
    A = Mat(H, W, CV_64FC3, Scalar(1, 1, 1));

    split(A, ABGR);
    minMaxLoc(blurredImageBGR[0], &minVal, &maxVal, &minLoc, &maxLoc);
    ABGR[0] = Scalar(maxVal);
    minMaxLoc(blurredImageBGR[1], &minVal, &maxVal, &minLoc, &maxLoc);
    ABGR[1] = Scalar(maxVal);
    minMaxLoc(blurredImageBGR[2], &minVal, &maxVal, &minLoc, &maxLoc);
    ABGR[2] = Scalar(maxVal);

    // Estimate the t matrix
    t = Mat(H, W, CV_64FC3, Scalar(0, 0, 0));
    split(t, tBGR);

    tBGR[0] = (Scalar(1) - blurredImageBGR[0].mul(Scalar(twBlue)) / ABGR[0]);
    tBGR[1] = (Scalar(1) - blurredImageBGR[1].mul(Scalar(twGreen)) / ABGR[1]);
    tBGR[2] = (Scalar(1) - blurredImageBGR[2].mul(Scalar(twRed)) / ABGR[2]);

    merge(tBGR, 3, t);
    merge(ABGR, 3, A);
    t.setTo(0.1, t < 0);

    // Estimate deahazed image
    dehazed = (image - A) / t + A;
    dehazed = dehazed.mul(Scalar(255, 255, 255));
    split(dehazed, dehazedSplit);
    split(image, imageBGR);

    // Clip and normalize image values
    for (int i = 0; i < H; i++) {
        for (int j = 0; j < W; j++) {
            valueBlue = abs(dehazedSplit[0].at<double>(i, j));
            valueGreen = abs(dehazedSplit[1].at<double>(i, j));
            valueRed = abs(dehazedSplit[2].at<double>(i, j));

            if (valueBlue > 255)
                valueBlue = 255;
            if (valueGreen > 255)
                valueGreen = 255;
            if (valueRed > 255)
                valueRed = 255;

            dehazedSplit[0].at<double>(i, j) = round(valueBlue);
            dehazedSplit[1].at<double>(i, j) = round(valueGreen);
            dehazedSplit[2].at<double>(i, j) = round(valueRed);
        }
    }

    merge(dehazedSplit, 3, dehazed);
    dehazed.convertTo(dehazed, CV_8UC3);
    result = dehazed;

    return 0;
}

Haider_Saleem · August 15, 2022, 8:40am

I implemented this code and it seems to be working with the provided algo, but only with high ram phones else causing heap issue on 4gb ram devices.


Mat argb2bgr(const Mat &m) {
	Mat _r(m.rows, m.cols, CV_8UC3);
	mixChannels({m}, {_r}, {1,2, 2,1, 3,0}); // drop a, swap r,b
    return _r;
}

and there’s another issue, the output mat when converted to bitmap, shows a red shade.

Here’s the output image.

berak · August 15, 2022, 8:45am

so, how do you convert it back ? (bgr vs rgb issue ?)

Haider_Saleem · August 15, 2022, 8:53am

I’m creating a bitmap like this

val outBmp = createBitmap(matOutput.width(),matOutput.height(),Bitmap.Config.RGB_565)

and then converting using

        Utils.matToBitmap(matOutput,outBmp)

berak · August 15, 2022, 8:57am

try to convert to rgb before:

Imgproc.cvtColor(matOutput, matOutput, COLOR_BGR2RGB);

Haider_Saleem · August 15, 2022, 9:14am

So, I did this

        Imgproc.cvtColor(matOutput, matOutput, COLOR_BGR2RGB)
        val bmpOut = createBitmap(matOutput.width(), matOutput.height(), Bitmap.Config.RGB_565)

        Utils.matToBitmap(matOutput,bmpOut)

and now output is in blue shade

Haider_Saleem · August 16, 2022, 12:46pm

Thanks Berak for all your help, dropping last channel worked for me and I guess while converting bitmap to mat it formed RGBA.

The solution:

    mixChannels({m}, {_r}, {0,0, 1,1, 2,2});  // drop a

Topic		Replies	Views
Mat creation from byte buffer noise Android/Java	6	1171	July 27, 2022
Kotlin: error on converting mat to bitmap Android/Java	9	1232	May 31, 2021
Inferencing ONNX model on a RGB image in Android-Java Android/Java dnn , java	7	958	August 10, 2023
Issue with Unwrapping Blob Data into Float Array After Using blobFromImage Android/Java dnn	3	450	March 27, 2024
How to convert a video from third party library into cv::Mat C++ videoio , core	8	1569	December 12, 2021

Camera2 byteArray to BGR uint8 mat conversion

Related topics