Mat creation from byte buffer noise

Android/Java
1

Hi all

Using OpenCv 4.5.4 on android.

having issue creating Mat object from byte buffer. Gives me white noise and random black bars. The byte buffer is created straight from an Image object from the camera2 api and I know the image is without issue.

Here is my code for conversion:

        ByteBuffer buffer = image.getPlanes()[0].getBuffer();
        byte[] bytes = new byte[buffer.remaining()];
        buffer.get(bytes);
        Mat mat = new Mat(image.getHeight(),image.getWidth(), CvType.CV_8UC1,buffer);
        mat.put(0,0,bytes);
        return mat;

Image showing typical results of conversion:

noise
noise1280×960 370 KB

I can create the Mat object without issue if I first create a bitmap and use Utils.bitmapToMat() , however this is extremely slow. This will be used in a real time image processing application and this initial conversion is the bottleneck.

If there is a fix, or a better way to do this that would be amazing

Thanks

2

that’s the bytes of a compressed image, interpreted as the bytes of an uncompressed image, which is wrong to do.

show us the input data. what is it, where is it from, what made it, …

3

Thanks for the reply

The input data is from a Image type object from Camera2’s ImageReader.OnImageAvailableListener. So this is straight from the camera, I am not loading any resources prior. The processing is performed on live images.

My image reader code that sits in the on available listener.

                    Image image = reader.acquireNextImage();
                    if (image != null) {
                        analyser.analyseImage(image);
                        image.close();
                    }

Image reader documentation : ImageReader

Image object documentation : Image object

As mentioned the image is fine, I can use Utils.bitmapToMat() to create the mat file without issue. I can perform all the processing I want on these mat files and the output is as expected.

My goal is to get this conversion from Image to Mat in the fastest possible time

Thanks

1 Like
4

thanks for the explanation.
i’ve no idea, why the output looks wrong, but looking at your code,
imo you should set either the buffer or the bytes
(i dont think, you even need those) like:

        ByteBuffer buffer = image.getPlanes()[0].getBuffer();
        Mat mat = new Mat(image.getHeight(),image.getWidth(), CvType.CV_8UC1,buffer);

however, this is only the 1st channel (R ? B ?). is this intended ?
how many planes are there ?
and what is the dataspace or ImageFormat of the Image ? are you sure, the buffer contains uncompressed pixels ?
can you please also show the Bitmap related code ?

5

maybe the data is JPEG/MJPEG… or the memory is uninitialized/garbage.

Either way, it’s an Android issue. You should look for answers relating to these APIs on an Android forum. They know what’s going on with these Android APIs.

From the OpenCV side, this can’t be fixed.

6

Ahhh so thats the issue! Yes the Image object is indeed in JPEG format, so I would presume that they are indeed already compressed. I didn’t realise it needed uncompressed values to create the mat file this way.

Berak there is only a single plane when the image reader is set to JPEG format. Also the dataspace function is only available in the latest beta of android and I unfortunately can’t yet use it.

Are there any other ways of creating a Mat file? Ideally faster than my current option which is :

        ByteBuffer buffer = image.getPlanes()[0].getBuffer();
        byte[] bytes = new byte[buffer.remaining()];
        buffer.get(bytes);

        // around 20ms to perform this decode function
        Bitmap bitmap = BitmapFactory.decodeByteArray(bytes,0, bytes.length);
        Mat mat = new Mat();
        Utils.bitmapToMat(bitmap, mat);

Thank you as well

1 Like
7

you could try to imdecode it from the bytes,
maybe it’s faster than the builtin android function