Imshow not working on iOS Chrome/Safari in opencv.js

, ,
1

I have a website that grabs frames from a user’s webcam via an HTML <video> element, copies them to a <canvas> element, and displays the canvas element using opencv.js cv.imshow(). The frame is also copied to an off-screen canvas to be sent to the server for some processing, this behaves as expected and returns the expected result.

The site functions as expected on Chrome (and derivations of Chrome such as Brave) on desktop (Linux) and Android Chrome, but for some reason the canvas shows as plain white (the frames from the video are not displayed) on iOS Chrome and Safari. This confirms that the frames from the video element and the user’s camera are captured correctly.

I have used the following functions to check that 1) the canvas is not blank (also confirmed by the fact that it works on Android Chrome, 2) the canvas is not hidden or otherwise not being displayed:

To check if the canvas is blank:

// returns true if all color channels in each pixel are 0 (or "blank")
function isCanvasBlank(canvas) {
    return !canvas.getContext('2d')
      .getImageData(0, 0, canvas.width, canvas.height).data
      .some(channel => channel !== 0);
  }

To check if the element is being displayed:

const observer = new IntersectionObserver((entries) => {
    if(entries[0].isIntersecting){
        // el is visible
        console.log('canvas_visible', true);
    } else {
        // el is not visible
        console.log('canvas_visible', false);
    }
});
observer.observe(canvas);

The output on chrome://inspector on iOS shows is_canvas_blank false and canvas_visible true after each frame.

Strangely, when I run the opencv.js tutorial on iOS chrome the frames are correctly displayed on the canvas (opencv.js videocapture tutorial). I implemented my code exactly as this tutorial describes (see below for minimal implementation).

HTML

<div id="localVideo">
    <video id="video"></video>
    <canvas id="canvas" height="480px" width="640px"></canvas>
    <canvas id="hiddenCanvas" height="480px" width="640px"></canvas>
</div>

JS

let video = document.getElementById('video');
let cap = new cv.VideoCapture(video);
let src = new cv.Mat(video.videoHeight, video.videoWidth, cv.CV_8UC4);
let dst = new cv.Mat(video.videoHeight, video.videoWidth, cv.CV_8UC1);

const FPS = 15;
function processVideo() {
    try {
        if (!streaming) {
            // clean and stop.
            delete cap;
            src.delete();
            dst.delete();
        }
        cap.read(src);
        cv.cvtColor(src, dst, cv.COLOR_RGBA2RGB);

        cv.imshow(canvas, dst);    
        cv.imshow(hiddenCanvas, src);

        let delay = 1000 / FPS;
        setTimeout(processVideo, delay);
    } catch (err) {
        console.log(err);
    }
}
// schedule the first one.
setTimeout(processVideo(), 0);

The HTML <video> element also correctly displays the video on iOS Chrome too.

It would be extremely helpful if you could explain why this is only an issue on iOS Chrome/Safari, why the tutorial site works but my copy doesn’t, and/or if I’ve missed something. Many thanks.

2

The problem turned out to be a CSS transform that was applied to the canvas to flip the frame along the vertical axis. For some reason, Android Chrome was able to handle this, but iOS was not.

I was able to resolve the issue by removing the transform and mirroring the frame using cv.flip() instead.

1 Like