[Opencv.js] Loading tensorflow model into opencv.js

Hi there,
I would like to ask whether there is any possibility to load tensorflow model (frozen_inference_graph.pb and corresponding .pbtxt file) into opencv.js? I’m struggling with it from several days. It appears that there is a problem with cv.blobFromImage function from opencv.js. It expects arguments like std or mean but when using tensorflow model it shouldn’t be necessary to specify them. I don’t know appropriate mean and std values for this type of model. In opencv-python there is a possibility to use cv.blobFromImage without std and mean but I don’t see this option in opencv.js.

please show code, actual errors, tell us what kind of model you’re trying to use, and how it got made, the whole chi-chi, thank you :wink:

This is the way I’m trying to start the video and process first frame. I am getting error when calling net.forward(). I am using model that I’ve found on this github: GitHub - marcomarasca/SDCND-Traffic-Light-Detection: Traffic Light Detection using the tensorflow object detection API. It is working just fine in opencv-python.

  startVideo: async function() {
  this.streaming = true;
  let video = this.$refs.camInput;
    .getUserMedia({ video: true, audio: false })
    .then(function(stream) {
      video.srcObject = stream;
    .catch(err => console.log('Error'));

  let src = new cv.Mat(video.height, video.width, cv.CV_8UC4);
  let cap = new cv.VideoCapture(video);
  const labels = await this.loadLables(this.labelsUrl);

  let net = cv.readNetFromTensorflow(this.modelPath, this.configPath);
      new cv.Size(this.inputSize[0], this.inputSize[1]),
      new cv.Scalar(this.mean[0], this.mean[1], this.mean[2]),
// cv.blobFromImage(
//       src,
   //   new cv.Size(this.inputSize[0], this.inputSize[1]),
  //    this.swapRB,
  //    false
 //   )
//   );
  try {
    const result = net.forward(); //getting error here
    const output = this.postProcess(result, labels, src);
    setTimeout(() => this.processVideo(cap, src, labels, net, video), 0);
  } catch (err) {

The error I get is just a strange number printing out in the console.

1 Like

can you be more concise, please ?

Zrzut ekranu z 2021-05-19 20-19-58

I removed try/catch block to see the exact error. Otherwise it is just “8661016”. This value is not constant (I mean I’m getting many “random” error values).

If someone likes JS as much as I do and does even this sort of stuff like OpenCV in this, here is how I finally imported this tensorflow model into OpenCV.js:

Import net like this:

let net = cv.readNet(this.configPath, this.modelPath);

Proper function for making blob from image: (run it instead of cv.blobFromImage in the code above.

    getBlobFromImage: function(inputSize, mean, std, swapRB, image) {
  let mat;
  if (typeof image === 'string') {
    mat = cv.imread(image);
  } else {
    mat = image;
  let matC3 = new cv.Mat(mat.matSize[0], mat.matSize[1], cv.CV_8UC3);
  cv.cvtColor(mat, matC3, cv.COLOR_RGBA2BGR);
  let input = cv.blobFromImage(
    new cv.Size(inputSize[0], inputSize[1]),
    new cv.Scalar(mean[0], mean[1], mean[2]),
  return input;

Mean values that seem to work: [123, 117, 104]
STD value: 1
swapRB: true