[Opencv.js] Loading tensorflow model into opencv.js

, ,
1

Hi there,
I would like to ask whether there is any possibility to load tensorflow model (frozen_inference_graph.pb and corresponding .pbtxt file) into opencv.js? I’m struggling with it from several days. It appears that there is a problem with cv.blobFromImage function from opencv.js. It expects arguments like std or mean but when using tensorflow model it shouldn’t be necessary to specify them. I don’t know appropriate mean and std values for this type of model. In opencv-python there is a possibility to use cv.blobFromImage without std and mean but I don’t see this option in opencv.js.

2

please show code, actual errors, tell us what kind of model you’re trying to use, and how it got made, the whole chi-chi, thank you :wink:

3

This is the way I’m trying to start the video and process first frame. I am getting error when calling net.forward(). I am using model that I’ve found on this github: GitHub - marcomarasca/SDCND-Traffic-Light-Detection: Traffic Light Detection using the tensorflow object detection API. It is working just fine in opencv-python.

  startVideo: async function() {
  this.streaming = true;
  let video = this.$refs.camInput;
  navigator.mediaDevices
    .getUserMedia({ video: true, audio: false })
    .then(function(stream) {
      video.srcObject = stream;
      video.play();
    })
    .catch(err => console.log('Error'));

  let src = new cv.Mat(video.height, video.width, cv.CV_8UC4);
  let cap = new cv.VideoCapture(video);
  const labels = await this.loadLables(this.labelsUrl);

  let net = cv.readNetFromTensorflow(this.modelPath, this.configPath);
  net.setInput(
    cv.blobFromImage(
      src,
      this.std,
      new cv.Size(this.inputSize[0], this.inputSize[1]),
      new cv.Scalar(this.mean[0], this.mean[1], this.mean[2]),
      this.swapRB,
      false
    )
  );
// cv.blobFromImage(
//       src,
   //   new cv.Size(this.inputSize[0], this.inputSize[1]),
  //    this.swapRB,
  //    false
 //   )
//   );
  try {
    const result = net.forward(); //getting error here
    const output = this.postProcess(result, labels, src);
    this.updateResult(output);
    setTimeout(() => this.processVideo(cap, src, labels, net, video), 0);
  } catch (err) {
    console.log(err);
  }

The error I get is just a strange number printing out in the console.

1 Like
4

can you be more concise, please ?

5

Zrzut ekranu z 2021-05-19 20-19-58

I removed try/catch block to see the exact error. Otherwise it is just “8661016”. This value is not constant (I mean I’m getting many “random” error values).

6

If someone likes JS as much as I do and does even this sort of stuff like OpenCV in this, here is how I finally imported this tensorflow model into OpenCV.js:

Import net like this:

let net = cv.readNet(this.configPath, this.modelPath);

Proper function for making blob from image: (run it instead of cv.blobFromImage in the code above.

    getBlobFromImage: function(inputSize, mean, std, swapRB, image) {
  let mat;
  if (typeof image === 'string') {
    mat = cv.imread(image);
  } else {
    mat = image;
  }
  let matC3 = new cv.Mat(mat.matSize[0], mat.matSize[1], cv.CV_8UC3);
  cv.cvtColor(mat, matC3, cv.COLOR_RGBA2BGR);
  let input = cv.blobFromImage(
    matC3,
    std,
    new cv.Size(inputSize[0], inputSize[1]),
    new cv.Scalar(mean[0], mean[1], mean[2]),
    swapRB
  );
  matC3.delete();
  return input;
}

Mean values that seem to work: [123, 117, 104]
STD value: 1
swapRB: true