Detect the mouth and tongue outside the mouth - openCV.js

1

Could someone help me please? I can’t detect the tongue or mouth. Im using openCV.js

Face works.

  // detect faces
  const msize = new cv.Size(0, 0);
  faceCascade.detectMultiScale(gray, faces, 1.1, 3, 0, msize, msize);
  for (let i = 0; i < faces.size(); ++i) {
    const roiGray = gray.roi(faces.get(i));
    const roiSrc = newImg.roi(faces.get(i));
    const point1 = new cv.Point(faces.get(i).x, faces.get(i).y);
    const point2 = new cv.Point(
      faces.get(i).x + faces.get(i).width,
      faces.get(i).y + faces.get(i).height
    );
    cv.rectangle(newImg, point1, point2, [255, 0, 0, 255]);

// detect mouth openCV.js

    mouthCascade.detectMultiScale(roiGray, mouth);

    // console.log(mouth.size());

    for (let h = 0; h < mouth.size(); ++h) {

      const point1 = new cv.Point(mouth.get(h).x, mouth.get(h).y);

      const point2 = new cv.Point(

        mouth.get(h).x + mouth.get(h).width,

        mouth.get(h).y + mouth.get(h).height

      );

      cv.rectangle(roiSrc, point1, point2, [0, 0, 255, 255]);

      // cv.rectangle(roiSrc, (x,y), (x+w,y+h), (0,255,0), 3)

    }

Could someone help me with the mouth and tongue?

2

could you add the image, and mention, which cascades you tried ?

there is nothing builtin for this.
however, once you have a stable mouth detection, you might try e.g. to train an SVM on “tongue in / tongue out” classification of the mouth region

3

Thanks for your response.

cascade mouth: haarcascade_mcs_mouth.xml

for the tongue I didn’t find any cascade

image I’m testing

Marilia-Gabriela-300x300

4

i dont know the mcs cascades too well, but most probably they were trained on a closed mouth pose, where both lips are visible, –
that might explain, why it cannot find the mouth here
(and i’d also give up on finding the mouth using cascades here. simply skip that part, and infer on the “lower half” of the detected face)

unfortunately, there is no simple machinelearning (like the SVM) in opencv.js but you could still try to either:

  • train a small cnn on toungue out / in images (later use the dnn module to infer it)
  • train a HogDescriptor
  • try a simple sum(absdiff) and find a threshold

again, dont expect a “ready-made” solution here !