# How to get facial landmarks on Android platform?

**URL:** <https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883>\
**Category:** Android/Java\
**Tags:** face\
**Created:** [February 6, 2023, 5:40am UTC](https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883 "2023-02-06T05:40:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cassiel](https://avatars.discourse-cdn.com/v4/letter/c/77aa72/32.png) [@Cassiel](https://forum.opencv.org/u/Cassiel)\
**Post date:** [February 6, 2023, 5:40am UTC](https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883/1 "2023-02-06T05:40:55Z")

</div>

I tried to use the dlib library on Android platform, but the detection speed is very slow. Is there any other way to get facial landmarks？

---

<div class="post-metadata">

**Author:** ![berak](https://avatars.discourse-cdn.com/v4/letter/b/85f322/32.png) [@berak](https://forum.opencv.org/u/berak)\
**Post date:** [February 6, 2023, 10:21am UTC](https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883/2 "2023-02-06T10:21:14Z")

</div>

> [@Cassiel](#):
>
> Is there any other way to get facial landmarks？

if you can build with contrib modules,  
there are [landmarks detectors here](https://docs.opencv.org/4.x/db/d7c/group__face.html)  
(68 pts, similar to dlib)  
(i’d recommend the LBF one)

if you can live with 5 lms (eyes, nose, mouth), there’s a [combined face / lm detector here](https://docs.opencv.org/4.x/df/d20/classcv_1_1FaceDetectorYN.html)  
(it’s not (yet) in the docs, but you can find [example usage here](https://github.com/opencv/opencv/blob/4.x/samples/dnn/face_detect.cpp))

> [@Cassiel](#):
>
> use the dlib library on Android platform, but the detection speed is very slow.

i find that hard to believe.  
are you doing something silly, like using their face detector before that (gross slow!),  
or even - load the 68mb model each time you use it ?

please share (relevant piece of) code !

---

<div class="post-metadata">

**Author:** ![Cassiel](https://avatars.discourse-cdn.com/v4/letter/c/77aa72/32.png) [@Cassiel](https://forum.opencv.org/u/Cassiel)\
**Post date:** [February 7, 2023, 2:14am UTC](https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883/3 "2023-02-07T02:14:36Z")

</div>

// at the beginning, i load the modle file(shape\_predictor\_68\_face\_landmarks.dat). this file is at the assets.

```auto
void FaceDetector::loadShapePredictor(JNIEnv *env, jobject thiz,
                        jobject assetManager,
                        jstring fileName) {
    const char *file_name = env->GetStringUTFChars(fileName, nullptr);
    env->ReleaseStringUTFChars(fileName, file_name);

    //get AAssetManager
    AAssetManager *native_asset = AAssetManager_fromJava(env, assetManager);

    //open file
    AAsset *assetFile = AAssetManager_open(native_asset, file_name, AASSET_MODE_BUFFER);
    //get file length
    size_t file_length = static_cast<size_t>(AAsset_getLength(assetFile));
    char *model_buffer = (char *) malloc(file_length);
    //read file data
    AAsset_read(assetFile, model_buffer, file_length);
    //the data has been copied to model_buffer, so , close it
    AAsset_close(assetFile);

    //LOGI("asset file length %d", file_length);

    //char* to istream
    membuf mem_buf(model_buffer, model_buffer + file_length);
    std::istream in(&mem_buf);

    //load shape_predictor_68_face_landmarks.dat from memory
    dlib::deserialize(this->predictor, in);

    //free malloc
    free(model_buffer);
}

```

// in ImageReader.OnImageAvailableListener(), get image and detect faces

```auto
private ImageReader.OnImageAvailableListener mOnPreviewImageAvailableListener = new ImageReader.OnImageAvailableListener() {
        @Override
        public void onImageAvailable(ImageReader reader) {
            Image image = reader.acquireLatestImage();
            System.out.println("image.getFormat(): " + image.getFormat());
            if (image != null) {
                if (mCamera2FrameCallback != null) {
                	// convert image to yuv_420_888_data
                    byte[] yuv_420_888_data = CameraUtil.YUV_420_888_data(image);
                    mCamera2FrameCallback.onPreviewFrame(yuv_420_888_data, image.getWidth(), image.getHeight());
                    int width = image.getWidth();
                    int height = image.getHeight();
                    int length = width * height * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
                    // detect faces
                    int countFaces = mFaceTracker.countFacesYUV(yuv_420_888_data, width, height);
                    System.out.println("it has " + countFaces + " faces");
                }
                image.close();
            }
        }
    };

```

```auto
extern "C"
JNIEXPORT jint JNICALL
Java_cn_twan_headposedlib_face_FaceTracker_countFacesYUV(JNIEnv *env, jobject thiz,
                                                         jbyteArray data, jint width, jint height) {
    // TODO: implement countFacesYUV()

    LOGCATE("countFacesYUV");
    jbyte *data_ = env->GetByteArrayElements(data, NULL);
    FaceDetector *mDetPtr = FaceDetector::getFaceTracker(env, thiz);
    cv::Mat rgbaMat(height, width, CV_8UC1, data_);

    cv::Mat bgrMat;
    cv::cvtColor(rgbaMat, bgrMat, cv::COLOR_YUV2BGR_NV21);
    jint size = mDetPtr->countFaces(bgrMat);

    return size;
}

```

```auto
int FaceDetector::countFaces(const cv::Mat &image) {
    if (image.empty())
        return 0;

    LOGD("image.channels: %d", image.channels());

    if (image.channels() == 1) {
        cv::cvtColor(image, image, cv::COLOR_GRAY2BGR);
    }

    cv::Mat img_gray;
    cv::cvtColor(image, img_gray, cv::COLOR_BGR2GRAY);
    dlib::array2d<dlib::bgr_pixel> dlib_image;
    dlib::assign_image(dlib_image, dlib::cv_image<uchar>(img_gray));

    det_rects.clear();

    det_rects = face_detector(dlib_image);

    for (unsigned long i = 0; i < det_rects.size(); i++) {
        LOGD("before predictor");
        dlib::full_object_detection shape = predictor(dlib_image, det_rects[i]);
        LOGD("shape.num_parts: %ld", shape.num_parts());
        for (unsigned long j = 0; j < shape.num_parts(); j++) {
            cv::circle(image, cv::Point2l(shape.part(j).x(), shape.part(j).y()), 2, cv::COLORMAP_PINK);
            LOGD("shapes[%ld].part(%ld): (%ld, %ld)", i, j, shape.part(j).x(), shape.part(j).y());
        }
        shapes.push_back(shape);
    }

    LOGD("shapes.size(): %d", shapes.size());

    return det_rects.size();
}

```

---

<div class="post-metadata">

**Author:** ![berak](https://avatars.discourse-cdn.com/v4/letter/b/85f322/32.png) [@berak](https://forum.opencv.org/u/berak)\
**Post date:** [February 7, 2023, 7:49am UTC](https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883/4 "2023-02-07T07:49:04Z")

</div>

why on earth is it even using landmarks ?  
it’s only _counting_ faces !

> [@Cassiel](#):
>
> `det_rects = face_detector(dlib_image);`

please profile this

> [@Cassiel](#):
>
> ```auto
> cv::cvtColor(rgbaMat, bgrMat, cv::COLOR_YUV2BGR_NV21);
>  
> 
> ```

algo works on grayscale images, convert directly:  
[https://docs.opencv.org/4.x/d8/d01/group\_\_imgproc\_\_color\_\_conversions.html#gga4e0972be5de079fed4e3a10e24ef5ef0a1cba7092b4f0302693526ca578159ea0](https://docs.opencv.org/4.x/d8/d01/group __imgproc__ color__conversions.html#gga4e0972be5de079fed4e3a10e24ef5ef0a1cba7092b4f0302693526ca578159ea0)

> [@Cassiel](#):
>
> ```auto
> if (image.channels() == 1) {
> cv::cvtColor(image, image, cv::COLOR_GRAY2BGR);
> }
> 
> ```

why ? does not make sense

---

<div class="post-metadata">

**Author:** ![Cassiel](https://avatars.discourse-cdn.com/v4/letter/c/77aa72/32.png) [@Cassiel](https://forum.opencv.org/u/Cassiel)\
**Post date:** [February 7, 2023, 9:45am UTC](https://forum.opencv.org/t/how-to-get-facial-landmarks-on-android-platform/11883/5 "2023-02-07T09:45:54Z")

</div>

I want to get landmarks to add stickers on the faces. at first, i want to judge whether it has faces, so, i return the count of faces. i haven’t used landmarks yet.  
And i met another problem. the count of faces is always 0 when i’m facing the front camera. i guess it may caused by the pictures data. but i have no idea.

i changed `cv::COLOR_YUV2BGR_NV21` into `cv::COLOR_YUV420p2GRAY`.

thanks for your help.