We recently released a product GlamAR. It is a live try-on app powered by the Augmented Reality technology of Fynd. With GlamAR, you can give yourself a full virtual makeover to find a look you love, right from your phone’s camera! This blog will walk you through the journey from experimentation to final product release.

It’s been around a year since we started experimenting with Augmented Reality in Fynd. As a part of our experimentation, we decided to build a virtual try-on app for beauty products. The idea was to help consumers make an informed purchase decision by letting them virtually try out different beauty products right from their mobiles.

‍

{{cta-box="/component"}}

‍

Evaluation Stage

Providing Augmented Reality based Beauty products Try On is a two-step process:
Step 1: Detect facial landmarks like lips, eyes, etc.
Step 2: Augment face with beauty features like lipstick, blush, eyeliner, etc.
To give the user a seamless experience, both these steps had to be accurate and in real-time.

For detecting facial landmarks, we evaluated the following:

• MLKit by Google
• OpenCV
• DLib

Initially, we used MLKit technology provided by Google, which uses deep learning under the hood to detect facial landmarks. For a given image, it provides the developer with coordinates of the landmarks. Provided landmarks are accurate as well as fast. But the problem with this approach was that MLKit is a Java-based framework and Java lacks good image processing technologies, making it difficult for us to easily manipulate the coordinates and augment beauty features. We had the option of using OpenCV for Android. However, to use it efficiently one needs expertise in android development with native NDK (C++) and computer vision skills.
We lacked the relevant skills, yet wanted to build this product soon. Hence, we looked for other ways to build the feature.
We also evaluated other technologies like Dlib. But porting such libraries to android was overhead since most of such libraries were not actively maintained and hence recent android versions were not supported.

Meanwhile, Google had released a new Augmented Reality technology ARCore. ARCore helps in building augmented reality experiences that seamlessly blend the digital and physical worlds on Android and iOS devices.

Eureka Moment

Recent ARCore version had Augmented Faces feature which provides a developer with 468 points 3D mesh that covers the whole face. This means that we could work on the entire facial area, plus it is blazing fast! It even works in low-light conditions. It had first-class support for Unity engine which we could use for texturing areas like chicks, lips, eyes, etc.

ARCore SDK is available for the Android Studio in Java and for Unity as a plugin. Directly using ARCore in the Android Studio is a meticulous task since it requires OpenGL programming knowledge. Hence. we decided to evaluate Unity3D. Our team lacked Unity3D expertise. It took us some time to learn. Soon we realized that learning Unity was not enough. To build textures of lips and other beauty features, knowledge of 3D modeling is also required. So we dived into learning again and reached a state where we could finally augment beauty features on the face in real-time. We will open-source our POC soon as a contribution to the opensource community.

Although, one problem with ARCore was the limited device support. Our customers are on diverse mobile platforms, so ARCore was not enough. But POC boosted our confidence in implementing this feature even on low-end devices.

Back To Basics

We came back to MLKit for real-time facial features detection and decided to use OpenCV for writing image processing algorithms to augment makeup features over an image. With some struggle, we could pass MLKT facial feature points to native android C++ functions. Mohit Pilkhan our new team member came to rescue us with his C++ skills to write image processing algorithms. Together with android and ML expert Anubhav Roy and Vineet Kushwaha, we implemented Lipstick, Eyeliner, Blush, Eye shadow and Kajal features. Swati Modi helped us with cool feature hair color try-on in a quick time. She has written one awesome blog explaining how she used MediaPipe for this feature.

Features currently supported in GlamAR are:

• Face
  - Lipstick
  - Blush
  - Eyeliner
  - Eye-shadow
• Hair
  - Hair Colour

For points interpolation, we have used tinyspline.

Mediapipe To Rescue

While developing facial features we faced difficulty in various areas including. Various areas including iterative testing while implementing and optimizing algorithms is time-consuming. GPU programming and parallel computing to achieve real-time speed are hard.

MediaPipe is a cross-platform framework for building multimodal applied machine learning pipelines. MediaPipe is something that google internally uses for its products since 2012 and open-sourced it in June 2019 at CVPR. To achieve real-time image processing we needed to get most out of the phone’s hardware considering its limited specifications. Few of techniques we thought of using are

• Delegating CPU heavy computation like image processing and model inference to GPU.
• Using threads for parallel computing.
• Using tracking of objects to gain fps.

MediaPipe has all of the above features. Apart from that, MediaPipe has the following features which make it an ideal candidate for our use case:

• It is cross-platform which makes it easier to implement and test algorithms.
• Built-in graphics utilities and profiling support which helps in majorly focusing on problem-solving.
• On-device performance. It is designed to support real-time performance on edge devices.
• Inbuilt support for tflite saves time in deploying models.

Currently, our hair segmentation feature uses MediaPipe. In the future, we will be migrating our face feature to MediaPipe as well.

At The Finishing Line

‍

After weeks of testing and tuning, we finally published our app GlamAR on Android Playstore last week. We will be releasing the V2 version of the same app with more beauty categories in the product. We are also working on removing MLKit dependency to reduce app size. Our senior developer Gaurav Gola along with the whole team is working hard to release the V2 version soon. We will talk more about that in the future blog.

Feel free to reach out to us on glamar[at]fynd.com for collaboration, suggestions, and feedback. And yes, don’t forget to check out our app and website.

#HappyFynding

Thanks to Pritha Saha.

‍