Have you ever wondered what leads to product development?

Is it the idea and the right mix of talent and passion? Well, all of those are important ingredients but the most significant is a problem statement. There should be a viable need for the product.

At Fynd, we develop innovative technology for the retail industry. When we studied the demands of the ecommerce industry, we realized that selling makeup on online mediums is challenging for beauty brands. Its tough because its only natural for buyers to reach a purchase decision based on physical product application.

Our product journey started with this problem statement and the need for Virtual Makeup technology. The pandemic further created the demand for safe, contactless shopping.

What is GlamAR ?

GlamAR is a Virtual Makeup try-on solution that assists shoppers in virtually trying makeup. They can try out multiple products, see what fits their need and make a hassle-free purchase decision.

Goals to Accomplish

We listed down the developments we would need to make to develop GlamAR:

• ML model to detect accurate facial landmark points in real-time.
• Technique to augment makeup.
• Stabilize and interpolate landmark points.
• Attain the average Frames Per Second or FPS, without glitch and lag.

Working Blocks

GlamAR includes different blocks, similar to the Lego blocks that come together. These blocks are diverse based on reusability, performance, connectivity and platform (either Android, iOS or Desktop).

We can distinguish these blocks into :

• Front End: Android, Web and iOS
• Machine Learning: Segmentation, landmark and Style prediction Model
• C++: Reusable code
• Back End: Backbone to GlamAR

These blocks have specific functionalities and are connected to make up the GlamAR product. The frontend verifies the authentication and then fetches the configuration and weight files from the backend. The input image is pre-processed and parsed to the Machine Learning model, and the model output is sent to C++ block as an input. The image is then augmented and is sent back to the frontend to display as an augmented image. This loop continues to provide a seamless real-time virtual try-on experience.

Front End

The front end consists of 3 domains: Android, Web and iOS. These include the rich UI, camera instance and other functional codes.
We have used Java and Kotlin for Android UI. Most of the image processing code is written in C++ making it memory-optimized, efficient and reusable. We have used NDK(C++) and web assembly to compile the code in Android and Desktop, respectively.

Android

We started with the vision to have a scalable, precise and self-serving solution for the Android platform, to be offered as an SDK. To achieve our vision, we divided our code into three separate modules: Face, Hair and Nail.

The division made the code more organized, and scalable for future updates to other try-on options such as Foundation, Jewellery etc. Separating the features in modules also helps us cater to the diverse needs of our clients in a more precise manner, for example, if a brand only wishes to introduce Facial makeup, they do not need to make space for the Hair module.

All processing is done on the client-side, so we ensure that our processing logic remains out of sight and obscured. To accomplish this, most of the internal code and algorithm is written in native android C++.

Another important goal was to keep the SDK as small as possible, hence we rebuilt the C++ libraries from source(for instance in OpenCV) with only a select few modules so that the whole library is not required.

Issues from the Technology Standpoint

• Common Resolution: The camera resolution depends on the device hardware. Low resolutions can hamper the output performance. To tackle this, we pre-processed each frame to resize it to a particular range.
• Lag and Glitch issue: To have a seamless virtual try-on experience, a frame rate of 10 and above is crucial. To achieve parallel processing, we run the process on two threads that work synchronously so that there is no glitch or lag. By doing so, we were able to achieve an average of 12–25 FPS, providing real-time try-on experience.

Machine Learning Model

Initially, we were using MLKit by Google, to detect the landmark points of the face. We then trained the custom landmark detection model on a dataset of million facial images that have been collected, manually annotated, and combined with the ones that are publicly available. The initial step for training an ML model is to have a rich and error-free data set, and we trained our custom models from scratch to meet our requirements.

Custom Model

• Makeup Prediction Model: We detect the skin and hair region using segmentation and detect the dominant colour for the same. Eye colour plays an important role in predicting eye makeup. Based on the colour attributes, we predict makeup from our colour mapping panel for different occasions, like party, casual, office etc.

• Nail Segmentation: We created a custom model by tweaking Mobilnet v2 layers for the encoder. Decoder part has few variants using upsampling block with either Transpose Convolution or Upsample2D+Convolution and ensuring proper skip connection between encoder and decoder. To prevent overfitting and help our network to learn robust features, we applied dropout regularization.

‍

• Hair Segmentation Model: MediaPipe is a cross-platform framework for building multimodal applied machine learning pipelines. We used MediaPipe to achieve real-time image processing and get the most out of the device hardware.

Few things to know about MediaPipe:

• Delegates CPU heavy computation like image processing and model inference to GPU.
• Uses threads for parallel computing and tracking of objects to gain FPS.
• Has inbuilt support for Tflite and saves time in deploying models.

Swati Modi helped us with the hair colour try-on feature in quick time. Check out her blog explaining how she used MediaPipe for this feature.

C++(Rendering and Algorithm)

We used C++ for compact memory critical tasks like image transformation, landmark stabilization, mesh creation or the rendering of makeup. We combined C++ with OpenCV and OpenGL to render the image efficiently.

Our C++ Algorithms :

• Landmark Stabilization: The points predicted by the model were not stable. To manage this, we wrote an algorithm using OpenCV to stabilize, taking the average threshold approach.
• Noise Removal: The raw image has noise, brightness and lighting issues, our algorithm for the same removes the noise, enhances the brightness and retouches the face accordingly.
• Rendering: Before we render the makeup, we interpolate some new points, taking the average, mean and tangential approach, such that these points cover the edge case. Once we have them, we generate the mesh and render makeup.
• Asset Creation: We designed an algorithm to create the asset at runtime and to authenticate the asset.

Curl package is integrated with C++ code to add network call support, to make the C++ code self-serving, reusable and platform-independent.

Back End

GlamAR’s robust backend is what drives the Admin Panel and the SDK. To assure a quick response, generally under 50ms in the SDK, we have heavily used asynchronous frameworks. This framework includes FastAPI as the web framework, MySQL as the database, and Tortoise-ORM in conjunction with Aerich as the ORM layer.

One of the key features that we developed in-house is the Dynamic Forms. Now adding or editing new product categories and their various configuration parameters doesn’t require writing a single line of code on the front end. To handle high volumes of traffic, we have deployed our services on Kubernetes, that scales up proportionally to the incoming server load.

What about the Business Perspective?

GlamAR is an Augmented Reality product, but how will it help brands, what’s the selling point, and is it really helpful?

Well yes! Brands need to offer more ways to shop that are safe and contactless. With GlamAR you can extend your unique product and in-store experience that shopper’s anticipate from your brand, but virtually.

To read more about the business outlook, take a look here.

Conclusion

In this blog, we take you through the technical details of GlamAR.

We have built GlamAR in a way that it’s easy to integrate with a mobile or web store, simple to use, and a complete self-serve platform where you can add or manage products.

But we do not stop here, it’s just the beginning. We are now looking to add Jewellery and Apparel try-on, so hang in there to experience the next revolution.

Get started with GlamAR! It is free with no time limit on Starter Plan.

For any questions, refer to our FAQs. If you need more information, write to us at glamar@fynd.com.

GlamAR as a product would have not been possible without the developers and their efforts. Kudos to Gaurav Gola, Swati Modi, Anubhav Roy, Neeraj Shukla,Tushar Moraye, Shubham Dalvi, Akash Chokshi, Mahesh Gudi, Mohit Pilkhan, Shubham KamlaPuri!

If you are interested in AI or enjoy coding, designing and what not, check out our openings here.

‍

‍