On-device real-time few-shot face stylization – Google Research Blog

In recent times, we’ve got witnessed rising curiosity throughout customers and researchers in built-in augmented actuality (AR) experiences utilizing real-time face characteristic technology and modifying capabilities in cell purposes, together with brief movies, digital actuality, and gaming. In consequence, there’s a rising demand for light-weight, but high-quality face technology and modifying fashions, which are sometimes based mostly on generative adversarial community (GAN) methods. Nonetheless, the vast majority of GAN fashions endure from excessive computational complexity and the necessity for a big coaching dataset. As well as, it is usually essential to make use of GAN fashions responsibly.

On this put up, we introduce MediaPipe FaceStylizer, an environment friendly design for few-shot face stylization that addresses the aforementioned mannequin complexity and knowledge effectivity challenges whereas being guided by Google’s accountable AI Ideas. The mannequin consists of a face generator and a face encoder used as GAN inversion to map the picture into latent code for the generator. We introduce a mobile-friendly synthesis community for the face generator with an auxiliary head that converts options to RGB at every stage of the generator to generate top quality photos from coarse to positive granularities. We additionally fastidiously designed the loss capabilities for the aforementioned auxiliary heads and mixed them with the frequent GAN loss capabilities to distill the scholar generator from the trainer StyleGAN mannequin, leading to a light-weight mannequin that maintains excessive technology high quality. The proposed answer is obtainable in open supply by MediaPipe. Customers can fine-tune the generator to be taught a mode from one or a number of photos utilizing MediaPipe Mannequin Maker, and deploy to on-device face stylization purposes with the personalized mannequin utilizing MediaPipe FaceStylizer.

Few-shot on-device face stylization

An end-to-end pipeline

Our purpose is to construct a pipeline to assist customers to adapt the MediaPipe FaceStylizer to totally different types by fine-tuning the mannequin with a number of examples. To allow such a face stylization pipeline, we constructed the pipeline with a GAN inversion encoder and environment friendly face generator mannequin (see under). The encoder and generator pipeline can then be tailored to totally different types by way of a few-shot studying course of. The person first sends a single or a number of related samples of the type photos to MediaPipe ModelMaker to fine-tune the mannequin. The fine-tuning course of freezes the encoder module and solely fine-tunes the generator. The coaching course of samples a number of latent codes near the encoding output of the enter type photos because the enter to the generator. The generator is then educated to reconstruct a picture of an individual’s face within the type of the enter type picture by optimizing a joint adversarial loss operate that additionally accounts for type and content material. With such a fine-tuning course of, the MediaPipe FaceStylizer can adapt to the personalized type, which approximates the person’s enter. It may then be utilized to stylize take a look at photos of actual human faces.

Generator: BlazeStyleGAN

The StyleGAN mannequin household has been extensively adopted for face technology and varied face modifying duties. To assist environment friendly on-device face technology, we based mostly the design of our generator on StyleGAN. This generator, which we name BlazeStyleGAN, is much like StyleGAN in that it additionally accommodates a mapping community and synthesis community. Nonetheless, for the reason that synthesis community of StyleGAN is the most important contributor to the mannequin’s excessive computation complexity, we designed and employed a extra environment friendly synthesis community. The improved effectivity and technology high quality is achieved by:

Decreasing the latent characteristic dimension within the synthesis community to 1 / 4 of the decision of the counterpart layers within the trainer StyleGAN,

Designing a number of auxiliary heads to rework the downscaled characteristic to the picture area to kind a coarse-to-fine picture pyramid to guage the perceptual high quality of the reconstruction, and

Skipping all however the ultimate auxiliary head at inference time.

With the newly designed structure, we prepare the BlazeStyleGAN mannequin by distilling it from a trainer StyleGAN mannequin. We use a multi-scale perceptual loss and adversarial loss within the distillation to switch the excessive constancy technology functionality from the trainer mannequin to the scholar BlazeStyleGAN mannequin and likewise to mitigate the artifacts from the trainer mannequin.

Extra particulars of the mannequin structure and coaching scheme may be present in our paper.

Visible comparability between face samples generated by StyleGAN and BlazeStyleGAN. The photographs on the primary row are generated by the trainer StyleGAN. The photographs on the second row are generated by the scholar BlazeStyleGAN. The face generated by BlazeStyleGAN has related visible high quality to the picture generated by the trainer mannequin. Some outcomes exhibit the scholar BlazeStyleGAN suppresses the artifacts from the trainer mannequin within the distillation.

Within the above determine, we exhibit some pattern outcomes of our BlazeStyleGAN. By evaluating with the face picture generated by the trainer StyleGAN mannequin (prime row), the pictures generated by the scholar BlazeStyleGAN (backside row) keep excessive visible high quality and additional scale back artifacts produced by the trainer because of the loss operate design in our distillation.

An encoder for environment friendly GAN inversion

To assist image-to-image stylization, we additionally launched an environment friendly GAN inversion because the encoder to map enter photos to the latent house of the generator. The encoder is outlined by a MobileNet V2 spine and educated with pure face photos. The loss is outlined as a mixture of picture perceptual high quality loss, which measures the content material distinction, type similarity and embedding distance, in addition to the L1 loss between the enter photos and reconstructed photos.

On-device efficiency

We documented mannequin complexities when it comes to parameter numbers and computing FLOPs within the following desk. In comparison with the trainer StyleGAN (33.2M parameters), BlazeStyleGAN (generator) considerably reduces the mannequin complexity, with solely 2.01M parameters and 1.28G FLOPs for output decision 256×256. In comparison with StyleGAN-1024 (producing picture measurement of 1024×1024), the BlazeStyleGAN-1024 can scale back each mannequin measurement and computation complexity by 95% with no notable high quality distinction and may even suppress the artifacts from the trainer StyleGAN mannequin.

Mannequin
   
Picture Measurement
   
#Params (M)
   
FLOPs (G)

StyleGAN
   
1024
   
33.17
   
74.3

BlazeStyleGAN
   
1024
   
2.07
   
4.70

BlazeStyleGAN
   
512
   
2.05
   
1.57

BlazeStyleGAN
   
256
   
2.01
   
1.28

Encoder
   
256
   
1.44
   
0.60

Mannequin complexity measured by parameter numbers and FLOPs.

We benchmarked the inference time of the MediaPipe FaceStylizer on varied high-end cell gadgets and demonstrated the leads to the desk under. From the outcomes, each BlazeStyleGAN-256 and BlazeStyleGAN-512 achieved real-time efficiency on all GPU gadgets. It may run in lower than 10 ms runtime on a high-end cellphone’s GPU. BlazeStyleGAN-256 can even obtain real-time efficiency on the iOS gadgets’ CPU.

Mannequin
   
BlazeStyleGAN-256 (ms)
   
Encoder-256 (ms)

iPhone 11
   
12.14
   
11.48

iPhone 12
   
11.99
   
12.25

iPhone 13 Professional
   
7.22
   
5.41

Pixel 6
   
12.24
   
11.23

Samsung Galaxy S10
   
17.01
   
12.70

Samsung Galaxy S20
   
8.95
   
8.20

Latency benchmark of the BlazeStyleGAN, face encoder, and the end-to-end pipeline on varied cell gadgets.

Equity analysis

The mannequin has been educated with a excessive range dataset of human faces. The mannequin is predicted to be honest to totally different human faces. The equity analysis demonstrates the mannequin performs good and balanced when it comes to human gender, skin-tone, and ages.

Face stylization visualization

Some face stylization outcomes are demonstrated within the following determine. The photographs within the prime row (in orange containers) characterize the type photos used to fine-tune the mannequin. The photographs within the left column (within the inexperienced containers) are the pure face photos used for testing. The 2×4 matrix of photos represents the output of the MediaPipe FaceStylizer which is mixing outputs between the pure faces on the left-most column and the corresponding face types on the highest row. The outcomes exhibit that our answer can obtain high-quality face stylization for a number of common types.

Pattern outcomes of our MediaPipe FaceStylizer.

MediaPipe Options

The MediaPipe FaceStylizer goes to be launched to public customers in MediaPipe Options. Customers can leverage MediaPipe Mannequin Maker to coach a personalized face stylization mannequin utilizing their very own type photos. After coaching, the exported bundle of TFLite mannequin information may be deployed to purposes throughout platforms (Android, iOS, Internet, Python, and many others.) utilizing the MediaPipe Duties FaceStylizer API in just some traces of code.

Acknowledgements

This work is made doable by a collaboration spanning a number of groups throughout Google. We’d prefer to acknowledge contributions from Omer Tov, Yang Zhao, Andrey Vakunov, Fei Deng, Ariel Ephrat, Inbar Mosseri, Lu Wang, Chuo-Ling Chang, Tingbo Hou, and Matthias Grundmann.