Portrait Retouching Importance

Mobile phone camera technology is evolving — wide-angle lens, optical image stabilization, to name but a few. Thanks to this, video recording and mobile image editing apps are emerging one after another, utilizing technology to foster greater creativity.

Among these apps, live-streaming apps are growing with great momentum, thanks to an explosive number of streamers and viewers.

One function that a live-streaming app needs is portrait retouching. The reason is that though mobile phone camera parameters are already staggering, portraits captured by the camera can also be distorted for different reasons. For example, in a dim environment, a streamer's skin tone might appear dark, while factors such as the width of camera lens and shooting angle can make them look wide in videos. Issues like these can affect how viewers feel about a live video and how streamers feel about themselves, signaling the need for a portrait retouching function to address these issues.

I've developed a live-streaming demo app with such a function. Before I developed it, I identified two issues developing this function for a live-streaming app.

First, this function must be able to process video images in real time. A long duration between image input to output compromises interaction between a streamer and their viewers.

Secondly, this function requires a high level of face detection accuracy, to prevent the processed portrait from deformation, or retouched areas from appearing on unexpected parts.

To solve these challenges, I tested several available portrait retouching solutions and settled on the beauty capability from HMS Core Video Editor Kit. Let's see how the capability works to understand how it manages to address the challenges.

How the Capability Addresses the Challenges

This capability adopts the CPU+NPU+GPU heterogeneous parallel framework, which allows it to process video images in real time. The capability algorithm runs faster, but requires less power.

Specifically speaking, the beauty capability delivers a processing frequency of over 50 fps in a device-to-device manner. For a video that contains multiple faces, the capability can simultaneously process a maximum of two faces, whose areas are the biggest in the video. This takes as little as 10 milliseconds to complete.

The capability uses 855 dense facial landmarks so that it can accurately recognize a face, allowing the capability to adapt its effects to a face that moves too fast or at a big angle during live streaming.

To ensure an excellent retouching effect, the beauty capability adopts detailed face segmentation and neutral gray for softening skin. As a result, the final effect looks very authentic.

Not only that, the capability is equipped with multiple, configurable retouching parameters. This feature, I think, is considerate and makes the capability deliver an even better user experience — considering that it is impossible to have a portrait retouching panacea that can satisfy all users. Developers like me can provide these parameters (including those for skin softening, skin tone adjustment, face contour adjustment, eye size adjustment, and eye brightness adjustment) directly to users, rather than struggle to design the parameters by ourselves. This offers more time for fine-tuning portraits in video images.

Knowing these features of the capability, I believed that it could help me create a portrait retouching function for my demo app. So let's move on to see how I developed my app.

Demo Development

Preparations

1. Make sure the development environment is ready.

2. Configure app information in AppGallery Connect, including registering as a developer on the platform, creating an app, generating a signing certificate fingerprint, configuring the fingerprint, and enabling the kit.

3. Integrate the HMS Core SDK.

Configure obfuscation scripts.

5. Declare necessary permissions.

Capability Integration

1. Set up the app authentication information. Two methods are available, using an API key or access token respectively:

API key: Call the setApiKey method to set the key, which only needs to be done once during app initialization.
```
  HVEAIApplication.getInstance().setApiKey("your ApiKey");
```
The API key is obtained from AppGallery Connect, which is generated during app registration on the platform.

It's worth noting that you do not need to hardcode the key in the app code, or store the key in the app's configuration file. The right way to handle this is to store it on cloud, and obtain it when the app is running.
Access token: Call the setAccessToken method to set the token. This is done only once during app initialization.
```
  HVEAIApplication.getInstance().setAccessToken("your access token");
```
The access token is generated by an app itself. Specifically speaking, call the oauth-login.cloud.huawei.com/oauth2/v3/token API and then an app-level access token is obtained.

2. Integrate the beauty capability.

// Create an HVEAIBeauty instance.
HVEAIBeauty hveaiBeauty = new HVEAIBeauty();

// Initialize the engine of the capability.
hveaiBeauty.initEngine(new HVEAIInitialCallback() {
    @Override
    public void onProgress(int progress) {
        // Callback when the initialization progress is received.
    }
    @Override
    public void onSuccess() {
        // Callback when engine initialization is successful.
    }
    @Override
    public void onError(int errorCode, String errorMessage) {
        // Callback when engine initialization failed.
    }
});

// Initialize the runtime environment of the capability in OpenGL. The method is called in the rendering thread of OpenGL.
hveaiBeauty.prepare();

// Set textureWidth (width) and textureHeight (height) of the texture to which the capability is applied. This method is called in the rendering thread of OpenGL after initialization or texture change.
// resize is a parameter, indicating the width and height. The parameter value must be greater than 0.
hveaiBeauty.resize(textureWidth, textureHeight);

// Configure the parameters for skin softening, skin tone adjustment, face contour adjustment, eye size adjustment, and eye brightness adjustment. The value of each parameter ranges from 0 to 1.
HVEAIBeautyOptions options = new HVEAIBeautyOptions.Builder().setBigEye(1)
    .setBlurDegree(1)
    .setBrightEye(1)
    .setThinFace(1)
    .setWhiteDegree(1)
    .build();

// Update the parameters, after engine initialization or parameter change.
hveaiBeauty.updateOptions(options);

// Apply the capability, by calling the method in the rendering thread of OpenGL for each frame. inputTextureId: ID of the input texture; outputTextureId: ID of the output texture.
// The ID of the input texture should correspond to a face that faces upward.
int outputTextureId = hveaiBeauty.process(inputTextureId);

// Release the engine.
hveaiBeauty.releaseEngine();

The development process ends here, so now we can check out how my demo works:

Not to brag, but I do think the retouching result is ideal and natural: With all the effects added, the processed portrait does not appear deformed.

I've got my desired solution for creating a portrait retouching function. I believe this solution can also play an important role in an image editing app or any app that requires portrait retouching. I'm quite curious as to how you will use it. Now I'm off to find a solution that can "retouch" music instead of photos for a music player app, which can, for example, add more width to a song — Wish me luck!

Conclusion

The live-streaming app market is expanding rapidly, receiving various requirements from streamers and viewers. One of the most desired functions is portrait retouching, which is used to address the distorted portraits and unfavorable video watching experience.

Compared with other kinds of apps, a live-streaming app has two distinct requirements for the portrait retouching function, which are real-time processing of video images and accurate face detection. The beauty capability from HMS Core Video Editor Kit addresses them effectively, by using technologies such as the CPU+NPU+GPU heterogeneous parallel framework and 855 dense facial landmarks. The capability also offers several customizable parameters to enable different users to retouch their portraits as needed. On top of these, the capability can be easily integrated, helping develop an app requiring the portrait retouching feature.

How to Develop a Portrait Retouching Function

Table of contents