How I Developed a Smile Filter for My App

Background

I recently read an article that explained how we as human beings are hardwired to enter the fight-or-flight mode when we realize that we are being watched. This feeling is especially strong when somebody else is trying to take a picture of us, which is why many of us find it difficult to smile in photos. This effect is so strong that we've all had the experience of looking at a photo right after it was taken and noticing straight away that the photo needs to be retaken because our smile wasn't wide enough or didn't look natural. So, the next time someone criticizes my smile in a photo, I'm just going to them, "It's not my fault. It's literally an evolutionary trait!"

Or, instead of making such an excuse, what about turning to technology for help? Actually, I have tried using some photo editor apps to modify my portrait photos, making my facial expression look nicer by, for example, removing my braces, whitening my teeth, and erasing my smile lines. However, maybe it's because of my rusty image editing skills, the modified images often turn out to be strange.

My lack of success with photo editing made me wonder: Wouldn't it be great if there was a function specially designed for people like me, who find it difficult to smile naturally in photos and who aren't good at photo editing, which could automatically give us picture-perfect smiles?

I then suddenly remembered that I had heard about an interesting function called smile filter that has been going viral on different apps and platforms. A smile filter is an app feature which can automatically add a natural-looking smile to a face detected in an image. I have tried it before and was really amazed by the result. In light of my sudden recall, I decided to create a demo app with a similar function, in order to figure out the principle behind it.

To provide my app with a smile filter, I chose to use the auto-smile capability provided by HMS Core Video Editor Kit. This capability automatically detects people in an image and then lightens up the detected faces with a smile (either closed- or open-mouth) that perfectly blends in with each person's facial structure. With the help of such a capability, a mobile app can create the perfect smile in seconds and save users from the hassle of having to use a professional image editing program.

Check the result out for yourselves:

Looks pretty natural, right? This is the result offered by my demo app integrated with the auto-smile capability. The original image looks like this:

Next, I will explain how I integrated the auto-smile capability into my app and share the relevant source code from my demo app.

Integration Procedure

Preparations

i. Configure necessary app information. This step requires you to register a developer account, create an app, generate a signing certificate fingerprint, configure the fingerprint, and enable required services.

ii. Integrate the SDK of the kit.

iii. Configure the obfuscation scripts.

iv. Declare necessary permissions.

Project Configuration

i. Set the app authentication information. This can be done via an API key or an access token.

• Using an API key: You only need to set the app authentication information once during app initialization.

MediaApplication.getInstance().setApiKey("your ApiKey");

• Or, using an access token: You only need to set the app authentication information once during app initialization.

MediaApplication.getInstance().setAccessToken("your access token");

ii. Set a License ID, which must be unique because it is used to manage the usage quotas of the service.

MediaApplication.getInstance().setLicenseId("License ID");

iii. Initialize the runtime environment for the HuaweiVideoEditor object. Remember to release the HuaweiVideoEditor object when exiting the project.

• Create a HuaweiVideoEditor object.

HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());

• Specify the preview area position. Such an area is used to render video images, which is implemented by SurfaceView created within the SDK. Before creating such an area, specify its position in the app first.

<LinearLayout    
    android:id="@+id/video_content_layout"    
    android:layout_width="0dp"    
    android:layout_height="0dp"    
    android:background="@color/video_edit_main_bg_color"    
    android:gravity="center"    
    android:orientation="vertical" />
// Specify the preview area position.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);

// Specify the preview area layout.
editor.setDisplay(mSdkPreviewContainer);

• Initialize the runtime environment. If license verification fails, LicenseException will be thrown.

After it is created, the HuaweiVideoEditor object will not occupy any system resources. You need to manually set when the runtime environment of the object will be initialized. Once you have done this, necessary threads and timers will be created within the SDK.

try {
        editor.initEnvironment();
   } catch (LicenseException error) { 
        SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());    
        finish();
        return;
   }

Function Development

// Apply the auto-smile effect. Currently, this effect only supports image assets.
imageAsset.addFaceSmileAIEffect(new HVEAIProcessCallback() {
        @Override
        public void onProgress(int progress) {
            // Callback when the handling progress is received.
        }
        @Override
        public void onSuccess() {
            // Callback when the handling is successful.
        }
        @Override
        public void onError(int errorCode, String errorMessage) {
            // Callback when the handling failed.
        }
});
// Stop applying the auto-smile effect.
imageAsset.interruptFaceSmile();
// Remove the auto-smile effect.
imageAsset.removeFaceSmileAIEffect();

And with that, I successfully integrated the auto-smile capability into my demo app, and now it can automatically add smiles to faces detected in the input image.

Conclusion

Research has demonstrated that it is normal for people to behave unnaturally when we are being photographed. Such unnaturalness becomes even more obvious when we try to smile. This explains why numerous social media apps and video/image editing apps have introduced smile filter functions, which allow users to easily and quickly add a naturally looking smile to faces in an image.

Among various solutions to such a function, HMS Core Video Editor Kit's auto-smile capability stands out by providing excellent, natural-looking results and featuring straightforward and quick integration.

What's better, the auto-smile capability can be used together with other capabilities from the same kit, to further enhance users' image editing experience. For example, when used in conjunction with the kit's AI color capability, you can add color to an old black-and-white photo and then use auto-smile to add smiles to the sullen expressions of the people in the photo. It's a great way to freshen up old and dreary photos from the past.