3D Audio Overview

Immersive experience is much talked about in the current mobile app world, given how it evokes emotions from users to merge the virtual world with reality.

3D audio is a fantastic gimmick that is capable of delivering such an experience. This tech provides listeners with an audio experience that mimics how they hear sounds in real life, mostly by using the binaural sound systems to capture, process, and play back audio waves. 3D audio allows the listener to know where audio sources are from, thereby delivering a richer experience.

The global 3D audio market, according to a report released by ReportLinker, is expected to reach 13.7 billion dollars by 2027 — which marks an immense financial opportunity, as long as this kind of audio effects can be enjoyed by as many as possible users.

The evolution of mobile app technology has made this a reality, making 3D audio more accessible than ever with no need for a bulky headset or a pair of fancy (but expensive) headphones. Truth to be told, I just lost one of my Bluetooth earphones down the drain a few weeks ago and I was struggling to manage without 3D audio. This made me realize that the built-in 3D audio feature is paramount for an app.

Well, in an earlier post I created a demo audio player with the 3D audio feature, thanks to the spatial audio capability of the UI SDK from HMS Core Audio Editor Kit. And in that post, I mentioned that after verifying the capability's functionality, I'd like to create my own UI rather than the preset one of the SDK. Therefore, I turned to the fundamental capability SDK from the kit, which provides an even more powerful spatial audio capability for implementing 3D audio and allows for UI customization.

Check out what I've created:

The capability helps my demo automatically recognize over 10 types of audio sources and can render audio in any of the following modes: fixed position, dynamic rendering, and extension. The dynamic rendering mode is used as an example here, which allows the following parameters to be specified: position of audio in a certain place, duration of a round of audio circling the listener, and the direction to which audio circles. In this way, the spatial audio capability is applicable to different music genres and application scenarios.

Let's see the demo development procedure in detail.

Developing the Demo

Preparations

Make sure the following requirements are met:

Software:

JDK version: 1.8 or later
Android Studio version: 3.X or later
- minSdkVersion: 24 or later
- targetSdkVersion: 33 (recommended)
- compileSdkVersion: 30 (recommended)
- Gradle version: 4.6 or later (recommended)

Hardware: a mobile phone used for testing, whose OS can be EMUI (version 5.0 or later) or Android (version 7.0 to 13)

Configure app information in AppGallery Connect. You need to register for a developer account, create a project and an app, generate a signing certificate fingerprint, configure the fingerprint, enable the kit for the project, and manage the default data processing location.
Integrate the app with the HMS Core SDK. During this step, ensure the Maven repository address for the HMS Core SDK is configured in the project.
Declare necessary permissions in the AndroidManifest.xml file, involving the vibration permission, microphone permission, storage write permission, storage read permission, Internet permission, network status access permission, and permission to obtaining the changed network connectivity state.

<uses-permission android:name="android.permission.VIBRATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CHANGE_NETWORK_STATE" />

SDK Integration

Set the app authentication information via:

An access token (which is obtained by calling the oauth-login.cloud.huawei.com/oauth2/v3/token API). Call setAccessToken to set an access token during app initialization.
```
  HAEApplication.getInstance().setAccessToken("access token");
```
An API key (which is allocated to the app during app registration in AppGallery Connect). Call setApiKey to set the key during app initialization.
```
  HAEApplication.getInstance().setApiKey("API key");
```

Call applyAudioFile to apply the spatial audio effect.

 // Apply spatial audio.
 // Fixed position mode.
 HAESpaceRenderFile haeSpaceRenderFile = new HAESpaceRenderFile(SpaceRenderMode.POSITION);
 haeSpaceRenderFile.setSpacePositionParams(
                             new SpaceRenderPositionParams(x, y, z));
 // Dynamic rendering mode.
 HAESpaceRenderFile haeSpaceRenderFile = new HAESpaceRenderFile(SpaceRenderMode.ROTATION);
 haeSpaceRenderFile.setRotationParams(new SpaceRenderRotationParams(
                                     x, y, z, circling_time, circling_direction));
 // Extension.
 HAESpaceRenderFile haeSpaceRenderFile = new HAESpaceRenderFile(SpaceRenderMode.EXTENSION);
 haeSpaceRenderFile.setExtensionParams(new SpaceRenderExtensionParams(radian, angle));
 // Call the API.
 haeSpaceRenderFile.applyAudioFile(inAudioPath, outAudioDir, outAudioName, callBack);
 // Cancel applying spatial audio.
 haeSpaceRenderFile.cancel();

The whole development procedure closes here, giving birth to an app that works like the GIF above.

Use Cases Beyond Music Playback

Music playback is just one of the basic use cases of the spatial audio capability. I believe that it can be adopted in many other scenarios, in navigation, for example. Spatial audio can help users navigate from A to B even easier. It could, for example, tell users to "Turn left" with the sound coming from the left side of a listener, taking immersion to a new level.

Karaoke apps on the other hand can count on spatial audio and audio source separation (a capability I've also used for my demo) for generating accompaniments with even better effects: The audio source separation capability first abstracts the accompaniment a user needs from a song, and the spatial audio capability then works its magic to turn the accompaniment into 3D audio, which mimics how an accompaniment would really sound like in a concert or recording studio.

Takeaway

3D audio contributes heavily to the immersive experience of a mobile app, as it can digitally imitate how sounds are perceived in the real world. Such an effect, coupled with the huge financial benefits of the 3D audio market and its expansive application scenarios, has thrown 3D audio into the spotlight for app developers.

What's more, devices such as headsets and headphones are no longer necessary for enjoying 3D audio, thanks to advancements in mobile app technology. A solution to implementing the feature comes from Audio Editor Kit, which is known as spatial audio and is available in two SDKs: UI SDK and fundamental capability SDK. The former has a preset UI featuring basic functions, while the latter allows for UI customization and offers more functions (including three rendering modes applicable to different use cases and music genres). Either way, with the spatial audio capability, users of an app can have an audio experience that resembles how sounds are perceived in the real world.

Walkthrough with J