Posted by the Smule Engineering team: David Gayle, Chris Manchester, Mark Gills, Trayko Traykov, Randal Leistikow, Mariya Ivanova.
Executive Summary
As the most downloaded singing app of all time, Smule Inc. has been investing on Android to improve the overall audio quality audio and more specifically to improve the lower latency, i.e. allowing singers to hear their voices in the headset as they perform. The teams specialized in Audio and Video allocated a significant part of 2021 into making the necessary changes to convert the Smule application used by over ten million Android users from using the OpenSL audio API to the Oboe audio library, enabling roughly a 10%+ increase in recording completion rate.
Introduction
Smule Inc. is a leader in karaoke, with an app that helps millions of people sing their favorite songs and share performances daily. The Smule application goes beyond traditional karaoke by focusing on co-creation, offering users the unique opportunity to share music and collaborate with friends, other singers on the platform, and their favorite music artists. Audio quality is paramount, and, in 2020, the Smule team saw potential to enhance the experience on Android.
Smule’s legacy OpenSL implementation wasn’t well-suited to leverage the blazing-fast hardware of new devices while supporting the diverse devices across its world-wide market. Smule’s development team determined that upgrading the audio system was a necessary and a logical improvement.
Oboe Rollout Strategy
Smule was faced with two possible routes for improvement: directly targeting AAudio, a high-performance Android C audio API introduced in Android O designed for applications that require low latency, or Oboe, which wraps both AAudio and OpenSL internally. After careful evaluation, Smule’s development team opted for Oboe’s easy-to-use code base, and broad device compatibility, and robust community support, which achieved the lowest latency and made the best use of the available native audio.
The conversion to Oboe represented a significant architectural and technologicaltechnology evolution. As a result, Smule approached the rollout process conservatively, with a planned, gradual release that started with a small selection of device models to d validate quality. Week after week, the team enabled more devices (reverting a limited number of devices exhibiting problems in Oboe back to OpenSL). This incremental, methodical approach helped to minimize risk and allowed the engineering team to handle device-specific issues as they occurred.
Improving the Audio Quality Experience
Smule switched to Oboe to help improve the app experience. They hoped to reduce dramatically audio playback crashes, eliminate issues such as echo and crackling during recording, and reduce audio latency. A recent article in the Android Developers blog shows that the average latency of the twenty most popular devices decreased from 109 ms in 2017 to 39ms today using Oboe. Whereas a monitoring delay of 109ms is heard as a distinct echo which interferes with live singing, 39ms is beneath the acceptable threshold for real-time applications. The latencies of top devices today are all within 22ms of one another, and this consistency is a big plus.
The lift in recording completion rate Smule has seen using Oboe is likely due to this lower latency, allowing singers to hear their voices in the headset as they perform with Smule’s world-class audio effects applied, but without an echo.
Using an effective collaborative GitHub portal dedicated to Oboe, the Google team played a significant role in Smule’s Oboe integration, providing them with key insights and support. Working together, the two teams were able to launch the largest Oboe deployment to date, reaching millions of active users. The Smule team contributed to addressing some Oboe code issues, and the Google team coordinated with certain mobile device makers to further improve Oboe's compatibility.
Audio quality is of the utmost importance to our community of singers, and we're thankful for our shared commitment to delivering the best possible experience as well as empowering musical creation on Smule. - Eric Dumas, Smule CTO.
Given the massive scale of the operation, it was only natural to face device-specific issues. One notable example was an OS built-in functionality that injected echo sound effects in the raw audio stream, which prevented Smule from correctly applying its own patented DSP algorithms and audio filters. Google’s team came to the rescue, providing lightning fast updates and patches to the library. The process of reporting Oboe issues was straightforward, well defined, and handled in a timely manner by the Google team.
Smule overcame other device-specific roadblocks together, including errors with specific chipsets. As an example, when Oboe was asking for mono microphone input, a few devices provided stereo inputs mixed into one fake mono microphone input. Smule created a ticket in Oboe’s GitHub, providing examples and reproducing the issue using the Oboe tester app.
The Google-developed Oboe tester app was a helpful tool in solving and identifying issues throughout the implementation. It proved especially useful in testing many of the features of Oboe, AAudio, and OpenSL ES, as well as testing Android devices, measuring latency and glitches, and much more. The application offers a myriad of features that can help to simulate almost any audio setup. The Oboe tester can also be used in automated testing, by launching it from a shell script using an Android Intent. Smule relied heavily on the automation testing, given the large number of devices covered in the integration.
Once Smule was confident the device-specific issues were resolved and the Oboe audio was stable enough, Smule switched to a wider split testing rollout approach. In just a few weeks, Smule increased the population using Oboe from 10% to 100% percent of the successful devices, which was only possible due to the positive feedback and green KPI metrics Oboe received continuously throughout the release journey.
The results speak for themselves. Smule users on Oboe are singing more - it’s as simple as that. Unique karaoke recordings and performance joins, or duets, increased by a whopping 8.07%, Unique uploads by 3.84%, and song performances were completed by 4.10% more. Smule has observed in Q3 and Q4 2021 an increase of the recording completion rate by 10%+.
Using the Firebase Crashlytics tool by Google, Smule has seen a decline in audio-related crashes since the full Oboe release, making the app more stable - even on lower-end devices. Smule’s dedicated customer support team was thrilled to report a 33% reduction in audio-related complaints, including issues like (unintended) robot voice and echo.
The decision to switch to Oboe has paid off in spades. The application is functioning better than ever and Smule is well-equipped to face further advancements in audio and hardware with the updated technology. Most importantly, Smule users are happy and making music, which is what it’s all about.
