Posted by Andrew Lewis - Software Engineer, Android Media Solutions
The creation of user-generated content is on the rise, and users are looking for more ways to personalize and add uniqueness to their creations. These creations are then shared to a vast network of devices, each with its own capabilities. The Jetpack Media3 1.0 release includes new functionality in the Transformer module for converting media files between formats, or transcoding, and applying editing operations. For example, you can trim a clip from a longer piece of media and apply effects to the video track to share over social media, or transcode media into a more efficient codec for upload to a server.
The overall goal of Transformer is to provide an easy to use, reliable and performant API for transcoding and editing media, including support for customizing functionality, following the same API design principles to ExoPlayer. The library is supported on devices running Android 5.0 Lollipop (API 21) onwards and includes device-specific optimizations, giving developers a strong foundation to build on. This post gives an introduction to the new functionality and describes some of the many features we're planning for upcoming releases!
Getting Started
Most operations with Transformer will follow the same general pattern:
- Configure a TransformationRequest with settings like your desired output format
- Create a Transformer and pass it your TransformationRequest
- Apply additional effects and edits
- Attach a listener to react to completion events
- Start the transformation
Of course, depending on your desired transformations, you may not need every step. Here's an example of transcoding an input video to the H.265/HEVC video format and removing the audio track.
// Create a TransformationRequest and set the output format to H.265
val transformationRequest = TransformationRequest.Builder().setVideoMimeType(MimeTypes.VIDEO_H265).build()
// Create a Transformer
val transformer = Transformer.Builder(context)
.setTransformationRequest(transformationRequest) // Pass in TransformationRequest
.setRemoveAudio(true) // Remove audio track
.addListener(transformerListener) // transformerListener is an implementation of Transformer.Listener
.build()
// Start the transformation
val inputMediaItem = MediaItem.fromUri("path_to_input_file")
transformer.startTransformation(inputMediaItem, outputPath) |
During transformation you can get progress updates with Transformer.getProgress. When the transformation completes the listener is notified in its onTransformationCompleted or onTransformationError callback, and you can process the output media as needed.
Check out our documentation to learn about further capabilities in the Transformer APIs. You can also find details about using Transformer to accurately convert 10-bit HDR content to 8-bit SDR in the "Dealing with color washout" blog post to ensure your video's colors remain as vibrant as possible in the case that your app or the device doesn't support HDR content.
Edits, effects, and extensions
Media3 includes a set of core video effects for simple edits, such as scaling, cropping, and color filters, which you can use with Transformer. For example, you can create a Presentation effect to scale the input to 480p resolution while maintaining the original aspect ratio, and apply it with setVideoEffects:Transformer.Builder(context)
.setVideoEffects(listOf(Presentation.createForHeight(480)))
.build() |
You can also chain multiple effects to create more complex results. This example converts the input video to grayscale and rotates it by 30 degrees:
Transformer.Builder(context)
.setVideoEffects(listOf(
RgbFilter.createGrayscaleFilter(),
ScaleToFitTransformation.Builder()
.setRotationDegrees(30f)
.build()))
.build() |
It's also possible to extend Transformer’s functionality by implementing custom effects that build on existing ones. Here is an example of subclassing MatrixTransformation, where we start zoomed in by 2 times, then zoom out gradually as the frame presentation time increases:
val zoomOutEffect = MatrixTransformation { presentationTimeUs ->
val transformationMatrix = Matrix()
val scale = 2 - min(1f, presentationTimeUs / 1_000_000f) // Video will zoom from 2x to 1x in the first second
transformationMatrix.postScale(/* sx= */ scale, /* sy= */ scale)
transformationMatrix // The calculated transformations will be applied each frame in turn
}
Transformer.Builder(context)
.setVideoEffects(listOf(zoomOutEffect))
.build() |
Here's a screen recording that shows this effect being applied in the Transformer demo app:
 |
For even more advanced use cases, you can wrap your own OpenGL code or other processing libraries in a custom GL texture processor and plug those into Transformer as custom effects. See the demo app for some examples of custom effects. The README also has instructions for trying a demo of MediaPipe integration with Transformer.
Coming soon
Transformer is actively under development but ready to use, so please give it a try and share your feedback! The Media3 development branch includes a sneak peek into several new features building on the 1.0 release described here, including support for tone-mapping HDR videos to SDR using OpenGL, previewing video effects using ExoPlayer.setVideoEffects, and custom audio processing. We are also working on support for editing multiple videos in more flexible compositions, with export from Transformer and playback through ExoPlayer, making Media3 an end-to-end solution for transforming media.
We hope you'll find Transformer an easy-to-use and powerful tool for implementing fantastic media editing experiences on Android! You can send us feature requests and bug reports in the Media3 GitHub issue tracker, and follow this blog to get updates on new features. Stay tuned for our upcoming talk “High quality Android media experiences” at Google I/O.
- Trevor Mulchay, Display & Video 360 API Team
- Nikita Agarwal, DCM API Team
- Evan Chou, DCM API Team
