Video quality matters, and when an HD or HFR playback isn’t smooth, we notice. Chrome noticed. YouTube noticed. So we got together to make YouTube video playback smoother in Chrome, and we call it Project Butter.
For some context, our brains fill in the motion in between frames if each frame is onscreen the same amount of time - this is called motion interpolation. In other words, a 30 frames per second video won’t appear smooth unless each frame is spaced evenly each 1/30th of a second. Smoothness is more complicated than just this - you can read more about it in this article by Michael Abrash at Valve.
Frame rates, display refresh rates and cadence
Your device’s screen redraws itself at a certain frame rate. Videos present frames at a certain rate. These rates are often not the same. At YouTube we commonly see videos authored at 24, 25, 29.97, 30, 48, 50, 59.94, and 60 frames per second (fps) and these videos are viewed on displays with different refresh rates - the most common being 50Hz (Europe) and 60Hz (USA).
For a video to be smooth we need to figure out the best, most regular way to display the frames - the best cadence. The ideal cadence is calculated as the ratio of the display rate to frame rate. For example, if we have a 60Hz display (a 1/60 second display interval) and a 30 fps clip, 60 / 30 == 2 which means each video frame should be displayed for two display intervals of total duration 2 * 1/60 second.
We played videos a bunch of different ways and scored them on smoothness.
Smoothness score
Using off the shelf HDMI capture hardware and some special video clips we computed a percentage score based on the number of times each video frame was displayed relative to a calculated optimal display count. The higher the score, the more frames aligned with the optimal display frequency. Below is a figure showing how Chrome 43 performed when playing a 30fps clip back on a 60Hz display:
Smoothness: 68.49%, ~Dropped: 5 / 900 (0.555556%)
The y-axis is the number of times each frame was displayed, while the x-axis is the frame number. As mentioned previously the calculated ideal display count for a 30fps clip on a 60Hz display is 2. So, in an ideal situation, the graph should be a flat horizontal line at 2, yet Chrome dropped many frames and displayed certain frames for as many as 4 display cycles! The smoothness score reflects this - only 68.49 percent of frames were displayed correctly. How could we track down what was going on?
Using some of the performance tracing tools built into Chrome, we identified timing issues inherent to the existing design for video rendering as the culprit. These issues resulted in both missed and irregular video frames on a regular basis.
There were two main problems in the interactions between Chrome’s compositor (responsible for drawing frames) and its media pipeline (responsible for generating frames) --
- The compositor didn’t have a timely way of knowing when a video frame needed display. Video frames were selected on the media pipeline thread while the compositor would occasionally come along looking for them on the compositor thread, but if the compositor thread was busy it wouldn’t get the notification on time.
- Chrome’s media pipeline didn’t know when the compositor would be ready to draw its next new frame. This led to the media pipeline sometimes picking a frame that was too old by the time the compositor displayed it.
In Chrome 44, we re-architected the media and compositor pipelines to communicate carefully about the intent to generate and display. Additionally, we also improved which video frames to pick by using the optimal display count information. With these changes, Chrome 44 significantly improved on smoothness scores across all video frame rates and display refresh rates:
Smoothness: 99.33%, ~Dropped: 0 / 900 (0.000000%)
Dale Curtis, Software Engineer, recently watched WARNING: SCARIEST GAME IN YEARS | Five Nights at Freddy's - Part 1
Renganathan Ramamoorthy, Product Manager, recently watched Video Game High School