Many other metrics and constraints affect the final user experience and bitrate savings, such as the PSNR (Peak Signal-to-Noise Ratio) and bitrate constraint.

By comparison, board games tend to have a single known environment. For instance, the set of videos uploaded to platforms like YouTube varies in content and quality, and any agent needs to generalise across videos, including completely new videos after deployment. However, to get MuZero to work on this real-world application requires solving a whole new set of problems. This works especially well in large, combinatorial action spaces, making it an ideal candidate solution for the problem of rate control in video compression. MuZero achieves superhuman performance across various tasks by combining the power of search with its ability to learn a model of the environment and plan accordingly. Bitrate is a major determinant in how much compute and bandwidth is required to serve and store videos, affecting everything from how long a video takes to load to its resolution, buffering, and data usage. As with other codecs, service providers using VP9 need to think about bitrate - the number of ones and zeros required to send each frame of a video. Our initial focus is on the VP9 codec (specifically the open source version libvpx), since it’s widely used by YouTube and other streaming services. However, because RL is particularly well-suited to sequential decision-making problems like those in codecs, we’re exploring how an RL-learned algorithm can help. Decades of hand engineering have gone into optimising these codecs, which are responsible for many of the video experiences now possible on the internet, including video on demand, video calls, video games, and virtual reality. These codecs make multiple decisions for each frame in a video. Most online videos rely on a program called a codec to compress or encode the video at its source, transmit it over the internet to the viewer, and then decompress or decode it for playback.

Since launching to production on a portion of YouTube’s live traffic, we’ve demonstrated an average 4% bitrate reduction across a large, diverse set of videos. With video surging during the COVID-19 pandemic and the total amount of internet traffic expected to grow in the future, video compression is an increasingly important problem - and a natural area to apply Reinforcement Learning (RL) to improve upon the state of the art in a challenging domain. Analysts predicted that streaming video will have accounted for the vast majority of internet traffic in 2021. In a preprint published on arXiv, we detail our collaboration with YouTube to explore the potential for MuZero to improve video compression. Now, in pursuit of DeepMind’s mission to solve intelligence, MuZero has taken a first step towards mastering a real-world task by optimising video on YouTube. But so far these agents have focused on solving games. MuZero, for example, mastered Chess, Go, Shogi, and Atari without needing to be told the rules. Its successors, AlphaZero and then MuZero, each represented a significant step forward in the pursuit of general-purpose algorithms, mastering a greater number of games with even less predefined knowledge. In 2016, we introduced AlphaGo, the first artificial intelligence program to defeat humans at the ancient game of Go. Collaborating with YouTube to optimise video compression in the open source VP9 codec.

Youtube video compression