Netflix Taps AI To Push Visual Effects Beyond the Green Screen

Researchers at Netflix say they may have rendered the omnipresent green screen obsolete. Described as an innovative advancement in the application of AI in the film and television industry, the Magenta Green Screen (MGS) leverages the power of artificial intelligence to enhance visual effects, making them more realistic and precise in real time.

The streaming giant’s new technology is an impressive addition to its diverse arsenal, which already employs deep learning to tailor user experience. Utilizing AI, Netflix made a name for itself in successfully profiling its users and suggesting content to viewers. AI has also helped Netflix create engrossing highlights, recaps, and trailers to increase viewership.

Green-screen technology is a widely-used process that involves filming actors against a bright green background, which can later be digitally removed and replaced. The process can be automated, but issues arise when green elements appear in the foreground or when fine details like hair strands or transparent objects are involved. To address these issues, operators often manually adjust settings or paint over each frame, a time-consuming process.

Netflix says MGS uses AI to achieve better results than legacy green screens in a fraction of the time that it usually takes to edit a scene.

How Netflix uses AI to improve its VFX game

The MGS method calls for filming actors against bright green LEDs, with red and blue LEDs illuminating them from the front, resulting in a magenta glow. This unique lighting configuration creates separate red, green, and blue channels. The green channel only records the background, making the foreground appear black, while the red and blue channels only capture the foreground, rendering the background black.

The introduction of AI into this system allows for the green channel to be replaced in real-time, which means actors can be realistically and instantly placed in the foreground of another scene. More remarkably, this AI technology can work accurately with transparent objects and intricate details like individual hair strands, which have been challenging with traditional methods.

The application of AI doesn’t stop at the filming stage. Since the MGS technique can result in magenta-tinted actors and objects, Netflix also employs AI to restore the full spectrum of color in the foreground. This is accomplished by using a reference photograph of the actors lit normally.

Netflix is just one of the many businesses riding the AI wave. Other startups and unicorns are leveraging the use of AI tech to better profile users, provide better customer experiences and optimize their products.

As Decrypt recently reported, Zoom developed a GPT-powered meeting summarizer. Also, Meta is using generative AI to create better ads, Google is improving its office suite thanks to its own LLM, and even The US Department of Defense is trying to come up with interesting use cases for AI as a strategic tool.

Despite the complexities associated with the MGS technique, Netflix believes in the potential of its AI-powered GS system. However, others in the industry have reservations.

Drew Lahat from video production company Geiger Post raised concerns about the practicality of the technique in fast-paced shooting environments and its competitiveness with other emerging methods. “It may work well in a fully controlled space,” he said, “but it would have to compete with other new techniques like virtual production stages, and win over producers in real-world scenarios.”

Nevertheless, as Netflix researcher Paul Debevec told New Scientist, “Computers already have provided such powerful tools to make a lot of stuff easier. [This is] another thing that we can make easier, so that the talented artists that we have can focus on the artistry, actually making things look better.”

Netflix is betting on AI to enhance not only viewers’ experiences but also to streamline things behind the scenes. Writers may hate AI sneaking into their businesses, but do VFX artists feel the same way?