Netflix’s AI-Assisted Green Screen Lighting Technique Holds Potential for Seamless Compositing
Compositing, the process of placing actors in front of a background that doesn’t actually exist, has been a fundamental part of filmmaking for years. Netflix is now revolutionizing this technique using machine learning – but with one significant caveat. The innovative approach relies on lighting actors in an eye-catching magenta hue.
Chroma keying, a traditional method of compositing, involves actors standing against a brightly colored backdrop (initially blue, later green) that is easily replaced with any desired background during post-production. While this technique is cost-effective and straightforward, it has some limitations. Transparent objects, intricate details like hair, and anything with an identical color to the background can pose problems. Nevertheless, chroma keying has remained prevalent due to its affordability, despite alternative methods such as light field cameras being available.
Enter Netflix researchers, who are presenting a fresh take on compositing by combining existing and innovative techniques. Their approach, called Magenta Green Screen, involves placing actors in a distinctive lighting setup. Behind the actors, a vivid green screen is actively lit, while in front of them, a mix of red and blue lighting creates a striking contrast.
As a result, the on-set visual experience for actors and crew is far from ordinary. Typically, actors are illuminated with a natural-looking light, allowing for minimal adjustments during post-production. However, using solely red and blue lighting distorts their appearance because natural light contains the full spectrum of colors, which is absent from this magenta setup.
Nonetheless, this technique simplifies the process of separating the foreground and background. Instead of capturing the full spectrum of colors, a regular camera documents only red, blue, and alpha channels. Consequently, the resulting mattes are more accurate and free from artifacts typically encountered when separating a full-spectrum input from a limited-spectrum key background.
While the magenta-lit subjects may lose their natural color during filming, Netflix’s machine learning model comes to the rescue during post-production. By training a convolutional neural network on full-spectrum rehearsal footage and its corresponding magenta-lit counterparts, the researchers develop an intelligent method for seamlessly restoring the missing green channel.
The restored color is virtually indistinguishable from an in-camera ground truth, culminating in a successful post-production phase. However, the challenge lies in the unconventional lighting setup actors must endure on set. Working in front of a greenscreen already feels unnatural to many actors, and the addition of harsh, inhuman lighting exacerbates the issue.
To address this concern, the researchers propose time-multiplexing the lighting, which involves rapidly alternating between magenta and green lighting. Although switching the lighting 24 times per second, the typical film and TV frame rate, can cause distractions and even be hazardous, increasing the switching frequency to 144 times per second creates the illusion of nearly constant lighting.
Implementing this technique, however, necessitates intricate synchronization with the camera to ensure it captures light only during the brief magenta moments. Furthermore, compensating for motion-related missing frames adds another layer of complexity.
While this approach is still experimental, it showcases an innovative and high-tech solution to a longstanding challenge in media production. The viability of such practices wasn’t possible merely five years ago, and although it remains uncertain whether this technique will become widely adopted on set, the potential it holds makes it worth exploring further.
