If you are a Producer, Director, or crew member looking for ways to adapt in a post-Covid-19 world, chances are you’ve encountered the term virtual production. It became a household term for most of us after seeing either the behind-the-scenes video from The Mandalorian or Matt Workman’s Unreal Engine promo from SIGGRAPH. But what is virtual production? And why is a concrete definition hard to come by?

There are a couple VFX techniques that are used interchangeably with virtual production, but aren’t actually accurate. XR (extended-reality) and Volumetric Capture are the two most visible techniques that are often conflated with virtual production, but here’s what they each actually mean:

• XR – LED walls used as live backdrops for in-camera VFX in situations where, traditionally, green screen would be used and backgrounds composited in post-production. Epic Games’ Unreal Engine is, so far, the most widely used base for real-time rendered environments.
• Volumetric Capture – Uses LiDAR (or another scanning technology) to digitally gather the volume and depth information and create a 3D capture of a live performance.

What is virtual production? Here are a few responses I got when I asked experts in this space.

David Stripinis (22-year VFX veteran, Avatar, Harry Potter and the Deathly Hallows: Part 2, Man of Steel) gave me the best definition of virtual production I’ve heard so far, summing it up like this:

Virtual Production exists on a spectrum. It’s about the Venn diagram overlap between the physical world and the digital. You can push and pull whatever parts into or out of that overlap that you need/want/can afford. If you just want to get mocap or a virtual camera into a full CG scene running live in MotionBuilder, that’s virtual production.  If you want to live track a camera on a chroma stage and do live comp on viewfinder or video village, that’s virtual production. If you want to run full LED panels to give rear projection against live actors, that’s virtual production.

The programs, the hardware, will all continue to evolve. The core concept of virtual production is to bring the immediacy and interactive nature of the real world into CG, and bring the feedback of the CG world to key creative stakeholders so they can make informed decisions. That often means directors or DPs, but also art directors, actors, editors, script supervisors and even composers. 

When I asked Bryan Dodson of Integrated Visions about the confusion around terminology and to help me define, “what is virtual production?” here’s what he had to say:

Well, I think they’re all (XR, virtual production, Volumetric Capture) interchangeably used to mean the same thing, but I think they are often used incorrectly. The term virtual production includes XR but is also inclusive of previsualization and real time compositing of green screen shots. From my understanding, Volumetric Capture uses LiDAR technology to create a 3D capture of a performance. Simple Volumetric Capture can be done with a Kinect. 

The term XR was heavily marketed by Disguise. It basically means the same thing as what the Unreal virtual production Field guide calls “virtual production with a live led backdrop”.

Communicating about this technology clearly is definitely one of the big challenges.

Virtual production utilizes a digital pipeline for multiple production departments NOT just post-production or VFX. Think previsualization, Production Designers building digital sets, Costume Designers creating digital wardrobe, motion capture of stunt performers, virtual units for Lighting, Camera Operators moving CG cameras, or any other department that could benefit from being connected to the digital pipeline.

So, when people say that The Mandalorian, Avatar, or Oblivion used virtual production techniques, they might not necessarily be wrong. But when you’re watching the behind-the-scenes video of The Mandalorian being shot against an LED wall, but the other elements are all practical, calling it a virtual production simply isn’t accurate. It’s more like a hybrid production. XR and Volumetric Capture are certainly components of virtual production, but to use them all as interchangeable terms would be inaccurate.

Virtual production is a term whose definition is evolving as rapidly as the technology used to implement it. I’ll continue to post new updates as terms and definitions become more ubiquitous and widely adopted. In the meantime, here are some resources if you’re interested in educating yourself further on virtual production:

• Unreal Engine Virtual Production Facebook Group
• Virtual Production of The Mandalorian
• Epic Games SIGGRAPH Promo
• Introduction to Unreal Engine & Volumetric Capture
• Unreal Engine Field Production Guide

Enjoying what you’re reading? Sign up.

Join monthly readers. Get exclusive content, and more. No spam, ever. Just random film industry ramblings from a guy in the industry.

Hi, I’m Sam! I’m currently the Head of Production at Crafty Apes VFX in Louisiana. I’ve also produced classic low budget movies you’ve never heard of like Ozark Sharks, Desolation, and A Deadly Affair through my production company Fable House.