Introduction

In this study, we define Virtual Production as the use of Computer-Generated Imagery, and Game Engine technologies to display virtual content on large LED walls in real-time synchronized with camera tracking. Also referred in the industry as ICVFX (In-Camera-Visual-Effects), meaning that high-quality imagery and visual effects are captured live in-camera as the scene unfolds, as opposed to adding them subsequently in post-production. 

The advantages of this type of filmmaking are that the entire crew can see what the scenes look like, as well as the performers who may feel more immersed in the virtual world, instead of being surrounded by a massive green screen. This allows the pipeline of shooting to be more collaborative and cohesive between multi-cross departments during the production phases. Another key solution ICVFX offers is that the environments can be modified in real-time, and unpredictable weather forecasts can be disregarded since everything is happening inside of a controlled studio. On top of this, there is no need to travel entire crews and equipment to remote locations, saving budget costs and reducing carbon footprint.  

This study was meant to explore these advantages even further by trying to achieve final image comparable to blockbusters with only a relatively small LED volume (8m wide by 3.5m tall), an affordable cinema camera (Black Magic Ursa Mini Pro 12K), a tracking system (Vicon Motion Capture System), and some lighting equipment. All the equipment was outside of the budget for this project. However, fortunately enough, these were provided by the education institution and the commissioning company.  

LED volumes, camera, and camera tracking system utilized for this study. At the XR stage in Breda University of applied sciences, The Netherlands. 

Preparations

The preparations first took place by scouting for potential existing movie scenes that could be realistically reproduced utilizing Virtual Production. These scenes were purposely selected if they were either filmed on location, or if entire sets were built to film them, and no green screen or chroma keying techniques were utilized. 

Due to the limited resources, acquiring many props, actors, or make up was outside of reach. Therefore, the scenes that had to be chosen needed to include minimal elements in the foreground, the performance of the actors had to be minimalistic, and preferably, their faces should not be visible to avoid the necessity of make up, and to reduce the obviousness of not having access to the original actors. 

Once the selection was decided, a set of instructions along with the original footage of the scenes were given to a 3D artist to construct the environments inside of a game engine.  

Experimental Reproductions 

The productions were an experimental aspect in this study to facilitate the research with data collection, not a professional final product by themselves. These experimental productions were defined as “experimental reproductions” for the fundamental reason that they were planned to recreate an already existing movie scene.  

Results

After analysing the data, the findings of this study found that most of the reproduced scenes achieved a comparable result to the original. However, some of them did not, showing that either Virtual Production on low resources cannot achieve the desired final image in all scenarios, or that due to the insufficient experience of the filming crew the end result was not on par with the original quality.  

Top half: Experimental Reproduction. Bottom half: original scene from Arrival (2016), by Denis Villeneuve, Paramount Pictures. 

Top half: Experimental Reproduction. Bottom half: original scene from Arrival (2016), by Denis Villeneuve, Paramount Pictures. 

Recommendations 

The need for skilled 3D artists is highly important to ensure that the content displayed on the LED volumes meets the desired quality and fidelity. Additionally, time allocation on the pre-production phase is needed compared to traditional filmmaking. This means that time spent on the post-production phase shifts to the pre-production phase. The uncertainty that traditional filmmakers experience discussed in this research gets resolved. 

This study recommends a blend between traditional filmmaking and Virtual Production where wide and complex scenes should be filmed on location or in a physically built set, and narrower shots should be done with Virtual Production. Finally, Virtual Production also excels when the setting of the story requires a fictitious background that is challenging to find or achieve in real life. 

Conclusions 

Based on the positive results, a strong inclination towards the benefits that Virtual Production can potentially offer can be assumed. However, there are many factors to consider that will determine whether a scene will yield successful results; The complexity of the shot, the amount of camera movement, the level of action in the shot, the amount of foreground and background elements and if they have to be seamlessly blended with the virtual environment, the type of framing and camera angle, the level and amount of practical or visual effects in the scene, and many others.  

A safe practise that will guarantee a more successful result would be to choose scenes spanning from medium-full shots to close-up shots. This will avoid the burden of having to blend the physical floor or walls that would be seen in wider shots with the LED volume. Additionally, the use of real props plays a big role in the extension of the real and virtual world. The more levels of depth you can create by adding foreground elements combined with the virtual background will aid the visual composition and realism of the shot, as well as allowing the actors to interact with actual real items. Lastly, including camera movements is encouraged if the virtual environment is made for it. When doing so, this will create a sense of parallax to the viewer comparable to what we see in the real world and will enhance the believability of the scene. Extreme camera movements may be more difficult to manoeuvre due to the dimensions of the LED walls or the refresh rate of the camera tracking system, which might limit the amount and speed of the movement.