Introduction

Artificial Intelligence (AI) has sparked interest in its many potential applications. This intrigue and application have been highlighted to have an influence on the film industry (Ewis et al., 2024, Translated with Claude AI and cross-referenced with DeepL Translate). In this report, we will explore a few AI tools used in connection with the in-camera visual effects (ICVFX) filming technique.  ICVFX is the use of LED volumes to combine real-life sets with virtual environments to produce a final image (Leininger et al., 2025). AI has been suggested as a potentially useful tool for the creation of these virtual environments (Nina Willment et al., 2023; Zwerman & Okun, 2023). As researched by Hilal (2025) virtual environments can also be used for multiple other purposes in the ICVFX workflow, such as scouting, concepting, previsualisation, and techvisualisation during pre-production. As shown in Figure 1, Hilal (2025) clearly suggests that pre-production is its prominent use case in filmmaking. This report will therefore also aim to explore the use of AI tools in the workflow and at which

Text-to-Image and Image-to-Image AI Tools

The virtual environment can currently be 2D, 2.5D or 3D. 2D is the lightest type of load and uses 2D images or videos as a background. Whilst 3D is heavier on the production process (Zwerman & Okun, 2023). Text-to-image or image-to-image tools can be useful for the creation of 2D virtual environments for ICVFX (Nina Willment et al., 2023). These tools could help reduce the need for technical knowledge, allowing filmmakers to experiment with virtual environments more easily (Hilal, 2025; Leininger et al., 2025; Song et al., 2023). Furthermore, these tools have the potential to enhance human-centred production design by also increasing efficiency in early discussions. Especially in translating the intended visual aesthetic to the ICVFX studio or potential stakeholders. Designers can create a variety of virtual environment scenarios to test on the LED volume, allowing for a more informed decision-making process in pre-production (Hilal, 2025).

Text-to-image tools generate images from text. Image-to-image tools generate the same output but use images as references rather than text. Ewis et al. (2024) and Leininger et al. (2025) list some of these tools stated as relevant for ICVFX:

• DALL-E
• Midjourney
• Cuebric
• Blockade Labs and Skybox
• Stable Diffusion and ControlNet

Dall-E and Midjourney

Both Dall-E and Midjourney are prompt-based AI systems for image generation. Dall-E is ChatGPT’s image-generation model. It is a neural network or a system that is similar to the human brain that generates something (an image) based on a descriptive text prompt (Ewis et al., 2024). An example of this image generation is shown in Figure 1. 

In Midjourney, the process involves writing a very detailed description of what should be generated before clicking on ” Imagine ” to generate an image (Ewis et al., 2024).  With Midjourney, users send prompts to a community feed, share the outputs, and create daily AI trends, whereas with Dall-E, users use an individual prompt box to submit their idea. Both AI tools can be used to generate visual concepts and final environments for the LED volume (Liu et al., 2022; Nina Willment et al., 2023). The imagery that is created in the concepting phase using AI can and is often replaced in the production phase by real photographs of a location, modelled assets, a Gaussian splat scan, a kit-bashed environment, among others (Hilal, 2025). AI environments are often replaced before production, according to D. Geurts (personal communication, December 8, 2025), for quality reasons. Furthermore, there are ethical and legal challenges to using AI-generated environments in production. These challenges, according to Ewis et al. (2024) encompass the issues of attribution/ownership, privacy and manipulation or deception.

However, according to D. Geurts (personal communication, December 8, 2025), some ICVFX productions have used AI-generated environments as the final filming environment, though Geurts states that this always results in a compromise on set. It is harder to make last-minute changes to the generated environment during filming, as a completely new image must be created if changes are required. D. Geurts (personal communication, December 8, 2025) mentions that “you can’t edit it [AI environment] anymore on the screen [LED volume] in the normal pipeline we have right now.” It was a cheaper option for this production because it was less labour-intensive, but it offered less control over the final environment and raised some ethical concerns that needed to be considered.

Cuebric

Cuebric states that it is designed to generate environments and can segment them for use as a virtual environment in ICVFX (Ewis et al., 2024; Rojas & Martínez-Cano, 2024).  However, according to D. Geurts (personal communication, December 8, 2025), “the biggest upside that we [Ready Set Studios] thought it [Cuebric] had is that it would be quite tailored to virtual production [ICVFX].” Testing and research showed that the  limitations of the software outweighed its potential benefits. D. Geurts (personal communication, December 8, 2025) stated that for ICVFX, detail is a requirement; Cuebric had the option to upscale resolution, but it did not necessarily provide extra detail in the image. D. Geurts (personal communication, December 8, 2025) further mentioned that it was “just another stable diffusion model.”

Skybox AI and 360 Environments

360-degree environments, whether generated by AI or captured with tools such as the Insta360 camera, are another option for an ICVFX background. This is also categorised as a 2D environment. Teixeira et al. (2024, p. 410) state that 360 2D environments “offer immersive environments without the computational demands of real-time rendering.” Although Teixeira et al. (2024) discussed this in the context of VR or immersive experiences, it remains relevant to LED volume filmmaking, as Unreal Engine 3D environments used in ICVFX need to run in real time. Running a 3D environment in real time on an LED volume allows filmmakers to make live changes on set (Zwerman & Okun, 2023). However, as also mentioned by Zwerman & Okun (2023, chapter. 18)

One limitation of real-time environments is often scene performance, which is a cross-section of the number of complex objects, textures, and other scene variables. This often demands that time is spent on optimization to get the best quality and on-set performance – particularly when striving for photorealism

Depending on the project’s budget, the time for optimisation may be limited; therefore, using 360 AI or photo environments instead of real-time 3D environments may be more cost-efficient (D. Geurts, personal communication, December 8, 2025). 360 images, whether AI-generated or not, offer more camera-angle options than a normal 2D image without the restrictions of real-time rendering.

There have been some text-to-image AI tools that have been made to support the creation of 360-degree LED volume backgrounds, as well as workflows for combining AI-generated images to create a 360-degree environment (Leininger et al., 2025; Song et al., 2023; Zhang et al., 2024).

Blockade Labs created an AI system called Skybox AI, which generates 360-degree panoramas (Marguerite Decourcelle, 2018).

Skybox AI presents a noteworthy option amidst the landscape of AI-driven 360 panorama generation tools. Its algorithmic efficiency and optimized workflows facilitate the swift creation of high-fidelity panoramas, effectively conserving computational resources and time for XR professionals (Teixeira et al., 2024, p. 411).

These panoramas were used in a study from Leininger et al. (2025) to explore the effects on previsualisation, prototyping, final production use, and production efficiency. Firstly, they used the 2D-generated panoramas “mapped onto a deformed sphere” (Leininger et al., 2025, p.66). The 360-degree environments generated were converted into 3D meshes using the same program. These 3D versions were described as “effective in outdoor settings … sparking creativity and moodboards/concept development” and Skybox AI was “appreciated for its ability to quickly generate environments and support early-stage production” (Leininger et al., 2025, p.67). However, due to its lack of realism, it would be less likely to be used in the final production in its current state. In another study by Song et al. (2023), they also experimented with using Skybox AI among other tools to create 360 AI-generated images. They used these tools to create a 360 high-dynamic-range image (HDRI) from different exposures. The HDRI 360 AI-generated environment was used as the background for filming with the LED volumes and was “offline” and not in “real-time” (Song et al., 2023), which imposed some limitations on camera movement. During filming, the focus was on the faces of the actors, with a shallow depth of field to minimise background details. In this use case, it was described as “effective” in replacing non-AI environments. Although described as effective, Song et al. (2023) also suggest that more research is needed to explore the use of AI in creating highly realistic filming environments.

Teixeira et al. (2024) describe Skybox AI as a useful entry point for those who are less technical. D. Geurts (personal communication, December 8, 2025) also claimed that AI tools may be more cost-efficient at the technical level. It allows individuals to quickly learn about environments for ICVFX and what works and what does not. Whilst it may be a useful entry point, Teixeira et al. (2024) noted several limitations of the Skybox AI tool:

• Misalignments in the generated 360 content
• Some inconsistencies between colours in the image
• Struggles in handling more abstract prompts
• Limited creative control of the generated 360 environment
• The quality of the output depended on the detail given in the initial prompt

2D to 3D with Stable Diffusion AI

2D imagery can be converted to 3D. One method found is through the program Skybox AI for 360 panoramas, which can also convert environments to 3D (Marguerite Decourcelle, 2018), another is utilising Stable Diffusion’s Automatic1111 WebUI Depth Extension (AUTOMATIC1111, 2022/2022; Thiry, 2022/2025). This aims to analyse depth information in a photograph. “The process essentially ‘stretches’ the flat image into a 3D space, creating a scene with visible depth and perspective” (Leininger et al., 2025, p.65). “The outputs of the script can be viewed directly or used as an asset for a 3D engine” (Thiry, 2022/2025). In the paper from Leininger et al. (2025) they experimented with this technique. The findings discuss that, though beneficial for low-fidelity tasks, the 3D version of the 2D image appears distorted and lacks depth. Be that as it may, it appears to be a viable option for exploring in pre-production.

AI Tools for 3D Model Creation

Real-time programs such as Unreal Engine (UE) use AI to support workers in ICVFX and other industries, such as animation, by allowing them to view their work in real time without waiting for each render (Nina Willment et al., 2023). A 3D environment in UE used as a virtual environment is often composed of assets or models, and the asset modelling process is particularly time-consuming (Mrigyisha Sawant, 2025). Real-time assets for ICVFX “are required to work under technical limitations and in multi-purpose situations where they can be moved, re-lit, and seen from a variety of angles – and on top of that, they will still be judged based on their visual fidelity” (Zwerman & Okun, 2023, chapter. 7). There have been advancements in generative AI tools that have been developed with efficiency in mind to support this process (Silva Jasaui et al., 2024).

LumaLabs ai

LumaLabs AI has both web-based and UE-based tools for asset and environment creation. The web-based asset creation tool is called Luma AI Genie; the UE AI plugin is called Luma AI. Luma AI is discussed in the Gaussian splatting subchapter.

Genie

Genie is a text-to-3D generative AI web-based tool that takes a prompt and outputs four textured models in a few seconds. After the generation of the four low-resolution models, there is an option to increase the resolution of a selected model, as highlighted in Figure 3 (Mrigyisha Sawant, 2025).

Genie is currently a promising tool for concepting or low-resolution asset requirements. It also provides an accessible, free solution compared to options such as Meshy (Begemann & Hutson, 2024; Mrigyisha Sawant, 2025). Nevertheless, it still comes with significant limitations, such as “overly complex geometries” that make them impractical to work with, for example, in animation (Begemann & Hutson, 2024, p.12).

Meshy AI

Meshy AI, like LumaLabs Genie, uses prompt-based asset generation. The prompt can be either text- or image-based and is considered ICVFX-ready (Ethan (Yuanming) Hu, 2021). Meshy, much like Genie, outputs four generations; however, these generations are initially untextured. There is then the possibility to manually texture or texture in the program (Mrigyisha Sawant, 2025). Meshy AI also allows exporting the generated asset in multiple file formats, making further work easier in programs such as Blender (Nurgazy et al., 2025). Compared to Genie, Meshy AI showed higher performance in areas such as refinement, realism, complex model work, texture and mesh quality, and geometric precision (Mrigyisha Sawant, 2025).

D. Geurts (personal communication, December 8, 2025) mentioned they “mostly use these 3D modeling tools for very quick […] pre lighting phase or last phases of production. And we just quickly need another object that wasn’t planned for.” However, one limitation noted by Geurts was that Meshy AI was slower than their current preferred alternative, Tencent Hunyuan 3D, though it was still considered a valuable option for ICVFX.

Tencent Hunyuan 3D

Tencent Hunyuan is both a text and an image-conditioned AI tool that generates 3D assets  (Yang et al., 2024).

“It is mainly built on two fully open-source foundation models: 1) Hunyuan3D-DiT: A shape-generation model combining a flow-based diffusion architecture with a high-fidelity mesh autoencoder (Hunyuan3D-ShapeVAE); 2) Hunyuan3D-Paint: a mesh-conditioned multi-view diffusion model for PBR material generation, producing high-quality, multi-channel-aligned, and view-consistent textures.” (Hunyuan3D et al., 2025, p. 2)

D. Geurts (personal communication, December 8, 2025) mentions that using Tencent Hunyuan, they “can just give four pictures of an object and in five minutes, have a mesh, a model that’s more than good enough to be in the 3D scene [for ICVFX], not as a hero model, but, on the side or behind something, it’s super well made.”

Gaussian Splatting and NerF Tools

Gaussian splatting is an efficient technique for 3D scene reconstruction and rendering. It is mentioned that, unlike previous methods such as NeRFs that used complex meshes or mathematical fields, Gaussian splats use Gaussian blobs or primitives, avoiding extra computing time to calculate the empty space that is not visible (Azzarelli et al., 2025; Collier, 2025; Frido Buunk & Guido Elzenaar, 2025). Gaussian splatting is mentioned as “the first method to achieve high visual quality in unbounded scenes while maintaining fast training and real-time rendering” (Kerbl et al., 2024, para. 10). Real-time rendering speeds for environments designed for ICVFX are important for displaying content at the same speed as the camera, at 24 frames per second or faster (Kadner, 2019).

PortalCam

PortalCam for XGrids has been highlighted as a technological advancement that is built around this gaussian splatting technique for the benefit of the virtual production industry, among others.

(Collier, 2025; Shawn Herold, 2025). Although not technically AI, it does have an AI processing unit, which is why it is included in this report. It combines LiDAR, which scans an environment using lasers, measuring the reflected light to derive a point cloud, with an AI processing unit to collect data automatically and multiple cameras to increase accuracy (Frido Buunk & Guido Elzenaar, 2025; Zwerman & Okun, 2023).

“The PortalCam outputs ready-to-use, high-quality 3D models that retain fine detail while reducing file sizes by up to 90%. It performs reliably in complex environments, handling dark areas, reflective surfaces, high-contrast scenes, and structures with minimal features” (Collier, 2025, para. 18). This process is done through automated model generation using Xgrids Lixel CyberColor (LCC). One of the main benefits of the PortalCam environments is the realism; supposedly providing a “hybrid approach, combining real-world capture with real-time rendering” (Shawn Herold, 2025, para. 8). One recent use of the PortalCam was for the MiG-21 Project a “5-year, multidisciplinary project that transforms a 51-foot by 24-foot decommissioned Cold War era, Soviet-designed MiG-21 fighter jet into a stunning work of art, entirely covered in tens of millions of colorful glass beads” (Ralph Ziman, 2019). The work from Paolo Tosolini, among others, captured this artwork, turning it into a 3D interactive online experience. Although not a project involving an LED volume, it highlights the potential of the PortalCam.

Volinga AI

Volinga AI is a UE-paid plugin that supports importing 3D Gaussian splat models into Unreal Engine. The AI-based plugin also opens creative possibilities for models scanned with the PortalCam, such as relighting, culling, or modifying parts of the environment. It also implements ACES colour management support and ICVFX feature support. (Shawn Herold, 2025; ‘Volinga × XGRIDS’, 2025).

Luma ai

Luma AI is also a UE plugin. It enables users to download NerF captures or Gaussian Splats called luma (.luma) fields or an interactive (splat) scene and use it in UE as generated blueprints (Luma Unreal Engine Plugin (0.41), n.d.). Essentially, Luma AI makes it possible to use these “volumetric captures” in UE and use them as environments (Gainz, 2024; Jim Thacker, 2023).

Other AI Tools

AI-powered Volumetric Diffusion Plugin

In studios filming in an LED volume, especially in educational settings, using a fog machine is often impractical or prohibited. A potential solution to this problem is Grayscale Labs’ Nano plugin for DaVinci Resolve, released in 2025. “Nano is a highlight-driven, depth-based plugin that creates virtual haze in a scene” (Allard ACS, 2025, para. 1). From the recorded footage, it makes a depth map estimation using AI, based on that estimation, it calculates where the light source comes from to add the light diffusion that is seen traditionally through haze (Joey Daoud, 2025). The haze also shifts in real-time, reacting to objects and is entirely controllable with “length, spread, decay, luminance, obstruction, CCT, and more pro controls” (Mange & Zeas-Sigüenza, 2025).

Conclusion

Throughout this report, it is evident that some AI tools are used in the ICVFX industry, and that these tools are predominantly active in the concepting and pre-production phases of the workflow, with some entering the production and post-production phase. Image-based generation, both 360 and standard imagery, appears to be beneficial for concepting and valuable for client communication; however, realism remains a limitation for it entering the production phase.  Asset generation tools are useful for last-minute background assets and could help visual artists meet tight deadlines or budget constraints. Gaussian splatting tools, such as the portal camera, may redefine the method of creating 3D environments. They provide the ICVFX industry with a fast and realistic option. Furthermore, tools like Nano can address practical problems, such as the inability to use haze in an ICVFX studio. Finally, it is important and necessary that ownership issues be considered when using an AI tool.

One thing is certain: this area is largely experimental, and this landscape is likely to continue to change and develop in the coming years.  As tools evolve, research should further explore the use of AI in the production workflow and continue to examine how ethical concerns are addressed.