Q&A: A VP Specialist’s AI Tools

AI has become an ever-growing part of the conversation across multiple industry sectors, including media, gaming, publishing, and marketing, to name a few. Daan Geurts addresses this topic for one part of the media sector, the use of AI in in-camera visual effects (ICVFX). Daan Geurts unpacks his experience with AI in his work as a virtual production specialist at Ready Set Studio in Amsterdam, the Netherlands. With a wealth of on-set and offset experience, he covers the AI tools he and ReadySet have used, what works and what does not, changes to the workflow, ethical considerations and the effect of AI on their hiring process and future.

Q1. Could you walk me through the AI tools ReadySet Studios currently uses and how you’ve used them?

Midjourney: We use Midjourney mostly for concepting and quickly understanding vague client feedback. Clients can be very vague about what they want, and we sometimes want more context. So, we generate images to help us understand the client’s vision. We’ve also used MidJourney to generate images that we use as input for other AI tools. If we don’t have a picture or something we want to create, but we are using an image to 3D AI tool, then MidJourney helps us with that. Essentially, we use Midjourney to generate that starting image.

Meshy: Meshy, we have used to create meshes and textures from text or image prompts.

Runway: Like Midjourney, we also used Runway for the concepting phase. Runway was the first pretty stable video generation tool that we used. It enabled us to get video from our 2D images. When the client doesn’t have a clear idea of the camera movement or the transition in the film, we use Runway to provide context for what we think it should be to help the client and to get more aligned feedback.

XGrids Portal Camera: We use it for Gaussian splats. It is our new 2.5D device. If a client wants a background and doesn’t need to move the camera much during production, the portal camera is the perfect option. We also want to use it as a sales tool, as it is very cost-effective for generating a quick background that still works with a camera. We would like to create a library of scanned backgrounds for clients to browse. Then we can more easily discuss the budget for environment creation.

Volinga: We have used Volinga a bit to edit Gaussian splats, but it is mostly used for relighting and combining them in Unreal Engine.

GaussianEditor: We have used the Gaussian Editor to edit splats, just to remove parts from them. However, we now mostly use SuperSplat.

SuperSplat: A Gaussian splat editor we use. Although I am unsure if it uses AI.

Nano Banana: Nano Banana is predominantly used during the concepting and editing phases of making plates for the virtual environment. Nano Banana is very good at changing a part of an image rather than the whole thing. It doesn’t regenerate the whole image from scratch again with the next iteration.

Marble from World Labs AI: We use Marble from World Labs to generate a Gaussian splat from images and text prompts. It is a great way for clients with a smaller budget to access a 3D environment they otherwise couldn’t afford.

Photoshop: We use Photoshop for its quick separation tools, not really for Firefly.

SAM3D from Meta: It does a few different things, such as image separation and video object separation using a mask. You can also generate a 3D model from images or videos. Although I have only played around with it a little.

Tencent Hunyuan 3D: We use Tencent Hunyuan to quickly create 3D models during the pre-lighting phase or the final phases of production. It is mainly if we need an object that was not planned for. It is not used as a hero model, but it is often good enough to be on the side or behind something.

Q2. Can you outline the strengths and limitations of each AI tool that you have mentioned using at ReadySet Studios?

Midjourney: Midjourney is super strong at generating concepts or generating images from the context that you’ve given it, so it does a good job of translating what you have in your head into an image. One of its limitations is that Midjourney’s creativity stops at a certain point. If you prompt it twice with the same text, it will often output the same image twice. That is often the case with many AI tools.

Meshy: Meshy is super-fast at generating 3D models, and its key strength is producing good-quality textures and iterative textures compared to other AI tools. For example, we used it in a production set in a Chinese store, and we needed 20 different textures for a plate we had modelled for the virtual environment on the LED volume. Meshy did that very well. However, it can provide some unexpected results, so you cannot rely on it for a quick result. It is also not as good as Tencent Hunyuan in generating the models, only the textures.

Runway: The strength of Runway was that, when we started using it, it was pretty stable and the first of its kind for our company. However, as of now, we believe other AI software has surpassed it in video generation accuracy. The problem with Runway is its ability to generate a character’s emotion; often, it guesses, and most of the time it is wrong.

XGrids Portal Camera: The nice part of the portal camera is that it provides an expected result, so you do not need to repeat the process. That’s a real strength. So, when you scan a location with the camera, if the parameters you set are good (good coverage of the location, no harsh shadows or lighting, and you run the scan a few times in a loop), it will often produce a good result. One issue is that when you give the scan to the software, you cannot input anything, so you have to trust its internal process. That said, it does often go as expected. There is also a lack of feedback during scanning. If a scan does not look good after processing, it will likely never look good or work, and it is difficult to find out why. Also, if you edit the Gaussian splat, then there is always a black hole underneath what you remove.

GaussianEditor: Its strengths are that it is a quick Gaussian splat editor and does what you expect, but SuperSplat is more user-friendly.

SuperSplat: Its main strength is its user-friendliness.

Nano Banana: The strengths of Nano Banana are that it is super stable, so it can work quickly on iterations. However, its limitations are that it can get quite stuck on a previous image. For example, if you just want to change the perspective of an object, it seems to want to keep the object in the same place as before.

Marble from World Labs AI: It provides the same possibility for movement that 3D does, but you lose out on detail. It works very well from the perspective that you have given it, but it is not stable with the architecture. For example, if it creates a lamp post, it may create two next to each other.

Photoshop: Photoshop has an excellent integration of its tools. I like that they do not overwhelm you with a new AI Photoshop but instead use AI to strengthen the existing tools. There are no noted limitations for Photoshop.

SAM3D from Meta: Its biggest strength is its ability to generate an object from within an image. Generally, with other tools, you have to separate the object from the image beforehand or give it a picture of just that object. Other tools may take the wrong part of the picture or try to generate the entire image. The limitation is that the 3D model generation is not as good as similar tools from China, such as Hunyuan Tencent 3D.

Tencent Hunyuan 3D: It is the best AI 3D modelling tool for us. You can give it four pictures of an object: front, back, left, right, and in five minutes, we have a mesh that is more than good enough for the 3D scene and super accurate. However, the textures are slightly less accurate. Meshy can produce better texture results in the end, but you often have to spend more time working on Meshy’s models. Tencent Hunyuan is much more accurate with models all the time. In the end, for us, it is easier to retexture an accurate model than to retexture an inaccurate one with a nice texture, because the model defines the shape.

Q3. What is the current status of AI in the company of ReadySet Studios and its influence on the production workflow?

In terms of integrating AI into the workflow, it is predominantly used in concepting and pre-production. In ReadySet, we now have a full R&D department focused on working with AI, exploring the potential of these tools, and adapting our workflow. However, currently, we do not have a workflow with AI tools integrated; we have tools provided by R&D to support tasks our team is already working on. For example, an artist will be modelling in Unreal Engine, and then they may use Tencent Hunyuan 3D to support with some modelling, but their job remains mostly the same.

It is interesting to see the huge variety of tools and their uses in ReadySet. Some tools are key pillars of our company, such as Midjourney, Nano Banana, and Runway. However, tools such as SAM3D from Meta are still in their early stages of development. Also, among the many AI tools we have discussed here, we do not have a generative AI tool for creating the final image of the LED volume background. From what we have seen, AI is not at that stage yet. The limitations often outweigh the benefits. The biggest issues with AI in ICVFX are that it often takes more work to feed a black box that can produce unexpected outcomes than it does to create a 3D scene or a full image from scratch. Colour, brightness and overall quality are problems as well. The bit depth of the colour and luma is only 8 bit instead of the preferred 10 bit. As I mentioned before, detail and scale can be a problem for AI tools. The output is designed to work on a laptop, TV, or phone, but when you actually blow them up on a large LED volume or view them through the camera lens, they don’t hold up as you would expect. This is particularly difficult with clients, as managing expectations is challenging. We have one client who only worked with AI-generated backgrounds in their productions; the concept is geared towards that. Although they know the limitations, it is still difficult for us because you only know the final result on the production day, when you see it through the production team’s camera. You cannot sign off on it being finished until then, which often means that it is hard to work with and you have to scramble to make last-minute changes.

Q4. Can you walk me through ReadySet’s ethical considerations regarding the use of AI?

In terms of ethics, we always try to consider the input we give to AI. So, any image, model, or concept that we feed into it is either created by our team or royalty-free. On the other hand, we aren’t always thinking about where the data we upload ends up. As a smaller company, it is quite hard to keep track of that. That is perhaps something we should consider more. We do try to keep everything as local as possible, though.

Q5. Can you walk me through how the integration of AI may influence new and current employees?

We often discuss the future of the workforce at ReadySet and what that means for our job profile and future job profiles within the company. For new employees, I would say that the profile remains much the same, but we do expect them to be willing to work with AI tools. However, what I have seen in the industry is a threat to junior employment. Making basic 3D models is something AI can do. But the problem then becomes that when you need the new mediors and seniors, they are not there anymore because there are no juniors. There is no one who can flow through the ranks. At ReadySet, we always keep that in mind and continue to hire juniors.

We have also seen discrimination against seniors or those in their 50s or 60s. There is often an assumption that they are old and that we live in such a new age of technology, they will not want to adapt. This is not the case at all.

Another issue we see is AI growing too much and replacing tasks, which does affect people’s livelihoods and careers. Although some people enjoy working with AI, others do not. The question is: is it fair to say that if you don’t want to use AI, you need to go work in a different field? If I reflect on myself, if we end up in a situation where everything is fully generated and we all basically become AI prompters, then I would be out of a job, because I just would not want that.

Q6. How do you see the future for ai in the industry?

Personally, I am somewhat doubtful about the sustainability of the business models of some of these AI tools. I see the power needed to make this. I tried calculating the expense of using my own machines and doing this locally; it is super expensive. I question if in a couple of years AI tools will still be this cheap. That is my reservation about adapting the pipeline to use subscription AI tools. In the end, you are in their hands. Therefore, for us, it is super important to keep the knowledge of making everything from scratch, which is why we use a tool for help, not to replace the work. It is an addition to your toolset.

The writer’s final thoughts

It is evident that AI is slowly becoming an integral part of ReadySet as a company with a wide variety of tools to assist with different tasks. Although Daan mentions that they are focusing on keeping foundational knowledge in-house and avoiding reliance on AI, AI tools are still widely used in the company. The integration of an AI-focused R&D team has enabled them to explore a wide range of tools and their applications.

AI remains mostly in the concepting and pre-production phases, showing both strengths and weaknesses, with some exceptions that move into production. These exceptions have come with limitations and issues, which are likely the reason for the lack of progress in integrating AI-generated environments into the production phase.

The future of AI in this industry for ReadySet is unclear. Daan reflects on the economic model of AI and questions whether it will remain financially accessible to companies in the coming few years.

Ethical considerations also play a clear role as Daan discussed the issue of data and discrimination in the workforce. One thing seems certain at ReadySet: a focus on not shifting the work profile too much and on maintaining junior hiring, despite AI’s ability to do this job, is important.