Cradle, along with the other VIBE partners, researched and created digital humans that can be used in virtual, mixed, and augmented reality. These digital avatars were aimed at offering new possibilities in the Brabant Economy, such as the creation of applications for healthcare training and patient communication. Cradle was responsible for the creation of the virtual humans. For this, the team employed the latest technologies to build a multi-function digital agent with high-fidelity looks.
The objective of this R&D project was to find and improve new ways of depicting human likeness through the use of realistic virtual avatars. This project aimed at improving each aspect associated with the believability of 3D avatars, with the goal of establishing Brabant as a global player in the area of photo-realistic human simulation.
Additionally, the developed character should serve as an embodied conversational agent for healthcare applications.
This research was supported by a grant of 7 million euros from the European Union, OP Zuid, the Ministry of Economic Affairs, and the municipality of Tilburg. There, multiple partners were involved in the development of the project such as Tilburg University, Blue Tea, Fontys, Indicia, Maxima MC, The Royal Netherlands Aerospace Centre, Noldus, ROC Tilburg, Samure and Spaarne Gasthuis.
During this collaboration CRADLE’s task was to develop a high-fidelity virtual humans For this, CRADLE made use of state-of-the-art technologies to support the creation of these digital characters.
To achieve a photorealistic digital human, the team built and made use of a custom-built photogrammetry studio. This allowed CRADLE to capture even the finest details of the models and strive towards photorealistic results.
For the development of the digital human, it was also necessary to create and iterate on a custom workflow. After some trial and error, a new pipeline was created allowing the artists to create and process a model within a week instead of a month.
Other challenges included building a text-to-speech and speech-to-text system to allow the avatar to respond in real time.
The result of the research is a range of digital humans that are ready to use within the VIBE application. There different characters can be loaded and portray different emotions.
All avatars are also capable of speech with the text-to-speech and speech-to-text systems. An additional integrated feature is the possibility to change the clothing and background of each digital human. They are also capable of following the interlocutor with their gaze and cycle through different behavioural patterns related to childbirth and labour complications.
The VIBE digital humans are now actively being tested by the MMC Maxima hospital in the context of medical training for labour and labour complications.
Furthermore, an application was made by our partners who use the digital humans as a communication tool for patients who underwent bariatric surgery.
Applications used for the project:
IUI 2023: https://iui.acm.org/2023/accepted_papers.html