VIBE – Virtual Humans in the Brabant Economy

Virtual Humans

Placeholder

Introduction

Cradle, along with the other VIBE partners, researched and created digital humans that can be used in virtual, mixed, and augmented reality. These digital avatars were aimed at offering new possibilities in the Brabant Economy, such as the creation of applications for healthcare training and patient communication. Cradle was responsible for the creation of the virtual humans. For this, the team employed the latest technologies to build a multi-function digital agent with high-fidelity looks.

Story

The objective of this R&D project was to find and improve new ways of depicting human likeness through the use of realistic virtual avatars. This project aimed at improving each aspect associated with the believability of 3D avatars, with the goal of establishing Brabant as a global player in the area of photo-realistic human simulation.

Additionally, the developed character should serve as an embodied conversational agent for healthcare applications.

This research was supported by a grant of 7 million euros from the European Union, OP Zuid, the Ministry of Economic Affairs, and the municipality of Tilburg. There, multiple partners were involved in the development of the project such as Tilburg University, Blue Tea, Fontys, Indicia, Maxima MC, The Royal Netherlands Aerospace Centre, Noldus, ROC Tilburg, Samure and Spaarne Gasthuis.

During this collaboration CRADLE’s task was to develop a high-fidelity virtual humans For this, CRADLE made use of state-of-the-art technologies to support the creation of these digital characters.

Process

To achieve a photorealistic digital human, the team built and made use of a custom-built photogrammetry studio. This allowed CRADLE to capture even the finest details of the models and strive towards photorealistic results.

For the development of the digital human, it was also necessary to create and iterate on a custom workflow. After some trial and error, a new pipeline was created allowing the artists to create and process a model within a week instead of a month.

Other challenges included building a text-to-speech and speech-to-text system to allow the avatar to respond in real time.

Results

The result of the research is a range of digital humans that are ready to use within the VIBE application. There different characters can be loaded and portray different emotions.

All avatars are also capable of speech with the text-to-speech and speech-to-text systems. An additional integrated feature is the possibility to change the clothing and background of each digital human. They are also capable of following the interlocutor with their gaze and cycle through different behavioural patterns related to childbirth and labour complications.

The VIBE digital humans are now actively being tested by the MMC Maxima hospital in the context of medical training for labour and labour complications.

Furthermore, an application was made by our partners who use the digital humans as a communication tool for patients who underwent bariatric surgery.

Currently, multiple papers about the research, creation and development of the digital human are under review and will soon be released.

Check out the practical use cases here:

VIBE Use Case: MMC Hospital, Virtual Patient
VIBE Use Case: Text-To-Speech / Speech-To-Text

Project website