This project was part of a scientific investigation developed while completing my Master's degree at the Human Interface Technology Laboratory of New Zealand (HIT Lab NZ), where we explored the use of virtual humans as training tools within immersive environments. Our goal was to evaluate the effectiveness of virtual human (VH) role-players as leadership training tools in virtual reality (VR) and mixed reality (MR) environments compared to the traditional training method using real human (RH) role-players in a real-world (RW) setting.
Structure followed to organise the narratives of each scenario.
The "Introduction" section provides relevant information about scenarios and followers to contextualise participants.
The "Follower Bio" section provides detailed information about followers and their development levels.
In the "Follower Presentation" section, the follower introduced themself.
The "Participant Statement" section provided four choices designed based on each leadership style, from which participants had to select only one per level.
The "Follower Response" section provided the follower's response to the specific statement selected by the participant.
The "Feedback" section provided information about the development level of the follower and the best leadership style required for each scenario.
As part of the stress level analysis, we used the Zephyr sensor attached to a chest strap to record the participant's heart rate and variability.
Natural-looking 3D characters created using Adobe Fuse CC.
Virtual human animation set up.
Project Details
To conduct our study, we developed this innovative and experimental training simulation that enabled us to assess three distinct conditions: RH role-players in RW (RH-RW), VH role-players in VR (VH-VR), and VH role-players in MR (VH-MR). We involved 30 participants who assumed leadership roles and interacted with either RHs or VHs during structured role-playing scenarios before and after receiving a leadership training session. We designed the training session and role-play narratives using the Situational Leadership II (SLII) model developed by The Ken Blanchard Companies. Our primary focus was to investigate two key aspects: (1) the comparative effectiveness of VH role-players and RH role-players during both pre- and post-sessions, and (2) the impact of the human-type (RH, VH) in conjunction with the environment-type (RW, VR, MR) on the overall outcomes.
Development
To create the VHs, we used Adobe Fuse CC to provide natural-looking 3D characters. We then used Mixamo to give our characters a complete human-rigged skeleton and facial blend shapes ready to animate. We completed the animation process using the Unity3D engine. For body animation, we used the Final IK plugin, an HTC VIVE Pro with two Vive controllers and three HTC Vive trackers. We used the HTC Vive Pro integrated microphone and the Unity plugin SALSA for the facial expressions to provide lip-sync movement and random eye movement. Finally, we used Storyline 360, an e-learning-authoring platform for instructional designers, to manage all the role-play narratives.
Team and Collaboration
This project involved a team of four members, including academic supervisors and a subject matter expert. My role involved coordination, research, and development. We also thank Blanchard New Zealand for participating and collaborating in this study.
Key Responsibilities
Research: Conducted an extensive review of scientific papers to lay the groundwork for the experiment and project.
Storyline: Organised multiple SLII role-play narratives in a coherent structure using Storyline 360.
Hardware Setup: Configured and executed a video see-through-based MR experience using a stereo camera in front of the HMD.
Data and Analysis: Collected and analysed data using both qualitative and quantitative tools.
Mechanics: Developed interactive elements and user interfaces in C# within Unity3D.
Lighting Setup: Optimised performance through lighting setup and lightmap baking.
Animation: Created, rigged and animated the virtual humans using Adobe Fuse CC, Mixamo, HTC Vive equipment and Unity.
Optimisation: Addressed performance issues with Unity's profiling tools.
Challenges
First, while having a fixed set of narratives ensured a consistent training experience, it also limited participant behaviour. Expanding the number of choices poses a challenge for storyline developers, and more elaborate narratives risk becoming unrealistic. Moving forward, we can explore dynamic storytelling techniques to maintain realism while offering more choices. Secondly, creating the role-play scenarios was time-consuming and required the expertise of a subject-matter expert. This collaboration, however, enriched the content significantly. Streamlining the development process and incorporating AI-driven tools could accelerate scenario creation in the future. Finally, the stereo camera used in the study had a limited field of view. It required a lower resolution to maintain a suitable frame rate and avoid cybersickness. Despite these concerns, we successfully implemented a workable solution. Future advancements in depth-sensing cameras and higher resolution capabilities promise to overcome these limitations, enhancing the overall experience.
With support from the findings in our study, we conclude that (1) virtual humans can be as effective as real-human role-players in supporting the practice of leadership skills. (2) The combination of human type (RH, VH) and environment type (RW, VR, and MR) positively impacts training outcomes. Although the differences were not statistically significant, RH and VH role-players made participants feel comfortable and engaged throughout the sessions. Surprisingly, some scores were similar, and even higher, with VHs in computer-generated environments compared to RHs.
In response to identified RHs limitations, VHs still face similar constraints, as they still need to be programmed, so the development of their training experiences may necessarily require the participation of experts, and the computer-generated environments in which VHs operate may also require specific equipment involving additional costs. However, our study has also shown that with adequate context and instructional support, VHs can become invaluable tools, where their human-like appearance, rich configurability, and ease of replication allow them to play a critical role in social skills training.
Additional Information
It's important to note that the media resources displayed here pertain specifically to the MR experience. For a comprehensive understanding of the study, including detailed findings and insights, we kindly refer you to:
Suárez, G., Jung, S., and Lindeman, R. W. (2021). Evaluating virtual human role‑players for the practice and development of leadership skills. Frontiers in Virtual Reality 2, 31. doi:10.3389/frvir.2021.658561.
