Peer2peer roleplays can be daunting, silly, embarrassing, inconsistent and only happen at class times. So they are either skipped by the academic, lead to a poor experience or are seen as low value for students even though the skills involved are so critical to all disciplines. Can an 'immersive' AI powered simulation of conversation-centric scenarios running in VR and on the web provide a solution? Can we deliver memorable and meaningful education experiences for students at the same time?
Building on the new wave of AI LLM capability, our AI Conversation Sim project sets out to allow academics to craft specialised scenarios aimed at helping their students learn specific skills related to conversations in their discipline areas. The students can then practice these simulated conversations through VR or Web interfaces that present a 3D environment and virtual AI agents that talk with the player in real-time. Leveraging an authoring tool, teachers can create scenarios covering use-cases such as journalism interviewing practice, boardroom negotiations and doctors taking patient histories.
Note that this project is being done in coordination with the UNSW IT AI guardrail processes to ensure that development and exploration in the use of AI is performed in safe, ethical and secure ways.
Project Details
The project is in active development and the initial Proof of Concept experiments in several courses have produced exciting results. The simulation uses a game engine visualisation which communicates with specialised AI models like ChatGPT to deliver a fluid conversational experience for the students in a carefully constructed experience relating to their coursework or discipline. There is exciting potential in being able to build experiences with briefing information, objective checking, feedback, timers, emotion, 3D characters, virtual reality, multiple AIs, transcripts, security, achievements, props, gestures and more.
WIP
The work in progress AI Conversation Sim has been explored in several classes to refine the development.
The Journalism class saw students practicing their interviewing skills in VR with an AI Senator and and AI Celebrity with fascinating results. The students reported increased confidence in these sorts of interviews following the experience. One student even commented that she wasn't so comfortable doing celebrity interviews anymore and having to ask those personal questions. This really shows how engaged with the exercise the students were and the level of impact the simulation had.
In the AI Literacy course students debated with the virtual agent and it was fascinating seeing them learn new arguments from the AI which they then used when they switched sides on the topics.
Emergency Medicine developed a series of specific patient interview scenarios to help students practice for their year 6 exams which would include mock Dr-Patient interactions. The students identified as needing the most support to succeed at these interviews were given support which included the 6 patient cases in the AI Conversation Sim accessed via the web in their own time.
TECHNOLOGY & DEVELOPMENT
The team of staff and work integrated learning students are using Unity as the core of the platform with web services to manage project editing and the API calls to a range of AI endpoints from Open.AI's ChatGPT to Hume.AI and ElevenLabs.
Videos
Use Cases
The aim of the project is to allow academics to create and hone their scenarios within the AI Conversation Sim via the web authoring toolset. Along with the students being able to access the experiences by both VR and Web this will allow the project to work at the scale desired for each course.
Current Use-Cases under Development
- ADA Language Learning – Practice conversations
- ADA Journalism – Interviewing skills
- ADA Education – Parent teacher interviews
- Business – Employees with a disability
- Business – Negotiation approaches
- Business – Board room presentation Q&A
- Business - AI literacy & debate
- UNSW College – English confidence
- PVCE – AI literacy module
- Medicine – Optometry clinical communication
- Medicine – Emergency Dr-Patient interactions
- Medicine – Pharmacy patient history taking
- Medicine – Psychiatry mental health interactions
- Science – Science communication
- Engineering – Collaborative communication
- Engineering - Photovoltaics client meeting
- Law – Class participation practice
Credits
"The AI Conversation Sim brought a key journalist skill - interviewing - to virtual life, with student feedback calling it 'spectacular!' "
Kerrie Davies, Arts & Media, Art Design & Architecture
“The AI Conversation Sim has been an amazing addition to my course, providing students with an immersive and interactive experience that greatly enhances their understanding of AI and improves their critical thinking and debate skills. It's a look into the future of education, where technology and innovation create great learning opportunities.”
Alba V. Olivares Nadal, Senior Lecturer, Information Systems and Technology Management, Business
 | Silver at the 2024 International QS Reimagine Education Awards. |
Academic Partners
Kerrie Davies (Senior Lecturer, Director of Masters of Journalism & Communication, Arts and Media, ADA)
Kelsey Burton (Lecturer, Management & Governance, Business)
Alba Olivares Nadal (Senior Lecturer, Information Systems and Technology Management, Business)
Sam Kirshner (Associate Professor, Information Systems and Technology Management, Business)
Veronica Jiang (Associate Professor, School of Marketing, Academic Disability Advisor, Business)
Betty Chan (Head of Critical Care, Medicine and Health)
Jack Marjot (Adjunct Lecturer, Medicine and Health)
Michelle Bonnette (Lecturer, Health Sciences, Medicine and Health)
May Lim (Nexus Fellow, Chemical Engineering)
Terry Cumming (Professor of Special Education, Deputy Head of School Learning & Teaching, Education, ADA)
Ellen Lee (Lecturer, School of Education, ADA)
Jacqueline Mees-Buss (Lecturer, Management & Governance, Business)
Christiaan Roell (Lecturer, Management & Governance, Business)
Megan Kalucy (Senior Lecturer, Psychiatry Education Lead, Medicine and Health)
Aparna Jacob (Director of Studies, Academic English Pathways, UNSW College)
Kimberly Duncan (Deputy Director, Academic Programs, UNSW College)
Brooke Donnelly (Academic Head, Embedded English, UNSW College)
Svetlana Tyulkina (Senior Lecturer, Global & Public Law)
Nayuta Yoshioka (Lecturer, School of Optometry & Vision Science, Medicine and Health)
Natalia Ortiz Ceberio (Senior Lecturer, Humanities & Languages)
Carol Oliver (Professor, Biological Earth & Environmental Science)
James Bedford (Academic Learning Facilitator, PVC Student Success)
Clem Wong (Education Delivery Solutions Manager, PVC Education)
Alice Joo (Lecturer, Korean Studies, Humanities and Languages, ADA)
Morphy Dumlao (Lecturer, Science Chemistry, Canberra)
Kyla Sumaylo (Project Officer, Education Design and Quality, Law)
Design & Development
Tim Dodds (Immersive Technology Specialist & Developer, PVC Education, UNSW)
Paul Dyer (Immersive Technology Specialist & Developer, PVC Education, UNSW)
Ashley Hall (Immersive Technology Specialist & Developer, PVC Education, UNSW)
Graham Hannah (Manager of Immersive Technology, PVC Education, UNSW)
Erica Leonar (Senior Education Designer, PVC Education, UNSW)
Work Integrated Learning Student Partners
Afrida Haque (Software Engineering Honours Thesis Project, UNSW)
Qingjiang Hu (Master of Simulation and Immersive Technology Capstone, UNSW)
Maitri Chakravarty (Software Engineering Honours Thesis Project, UNSW)
Cherise Hoang (Software Engineering Honours Thesis Project, UNSW)
Ben Thomas (Software Engineering Honours Thesis Project, UNSW)
Zeneng Harry Zhao (Software Engineering Honours Thesis Project, UNSW)
William Wan (Software Engineering Honours Thesis Project, UNSW)
Get Involved
There are internal Teams Channels and working groups covering the development and use-cases. To join the team, get a demo or have your use-case part of the development just get in touch.
- Email: [email protected]
There are internal Teams Channels and working groups covering the development and use-cases. To join the team, get a demo or have your use-case part of the development just get in touch.
- Email: [email protected]