Mixed-Reality Games to Socially Engage Students with Autism and Their Peers

Lisa Tolentino · Philippos Savvides · David Birchfield · Mina Johnson-Glenberg

Wed., June 09, 5:00–7:00, Great Hall

Over the course of 1 ½ years, a team of educators, designers, and researchers have been developing mediated learning games for high school students with autism. These scenarios were built using an embodied learning environment called the Situated Multimedia Arts Learning Lab (SMALLab).

SMALLab is a mixed-reality virtual learning environment that allows the body to function as an expressive interface using dynamic visual, textual, physical, and sonic media. SMALLab is a highly collaborative space that provides multiple pathways to learning through multimodal feedback and interface options that include motion capture, robotics, and custom-designed interaction objects. The unique features of this environment make it an optimal space for synthesizing digital media and tangible interfaces to create new kinds of interactive learning games that can support students who are typically marginalized through traditional teaching and learning methods.

The team worked in close partnership with high school special education teachers to identify key social challenges for their students. To support their specific social learning goals, our team developed two game-based scenarios, “Social Robot” and “Pong”. Both games strategically situate social learning goals in the natural context of game-play, addressing communication and social interaction through play.

In the “Social Robot” scenario, students learn to seek and direct requests for help with the help of a robotic partner. A student guides a robot to collect a series of images containing students’ faces. The game is complete when all of the images have been collected. During game play, the teacher has the opportunity to trigger “sabotage” mode. When this occurs, the robot stops in its tracks and spins in place, flashes red, and sounds an alarm to indicate that something is wrong. To proceed with the game, the student must interact with the teacher to ask for help in “fixing” the robot. If a student successfully asks for help, the teacher can “fix” the robot by placing it in normal play mode again. Game play will resume and the student continues until all discs are collected.

In the Pong scenario design, we supported social referencing and turn-taking opportunities by finding ways to connect students across a larger distance. Students stand along the edges of the space and pass a virtual ball between them. To pass a ball, students hit a virtual ball once it has reached their side of the space. The ball follows the person around until it is hit back to the person on the opposite side of the space. As a player serves the ball, a virtual path corresponding to the server’s color is dynamically redrawn to follow the ball. This colored path highlights the connection between players.

We will present these scenarios and results from an observational design experiment comparing students’ responses to similar social opportunities in the classroom to those in SMALLab. We will also discuss how applying game design principles in this context was successful in increasing students’ motivation to socially engage students on the autism spectrum with their teachers and peers.