Embedded Assessment: Evaluating In-Game Data to Adapt the Learning Environment

Jody Underwood · Stacy Kruse · Peter Jakl

Fri., June 12, 11:00–12:30, Browsing Library (2nd floor, West Central)

Research on assessment in games is a burgeoning field. In one effort, quests require students to solve a mystery by interviewing non-player characters for relevant information and writing a newspaper article describing the solution (e.g., Barab, Arici, & Jackson, 2005). The written artifacts are evaluated by real people. Other efforts do conversation analysis of discussions that take place in-game (e.g., Shaffer, 2007) and out-of-game (e.g., Steinkuehler, 2008), but neither looks at game activity directly. A fourth approach is to carry out a pre- and post-test experimental design to determine what the student has learned (e.g., Zapata-Rivera, et al., 2007). Each of these approaches has its merit, often showing that students are learning concepts in the gaming environment. However, there has been little effort to study in-game activity to see what facilitates or creates bottlenecks to learning concepts, to say nothing of skills that are being learned. What’s more, studying what is being learned in-game by looking at out-of-game artifacts (e.g., most of the examples mentioned above) removes the possibility of adapting the environment to the learner in real time.

Shute, Ventura, Bauer, and Zapata-Rivera (2009) propose an approach for embedding assessment into games to reveal what is being learned during the gaming experience. Instead of assessing particular content, they are interested in analyzing a sequence of events to infer abilities. We build on this effort toward the end of giving feedback and adapting the environment to the individual learner.

What types of in-game events can inform us about knowledge, procedure, and 21st century skills being learned or displayed? Knowledge, as always, is assessed through single events with outcomes (e.g., cracking a code). Procedures are assessed by finding patterns in paths taken to solve a problem (e.g., approaches to cracking a code). By skills, we refer to such things as teamwork and critical thinking, which are harder to assess because the process is independent of the outcome. That is, an answer can be correct, but it alone does not tell you if the associated teamwork was of good quality. For procedures and skills, we use such approaches as genetic algorithms and visual data mining to look at patterns and make inferences about “good” or efficient paths.

In this presentation, we discuss Leverage, a platform-agnostic technology that allows researchers and developers of any digital learning environment to record, aggregate, and make inferences about learning and performance by using game events as embedded assessment, as well as adapt the environment to the individual learner. We will include examples from projects in which we are using and investigating these capabilities.