Value-Added Assessment of Learning and Practice: The Challenges of Measuring the Growth of Learning in Games

Christopher Thorn

Thu., June 11, 2:00–3:00, Inn Wisconsin (2nd floor, East/Southeast)

As one of the leaders of the Value-Added Research Center, I’m interested in the challenges of assessment in games and other interactive environments. We think about “assessment” in quite a broad sense that includes both high states, formal tests as well as local diagnostic measures used only by individual teachers or teams. At the same time, we are quite interested in measures of process and behavior for adults and kids, and we work with district, school, and union leaders to recommend frameworks that are both directive (use strong rubrics that suggest good practice) and participatory (observational teams include both new and experienced teachers and leaders).

As we consider the considerable impact games and simulations can have on child and adult learning, we struggle with how to adapt our thinking on how best to build systems for high-stakes accountability and no- (or low-) stakes feedback. We believe that the feedback and incentive systems in MMORPGs might provide some compelling design principles for rethinking how we support assessment and diagnostics across a wide range of educational settings.

One of the core notions of a value-added approach to assessment is that we move away from the limited picture of attaining proficiency. Proficiency is about a society’s obligation to its citizens. The schooling should deliver a basic level of mastery to all students as a public good, as part of one’s basic rights. What proficiency models do not do well is address the strengths (or weaknesses) that the children and adults in an educational system bring with them when they enter the classroom. Value-added models are used to isolate the contribution of a given educational activity (a course, an classroom, a professional development session) to growth of an individual’s skills and knowledge while controlling for many of the factors that account for differences in individual starting points.

We can see aspects of these notions in modern gaming. While many games ask the player to rank their own ability by choosing a difficulty level, Call of Duty 4, for example, runs the player through a non-lethal training scenario that assesses key aspects of play against a model of ability requirements for differing difficulty levels. While this model focuses on the single-player versus AI, these notions could apply to multiplayer systems as well. Many games already use ladders and rank achievement as core notions in advancement (formal) and prestige (informal)—both of which are leveraged in team or clan systems.

A value-added model of assessment in this environment would look at player (student) characteristics at the outset of a game or simulation. This would include demographic data and outcomes from prior assessments (experience, gold, skills, puzzles solved, level of difficulty, etc.) and provide a level playing field for assessing the contribution of the current game/simulation to player learning.