http://epistemicgames.org/eg/wp-content/uploads/Bagley-Dissertation-FINAL.pdf

This dissertation examines how one approach to environmental education, the epistemic game Urban Science, extends an environmental education framework into a virtual environment that creates opportunities for young people to develop a combination of the skills, knowledge, values, identity, and epistemology needed to be environmentally literate. The aim of this study is to determine whether having mentors communicate with players through a virtual chat program rather than face-to-face changes anything about the players’ experience. Specifically, this study examines virtual chat versus face-to-face conditions during Urban Science and asks:

1. Was the mentors’ discourse during the reflection meetings different between the two conditions?
2. Was the players’ discourse during the reflection meetings different between the two conditions?
3. Were the players’ outcomes different between the two conditions?
4. Was the players’ level of engagement different between the two conditions?

Overall, these results suggest that since using more words did not impact the quality of the players’ professional discourse during the reflection meetings, the exit interviews outcomes, the quality of their final proposals, or their level of engagement, mentoring via chat is a viable method for mentoring in the context of epistemic games. A bolder interpretation of the results suggests that since mentoring in virtual and face-to-face conditions produced similar effects on players, epistemic mentoring could be automated and still retain the quality of interactions and players’ level of engagement. If the epistemic mentoring is automated, epistemic games like Urban Science could become more widely available to young people giving them the opportunity to help the world “move beyond what we already know in order to break beyond the boundaries of now to a more beautiful fabric of the future.”

As for me, well, in addition to helping the team pull together the new game design, new materials, and new assessments, I’ve been traveling to New York, London, Copengahen, and Utrecht. More on that in some additional posts, I hope. Oh yes, and we’ve also all been trying to finish the semester here at UW.

But in the midst of all of this, a piece on CNN [link removed by source] caught my eye as a must-read:

Fathers and kids bond playing video games

(AP) — When Will Nickelson and his daughter want to spend some quality time together, they fire up Nintendo Co.’s Wii and play a few rounds of “Wii Sports” or “Mario Party 8.”

…

The generation that grew up with “Pac-Man” and “Pong” are now having children of their own. And across the nation, fathers and their kids are finding the virtual worlds of video games a popular place to bond.

Many fathers say the games bring them closer to their kids by providing a safe, convenient way to stay in touch and talk to their children on their own terms.

What’s great is that I’ve been making this argument in my interviews lately, and it is nice to see it reaching the mainstream press–even if it didn’t come from me!

I do think, though, that the article still misses an important point. The piece quotes Dr. Arminta Jacobson, director of the Center for Parent Education at the University of North Texas, who warns that

Limits are necessary because video games don’t encourage reflective thinking skills, language development, social skills or physical activity.

“Fathers could set aside a time each week to play video games but also set aside times to read, take walks and just talk,” she said.

Now I certainly agree with the conclusion–of course kids need a balance of activities, including reading, talking, and getting outside to play. But I couldn’t disagree more with the claim that “video games don’t encourage reflective thinking skills, language development, social skills or physical activity.”

There is ample evidence to show that computer and video games–well designed games, and games played well–can do all of these things.

It is great for parents to get the message that computer games are good for parents and kids to play because they are fun and give parents and kids common ground and common experiences about which they can talk and “bond.” But we should also be talking about how to choose good games and play them, and talk about them, in ways that do encourage reflective thinking, language development and social skills. Epistemic games are one great way to do that, and an important message of How Computer Games Help Children Learn is about how.

(For those new to the Blog, our AutoMentor project is all about providing computer-generated mentors in epistemic games that will give the same kind of feedback as real mentors. To make that happen, the most recent versions of our games have mentors communicating with players through chat sessions online, rather than face-to-face.)

I just returned from running a short study at the Massachusetts Audubon Society’s Drumlin Farm Wildlife Sanctuary outside of Boston that sheds some interesting light on that question.

In the study, kids played two slightly different versions of Urban Science.

One group was ten high school students who interacted with mentors through an internal chat program.

The other group was eleven high school students who interacted face-to-face with mentors in the room.

The players were randomly assigned to each group, and everything else about the two games was the same (or as close to the same) as we could make it.

We’re just starting to analyze the data, but on at least one key dimension we already have a really strong result.

When the game ends, players take an exit interview that contains seven questions about how engaged they are in the game.

The first question is whether they thought the game was fun. With only two exceptions, every player said the game was fun. (And to be fair, those two exceptions were…well, exceptional kids.)

The other six questions are adapted from Green and Brock’s (2000) narrative questionnaire which is an instrument used to measure the level of engagement that readers have in a book.

Here’s the thing:

There was no significant difference between the two groups on these measures of engagement.

In fact, on average, the players chatting with mentors online were slightly more engaged in the game than the players who were talking with mentors face-to-face–although again, that difference was not statistically significant.

So, are epistemic games engaging? Yes. Are they engaging just because kids are in the room with mentors? No.

Which means, in theory, if we can automate the mentors through the relatively thin medium of chat, we may be able to get a lot of engagement without a lot of overhead…

For those interested in the details, a technical report will soon follow.

http://www.wcer.wisc.edu/publications/workingPapers/Working_Paper_No_2010_08.php

Preparing citizens to address the complex problems inherent in cities requires changing society to embrace a new kind of education. One way to train people to think about complex problems is to identify and study how professionals who think in those ways develop their epistemic frame. This ethnographic study of an urban planning practicum examines the ways urban planners master and appropriate relevant expertise. Specifically, epistemic network analysis is used to look at presentation feedback sessions during 2 weeks of the urban planning practicum and explore emergent relationships between the teacher’s and the students’ planning expertise. The results of this study indicate that epistemic network analysis offers a powerful set of techniques for analyzing the kinds of situated understanding that result from sociocultural learning and for observing the translation of pedagogy into practice in various types of learning environments.

Working as a mentor in an epistemic game is also very difficult.

We work with a limited amount of information about what the players are doing: we chat with the players via instant message, but we can’t see the players screens, can’t read their body language, and can’t hear the live conversations that they have offline.

So we are thrilled when we get information from our partners who are on the ground with the players. Here are two, from two versions of Urban Science we are running right now:

1. “Just thought you should know that I found out today that Amanda [not her real name] goes home and talks about this program every week non-stop to her parents over dinner. So, she’s pretty excited about it and enjoying herself.”

2. “That was so cool today. The changes you have made since last year have really made a difference. The access to chatting with mentors and the ability to track the targets as the changes are being made in the maps make all the difference. On the way to lunch, I overheard kids talking about the changes they made and getting close. The child with Aspergers was really engaged and wanting to make the target changes.”

We propose, then, the development of a new method of STEM assessment—called epistemic network analysis (ENA) that focuses not on whether students master specific scientific facts, math skills, or engineering concepts, but on whether and how students link the skills, knowledge, identity, values, and epistemology of a STEM practice into a coherent way of thinking about complex STEM problems.
We describe ENA as “potentially transformational” because it is in its early stages, and involves a radically different and interdisciplinary approach to the problems of STEM assessment. In this proposal we link prior work on an innovative theory of STEM thinking with the mathematical and conceptual tools of social network analysis to create a new conceptual and statistical approach to the measurement of STEM thinking and STEM learning.

We have been developing ENA as an assessment tool in the context of a particular theory of learning (the epistemic frame hypothesis) that applies to a specific kind of STEM learning computer game (epistemic games). However, we want to emphasize ENA is an approach to assessment that could be used in any situation of complex STEM thinking where the connections between things being learned are more important than isolated pieces themselves.

Intellectual Merit
In this proposal, we link work across several domains—including learning theory, psychometrics, and sociology—to develop a new assessment technique designed to measure the impact of technology-based STEM learning environments. This emerging research is, by its nature, uncertain. However, the collective work of the project team—including joint work on the pilot phases of this project—suggests that this research will produce conceptual, theoretical, and methodological results with the potential for far-reaching and long-term impacts on the theory and practice of network analysis, visualization, and their applications to STEM learning.

Broader Impact
The development of the cognitive model proposed here will help educators in a wide variety of fields analyze and improve STEM education by providing a means to dynamically assess the development of complex STEM thinking. The tools we develop will contribute to the field of network analysis, and our development process will enhance the skills and career trajectories of at least five young investigators.

Technology. The project will add two important components to prior work on NSF-funded STEM computer games. We will develop automated mentoring technology, AutoMentor, building on previous research on automated tutoring systems (specifically on AutoTutor, a computer tutor that helps students learn about science and technology topics by holding a conversation in natural language with the learner) and Evidence Centered Assessment Design (specifically, Epistemic Network Analysis, a methodology developed with NSF funding to assess students’ ability to think and act like STEM professionals through
game play).

Hypothesis. In so doing, the project explores a specific hypothesis about STEM mentoring: A sociocultural model as the basis of an automated tutoring system can provide a computational model of participation in a community of practice, which will produce effective professional feedback from nonplayer-characters in a STEM learning game.

Method. The project will use a Wizard of Oz methodology, in which data will be collected about player/mentor interactions over multiple instances of game play, and the resulting database used to develop and validate a system for automatically coding interactions. The coded database will then be used to generate automated responses to player actions in the game, and the resulting system will be tested to see whether players’ STEM learning with automated mentoring are comparable to outcomes with live mentors.

Team. The project team includes leading researchers in intelligent tutoring systems (Graesser),
assessment (Mislevy), and game-based learning (Shaffer). The team also includes a computer scientist (Gleicher), STEM content expert (Asligul Gocmen, Assistant Professor of Urban and Regional Planning,
University of Wisconsin-Madison), measurement expert (Andre A. Rupp, Assistant Professor of Measurement, Statistics, and Evaluation, University of Maryland) and a collaborating institution with expertise in STEM educational programming (Massachusetts Audubon Society). The combination of these areas of expertise is, we believe, unique and novel, and has the potential to transform work in each of the core areas of the proposal: intelligent tutoring, assessment, and game-based learning.

Intellectual Merit
The development of AutoMentor will represent a significant contribution to our knowledge about game-based learning and the science of learning more generally. The development of a computational
model of participation in a community of practice will provide an important link between traditional cognitive science and situated views of learning. It will also potentially contribute to research in artificial intelligence and intelligent agents.

Broader impact
This work will provide a powerful technology for incorporating professional STEM expertise in STEM education activities. The project enhances the infrastructure for joint research by forming a collaborative partnership among three research institutions (the University of Wisconsin-Madison, the University of Maryland, and the University of Memphis) and an educational delivery organization (The
Massachusetts Audubon Society). Results will be disseminated through scientific papers and conferences, but also through the work of the Massachusetts Audubon Society. The game incorporating AutoMentor will be available for use by schools and non-profit organizations.

While we were talking, one of her students walked into the room. The teacher enthusiastically told her that the class would be playing Urban Science again this spring. The student looked at us and wordlessly unzipped her coat to reveal the Epistemic Games t-shirt that all of the players got the previous year.

While I don’t want to go too far in interpreting the synchronicity of this encounter, I couldn’t help but think that 5th graders do not make sartorial choices lightly. It can sometimes be hard to know the inner transformations that happen as kids are learning and growing. But every once in a while, if you are lucky, you can get an unzipped glimpse of what kids take with them.

Computer Games in Education

Oct. 22, 2009

by Emily Mawer

A research team at the University of Wisconsin-Madison will study computer games and learning with federal research grants.

The principal investigator on several of the grants, David Williamson Shaffer, a professor of educational psychology at UW-Madison, said computers games allow students to live in a simulated world where they can face real life problems.

“What computer games do is give young people an opportunity to prepare for the kind of innovative and creative real world problem solving that they need to deal with in a global economy,” Shaffer said.

The largest grant, from the National Science Foundation, is devoted to research surrounding the Urban Science computer game, previously created at UW-Madison.  In the game, middle school and high school students become urban planners and solve problems that planners typically face, including going on site visits, talking to stake holders and using feedback to create a design proposal.  The research will focus on creating mentors for the game to coach students through their questions.

Shaffer explained that students studying to be urban planners are given the opportunity to try out parts of the planning process and then talk about their work with mentors.

“It is those conversations that turn the action that they are doing into understanding about the way the profession works,” Shaffer said.  “So in the game we recreated that.”

As the game currently exists, students can seek help from adult mentors through online chats.  Shaffer’s team, in partnership with the Massachusetts Audubon Society, will collect a database of commonly asked questions and answers.  Then they will analyze the information and create a system within the game that can respond appropriately to students.

Shaffer hopes that this part of the research will be applicable beyond this one game.

“The real payoff is in being able to use this same approach to make high quality professional mentoring available in all kinds of fields and all kinds of games,” Shaffer said.

The Urban Science research will take place over the next five years, with the first trials over the winter.

In addition to the Urban Science project, research grants are funding the creation of a computer game to make the engineering school at the University more diverse.

Shaffer is working with Naomi Chesler, an associate professor in the School of Engineering, to create a game that will increase retention in the engineering school, especially retention of woman and minorities.

“The idea is that a diverse workforce makes it possible to communicate with other countries and other places more effectively, because you have people from a variety of backgrounds,” Shaffer said.  “It brings other ideas and other perspectives.”

In the game, Nephrotex, engineering students will build a component of a dialysis machine using nanotechnology.  Shaffer said the idea is to give students a realistic experience of engineering design and to allow them to follow the process from beginning to end.

“Learning to become engineers is not just about math and science, but also about learning to see the world through a certain viewpoint,” Chesler said.

It is the math and science coursework that may cause so many students to decide against an engineering major, according to Shaffer.

”It is this kind of humanistic perspective on engineering that is often missing from the early parts of the curriculum,” Shaffer said.  “So, you get people who want to solve problems for other people, instead they end up solving a bunch of math equations and they get discouraged and drop out.”

The Nephrotex game will provide the kind of real world experience students need to stay motivated in their other courses, Shaffer said.

Nephrotex will be ready within the next two years, according to Chesler.