Working as a mentor in an epistemic game is also very difficult.

We work with a limited amount of information about what the players are doing: we chat with the players via instant message, but we can’t see the players screens, can’t read their body language, and can’t hear the live conversations that they have offline.

So we are thrilled when we get information from our partners who are on the ground with the players. Here are two, from two versions of Urban Science we are running right now:

1. “Just thought you should know that I found out today that Amanda [not her real name] goes home and talks about this program every week non-stop to her parents over dinner. So, she’s pretty excited about it and enjoying herself.”

2. “That was so cool today. The changes you have made since last year have really made a difference. The access to chatting with mentors and the ability to track the targets as the changes are being made in the maps make all the difference. On the way to lunch, I overheard kids talking about the changes they made and getting close. The child with Aspergers was really engaged and wanting to make the target changes.”

We propose, then, the development of a new method of STEM assessment—called epistemic network analysis (ENA) that focuses not on whether students master specific scientific facts, math skills, or engineering concepts, but on whether and how students link the skills, knowledge, identity, values, and epistemology of a STEM practice into a coherent way of thinking about complex STEM problems.
We describe ENA as “potentially transformational” because it is in its early stages, and involves a radically different and interdisciplinary approach to the problems of STEM assessment. In this proposal we link prior work on an innovative theory of STEM thinking with the mathematical and conceptual tools of social network analysis to create a new conceptual and statistical approach to the measurement of STEM thinking and STEM learning.

We have been developing ENA as an assessment tool in the context of a particular theory of learning (the epistemic frame hypothesis) that applies to a specific kind of STEM learning computer game (epistemic games). However, we want to emphasize ENA is an approach to assessment that could be used in any situation of complex STEM thinking where the connections between things being learned are more important than isolated pieces themselves.

Intellectual Merit
In this proposal, we link work across several domains—including learning theory, psychometrics, and sociology—to develop a new assessment technique designed to measure the impact of technology-based STEM learning environments. This emerging research is, by its nature, uncertain. However, the collective work of the project team—including joint work on the pilot phases of this project—suggests that this research will produce conceptual, theoretical, and methodological results with the potential for far-reaching and long-term impacts on the theory and practice of network analysis, visualization, and their applications to STEM learning.

Broader Impact
The development of the cognitive model proposed here will help educators in a wide variety of fields analyze and improve STEM education by providing a means to dynamically assess the development of complex STEM thinking. The tools we develop will contribute to the field of network analysis, and our development process will enhance the skills and career trajectories of at least five young investigators.

Technology. The project will add two important components to prior work on NSF-funded STEM computer games. We will develop automated mentoring technology, AutoMentor, building on previous research on automated tutoring systems (specifically on AutoTutor, a computer tutor that helps students learn about science and technology topics by holding a conversation in natural language with the learner) and Evidence Centered Assessment Design (specifically, Epistemic Network Analysis, a methodology developed with NSF funding to assess students’ ability to think and act like STEM professionals through
game play).

Hypothesis. In so doing, the project explores a specific hypothesis about STEM mentoring: A sociocultural model as the basis of an automated tutoring system can provide a computational model of participation in a community of practice, which will produce effective professional feedback from nonplayer-characters in a STEM learning game.

Method. The project will use a Wizard of Oz methodology, in which data will be collected about player/mentor interactions over multiple instances of game play, and the resulting database used to develop and validate a system for automatically coding interactions. The coded database will then be used to generate automated responses to player actions in the game, and the resulting system will be tested to see whether players’ STEM learning with automated mentoring are comparable to outcomes with live mentors.

Team. The project team includes leading researchers in intelligent tutoring systems (Graesser),
assessment (Mislevy), and game-based learning (Shaffer). The team also includes a computer scientist (Gleicher), STEM content expert (Asligul Gocmen, Assistant Professor of Urban and Regional Planning,
University of Wisconsin-Madison), measurement expert (Andre A. Rupp, Assistant Professor of Measurement, Statistics, and Evaluation, University of Maryland) and a collaborating institution with expertise in STEM educational programming (Massachusetts Audubon Society). The combination of these areas of expertise is, we believe, unique and novel, and has the potential to transform work in each of the core areas of the proposal: intelligent tutoring, assessment, and game-based learning.

Intellectual Merit
The development of AutoMentor will represent a significant contribution to our knowledge about game-based learning and the science of learning more generally. The development of a computational
model of participation in a community of practice will provide an important link between traditional cognitive science and situated views of learning. It will also potentially contribute to research in artificial intelligence and intelligent agents.

Broader impact
This work will provide a powerful technology for incorporating professional STEM expertise in STEM education activities. The project enhances the infrastructure for joint research by forming a collaborative partnership among three research institutions (the University of Wisconsin-Madison, the University of Maryland, and the University of Memphis) and an educational delivery organization (The
Massachusetts Audubon Society). Results will be disseminated through scientific papers and conferences, but also through the work of the Massachusetts Audubon Society. The game incorporating AutoMentor will be available for use by schools and non-profit organizations.

While we were talking, one of her students walked into the room. The teacher enthusiastically told her that the class would be playing Urban Science again this spring. The student looked at us and wordlessly unzipped her coat to reveal the Epistemic Games t-shirt that all of the players got the previous year.

While I don’t want to go too far in interpreting the synchronicity of this encounter, I couldn’t help but think that 5th graders do not make sartorial choices lightly. It can sometimes be hard to know the inner transformations that happen as kids are learning and growing. But every once in a while, if you are lucky, you can get an unzipped glimpse of what kids take with them.

Computer Games in Education

Oct. 22, 2009

by Emily Mawer

A research team at the University of Wisconsin-Madison will study computer games and learning with federal research grants.

The principal investigator on several of the grants, David Williamson Shaffer, a professor of educational psychology at UW-Madison, said computers games allow students to live in a simulated world where they can face real life problems.

“What computer games do is give young people an opportunity to prepare for the kind of innovative and creative real world problem solving that they need to deal with in a global economy,” Shaffer said.

The largest grant, from the National Science Foundation, is devoted to research surrounding the Urban Science computer game, previously created at UW-Madison.  In the game, middle school and high school students become urban planners and solve problems that planners typically face, including going on site visits, talking to stake holders and using feedback to create a design proposal.  The research will focus on creating mentors for the game to coach students through their questions.

Shaffer explained that students studying to be urban planners are given the opportunity to try out parts of the planning process and then talk about their work with mentors.

“It is those conversations that turn the action that they are doing into understanding about the way the profession works,” Shaffer said.  “So in the game we recreated that.”

As the game currently exists, students can seek help from adult mentors through online chats.  Shaffer’s team, in partnership with the Massachusetts Audubon Society, will collect a database of commonly asked questions and answers.  Then they will analyze the information and create a system within the game that can respond appropriately to students.

Shaffer hopes that this part of the research will be applicable beyond this one game.

“The real payoff is in being able to use this same approach to make high quality professional mentoring available in all kinds of fields and all kinds of games,” Shaffer said.

The Urban Science research will take place over the next five years, with the first trials over the winter.

In addition to the Urban Science project, research grants are funding the creation of a computer game to make the engineering school at the University more diverse.

Shaffer is working with Naomi Chesler, an associate professor in the School of Engineering, to create a game that will increase retention in the engineering school, especially retention of woman and minorities.

“The idea is that a diverse workforce makes it possible to communicate with other countries and other places more effectively, because you have people from a variety of backgrounds,” Shaffer said.  “It brings other ideas and other perspectives.”

In the game, Nephrotex, engineering students will build a component of a dialysis machine using nanotechnology.  Shaffer said the idea is to give students a realistic experience of engineering design and to allow them to follow the process from beginning to end.

“Learning to become engineers is not just about math and science, but also about learning to see the world through a certain viewpoint,” Chesler said.

It is the math and science coursework that may cause so many students to decide against an engineering major, according to Shaffer.

”It is this kind of humanistic perspective on engineering that is often missing from the early parts of the curriculum,” Shaffer said.  “So, you get people who want to solve problems for other people, instead they end up solving a bunch of math equations and they get discouraged and drop out.”

The Nephrotex game will provide the kind of real world experience students need to stay motivated in their other courses, Shaffer said.

Nephrotex will be ready within the next two years, according to Chesler.

Federal grants power research on computer games and learning

Sept. 29, 2009

by Dennis Chaptman

A research team at the University of Wisconsin-Madison recently won $4.5 million in federal grants to study computer games and learning.

"Funding at this scale gives us an unprecedented opportunity to use computer games to teach — and to test what kids have learned — in a new way," says David Williamson Shaffer, a professor of educational psychology in the UW-Madison School of Education who is principal investigator on three of the four grants. "This funding lets us use the power of computer technology to prepare kids for the world of global competition that computer technology has created."

The largest of the National Science Foundation grants — for $3.5 million — creates a research consortium of three universities to develop technology that will let computers teach real-world problem-solving.

The team will use the educational computer game "Urban Science," which was developed at UW-Madison with previous funding from the National Science Foundation and the Macarthur Foundation.

In the game, middle- and high-school students learn about mathematics, science and technology by working as urban planners. Guided by adult mentors as part of a fictitious urban planning firm, they develop real plans for sustainable land use.

In partnership with the Massachusetts Audubon Society, students across Massachusetts will have an opportunity to play "Urban Science," and the researchers will collect data on what the students and adult mentors do in the game.

The research team will use this data to create computer-generated, artificially intelligent characters in game to coach students, so ultimately the game will be easy for teachers and students to use as part of science classes or in after-school programs.

"But more important," says Shaffer, "once we know how to create computer-generated mentors for this game, we can provide mentoring as part of any educational game."

The research consortium has an additional $300,000 grant to develop an assessment system that will do a better job than existing standardized tests at showing whether students have learned to solve real-world problems like those they face in "Urban Science."

"The problem with standardized tests," says Shaffer, who is also a game scientist with the Wisconsin Center for Educational Research, "is that they test the wrong thing. They test whether students have learned basic facts and basic skills, and not whether they can solve real problems."

Shaffer and his team are developing and validating an assessment system to provide instant feedback on students’ learning without using traditional standardized tests.

"Again," says Shaffer, "if we can do it for this game, we can do it for any game — and we can do it any time we care more about solving complex problems rather than learning basic skills. This is a new way of thinking about assessment for a new century."

Two more grants let Shaffer and his colleagues use the same techniques to improve undergraduate instruction at UW-Madison. Collaborating with associate professor Naomi Chessler in the College of Engineering, the team is developing computer games to make engineering a more diverse profession.

A $500,000 project will develop the computer game "Nephrotex," which Shaffer and Chesler say has the potential to increase the number of women and minority students in engineering programs. In the game, players work as part of a fictitious engineering firm using nanotechnology to design a better dialysis machine.

"The game will give undergraduates an early look at the kind of work real engineers do," says Chesler, a view of the profession that isn’t always clear as students plow through required math and science courses.

"Too often, students don’t get a chance to see the human side of engineering until they’ve gone through a lot of coursework," says Shaffer. "Students get intimidated and drop out. If we can build a game that lets freshmen interested in engineering solve complex problems, they can get a better understanding of how the things they learn in their other classes make sense and fit together."

The game is also funded, in part, by another $200,000 grant led by professor Wendy Crone in the College of Engineering to create a nanotechnology certificate program for engineering undergraduates.

Shaffer is quick to point out that projects on this scale are possible, partly because of the ongoing support UW-Madison has shown for research on computer games and learning. With faculty working in several departments across campus, "UW is one of the leaders in this field," says Shaffer.

Students with video gaming did better with the zoning maps; they had developed the visual hand to eye skills to be able to infer meaning with their actions. Students who spent less time gaming had greater difficulty with the maps….

The key point? That the visual skills from gaming are not necessarily just hand-to-eye. They are hand-to-eye-to-mind.

A recent hoax, in which the Manhattan Airport Foundation proposed a new airport be built in Central Park, fooled both the Huffington Post and Inhabitat, a weblog about sustainable design.

The Foundation’s site is convincing. The site uses slick design, planning language and high standards of presentation, but funniest of all, justifies itself by touting the environmental benefits (with scientific studies cited) and the variety of stakeholders who support it.

That it is even possible to make such an absurd proposal so convincing speaks to the complicated nature of land-use decisions and urban planning in general. It reminds me of one team of Urban Science planners who technically made the numbers work in a land-use proposal by turning the Overture Center, Madison’s premier arts and cultural center, into a parking lot. They presented their plan to live stakeholders, who were predictably outraged, and that team learned a valuable lesson about thinking through all the consequences of their decisions.

Maybe the folks at the Huffington Post would benefit from playing Urban Science…