Elizabeth Bagley completed her Ph.D. at the University of Wisconsin-Madison in Educational Psychology and Environment and Resources in July 2011. She is currently a Postdoctoral Fellow at the University of Illinois at Urbana-Champaign. She can be contacted at e: ebagley at illinois dot edu.
Abstract Virtual environments allow users to explore complex concepts using
simulations of real-world problems. In order to examine whether having mentors
communicate with players through a virtual chat program rather than face-to-face
changed anything about the players’ experience, this study compared a virtual chat
and a face-to-face condition of the epistemic game, Urban Science. 21 high school
aged players were randomly assigned to either the virtual chat or face-to-face
condition, and all players played Urban Science for 10 hours. As part of the game,
they participated in six reflection meetings led by their mentor, completed intake
and exit interviews, and wrote a final proposal. Using discourse analysis, epistemic
frame theory, and epistemic network analysis, this paper examines the mentors’ and
players’ reflection meeting discourse and the players’ interviews and final
proposals and asks whether there were differences in discourse, outcomes, or
engagement levels between the two conditions. This study concludes that virtual
mentoring can be just as effective as face-to-face mentoring and suggests that
mentoring via chat is a viable method for mentoring in the context of epistemic
games.
Bagley, Elizabeth A. S. (2011) Stop Talking and Type: Mentoring in a Virtual and Face-to-Face Environmental Education Environment. University of Wisconsin-Madison.
http://epistemicgames.org/eg/wp-content/uploads/Bagley-Dissertation-FINAL.pdf
Morgan, B., Burkett, C., Bagley, E., & Graesser, A. (2011). Typed versus Spoken Conversations in a Multi-Party Epistemic Game. Paper to be presented as poster at the Artificial Intelligence in Education Conference (AIED), Christchurch, NZ.
http://epistemicgames.org/eg/wp-content/uploads/aied2011_submission_153.pdf
One of the big questions that we have been looking at in the Epistemic Games Group in recent years is whether the newer “all online” versions of our games give players the same kind of experience (and thus offer the same opportunities for learning) as the “live” versions we were using several years ago.
(For those new to the Blog, our AutoMentor project is all about providing computer-generated mentors in epistemic games that will give the same kind of feedback as real mentors. To make that happen, the most recent versions of our games have mentors communicating with players through chat sessions online, rather than face-to-face.)
I just returned from running a short study at the Massachusetts Audubon Society’s Drumlin Farm Wildlife Sanctuary outside of Boston that sheds some interesting light on that question.
In the study, kids played two slightly different versions of Urban Science.
One group was ten high school students who interacted with mentors through an internal chat program.
The other group was eleven high school students who interacted face-to-face with mentors in the room.
The players were randomly assigned to each group, and everything else about the two games was the same (or as close to the same) as we could make it.
We’re just starting to analyze the data, but on at least one key dimension we already have a really strong result.
When the game ends, players take an exit interview that contains seven questions about how engaged they are in the game.
The first question is whether they thought the game was fun. With only two exceptions, every player said the game was fun. (And to be fair, those two exceptions were…well, exceptional kids.)
The other six questions are adapted from Green and Brock’s (2000) narrative questionnaire which is an instrument used to measure the level of engagement that readers have in a book.
Here’s the thing:
There was no significant difference between the two groups on these measures of engagement.
In fact, on average, the players chatting with mentors online were slightly more engaged in the game than the players who were talking with mentors face-to-face–although again, that difference was not statistically significant.
So, are epistemic games engaging? Yes. Are they engaging just because kids are in the room with mentors? No.
Which means, in theory, if we can automate the mentors through the relatively thin medium of chat, we may be able to get a lot of engagement without a lot of overhead…
For those interested in the details, a technical report will soon follow.
Graesser, A., Cai, Z., Wood, J., Hatfield, D., Bagley, E., Nash, P., & Shaffer, D.W. (2010). Comments of Journalism Mentors on News Stories: Classification and Epistemic Status of Mentor Contributions. Paper presented at the Intelligent Tutoring Systems Conference (ITS), Pittsburgh, PA.
http://epistemicgames.org/eg/wp-content/uploads/graesser-automentor-0410141.pdf
