Abstract Virtual environments allow users to explore complex concepts using
simulations of real-world problems. In order to examine whether having mentors
communicate with players through a virtual chat program rather than face-to-face
changed anything about the players’ experience, this study compared a virtual chat
and a face-to-face condition of the epistemic game, Urban Science. 21 high school
aged players were randomly assigned to either the virtual chat or face-to-face
condition, and all players played Urban Science for 10 hours. As part of the game,
they participated in six reflection meetings led by their mentor, completed intake
and exit interviews, and wrote a final proposal. Using discourse analysis, epistemic
frame theory, and epistemic network analysis, this paper examines the mentors’ and
players’ reflection meeting discourse and the players’ interviews and final
proposals and asks whether there were differences in discourse, outcomes, or
engagement levels between the two conditions. This study concludes that virtual
mentoring can be just as effective as face-to-face mentoring and suggests that
mentoring via chat is a viable method for mentoring in the context of epistemic
games.

http://epistemicgames.org/eg/wp-content/uploads/Bagley-Dissertation-FINAL.pdf

This dissertation examines how one approach to environmental education, the epistemic game Urban Science, extends an environmental education framework into a virtual environment that creates opportunities for young people to develop a combination of the skills, knowledge, values, identity, and epistemology needed to be environmentally literate. The aim of this study is to determine whether having mentors communicate with players through a virtual chat program rather than face-to-face changes anything about the players’ experience. Specifically, this study examines virtual chat versus face-to-face conditions during Urban Science and asks:

1. Was the mentors’ discourse during the reflection meetings different between the two conditions?
2. Was the players’ discourse during the reflection meetings different between the two conditions?
3. Were the players’ outcomes different between the two conditions?
4. Was the players’ level of engagement different between the two conditions?

Overall, these results suggest that since using more words did not impact the quality of the players’ professional discourse during the reflection meetings, the exit interviews outcomes, the quality of their final proposals, or their level of engagement, mentoring via chat is a viable method for mentoring in the context of epistemic games. A bolder interpretation of the results suggests that since mentoring in virtual and face-to-face conditions produced similar effects on players, epistemic mentoring could be automated and still retain the quality of interactions and players’ level of engagement. If the epistemic mentoring is automated, epistemic games like Urban Science could become more widely available to young people giving them the opportunity to help the world “move beyond what we already know in order to break beyond the boundaries of now to a more beautiful fabric of the future.”

http://epistemicgames.org/eg/wp-content/uploads/aied2011_submission_153.pdf

Multi-party chat is a standard feature of popular online games and is increasingly available in collaborative learning environments, including epistemic games. However, little is known about the linguistic qualities of group conversation, especially regarding modality of communication. This paper addresses the differences between spoken and typed conversations as high school students interacted with the epistemic game Urban Science. Coh-Metrix analyses showed that speech was associated with more global aspects of text (narrativity, cohesion) whereas typed input was associated with more local aspects (syntactic simplicity, word concreteness). The spoken conversations were also more verbose than typed conversations. These findings suggest that the modality in group communication should be considered, particularly with respect to the breaks in discourse cohesion that are more prominent in typed group chat.

(For those new to the Blog, our AutoMentor project is all about providing computer-generated mentors in epistemic games that will give the same kind of feedback as real mentors. To make that happen, the most recent versions of our games have mentors communicating with players through chat sessions online, rather than face-to-face.)

I just returned from running a short study at the Massachusetts Audubon Society’s Drumlin Farm Wildlife Sanctuary outside of Boston that sheds some interesting light on that question.

In the study, kids played two slightly different versions of Urban Science.

One group was ten high school students who interacted with mentors through an internal chat program.

The other group was eleven high school students who interacted face-to-face with mentors in the room.

The players were randomly assigned to each group, and everything else about the two games was the same (or as close to the same) as we could make it.

We’re just starting to analyze the data, but on at least one key dimension we already have a really strong result.

When the game ends, players take an exit interview that contains seven questions about how engaged they are in the game.

The first question is whether they thought the game was fun. With only two exceptions, every player said the game was fun. (And to be fair, those two exceptions were…well, exceptional kids.)

The other six questions are adapted from Green and Brock’s (2000) narrative questionnaire which is an instrument used to measure the level of engagement that readers have in a book.

Here’s the thing:

There was no significant difference between the two groups on these measures of engagement.

In fact, on average, the players chatting with mentors online were slightly more engaged in the game than the players who were talking with mentors face-to-face–although again, that difference was not statistically significant.

So, are epistemic games engaging? Yes. Are they engaging just because kids are in the room with mentors? No.

Which means, in theory, if we can automate the mentors through the relatively thin medium of chat, we may be able to get a lot of engagement without a lot of overhead…

For those interested in the details, a technical report will soon follow.

http://epistemicgames.org/eg/wp-content/uploads/graesser-automentor-0410141.pdf

We identified the speech act categories and clusters of discourse comments of journalism mentors who interact with students editing news stories. Two important speech act categories are evaluations and suggestions. Latent semantic analysis and principal components analyses helped us discover clusters of comments involving evaluations and suggestions. The comments of mentors were also significantly aligned with epistemic frame elements that motivate the comments at a deeper level of discourse and pedagogy. Such alignments were validated by logistic regression analyses on a sample of hand-coded judgments of the frame elements. There was some modest transfer from a journalism practicum corpus to a game corpus. These analyses provide an important first step in building a virtual AutoMentor for multiparty epistemic games.

http://www.wcer.wisc.edu/publications/workingPapers/Working_Paper_No_2010_08.php

Preparing citizens to address the complex problems inherent in cities requires changing society to embrace a new kind of education. One way to train people to think about complex problems is to identify and study how professionals who think in those ways develop their epistemic frame. This ethnographic study of an urban planning practicum examines the ways urban planners master and appropriate relevant expertise. Specifically, epistemic network analysis is used to look at presentation feedback sessions during 2 weeks of the urban planning practicum and explore emergent relationships between the teacher’s and the students’ planning expertise. The results of this study indicate that epistemic network analysis offers a powerful set of techniques for analyzing the kinds of situated understanding that result from sociocultural learning and for observing the translation of pedagogy into practice in various types of learning environments.

http://epistemicgames.org/eg/wp-content/uploads/engineering-workshop.pdf

Engineering institutions nationwide are pursuing first-year engineering design courses to attract and retain nontraditional students. However, these courses often have high enrollment rates and can be resource intensive. Virtual design projects offer a potential solution to the physical resources requirements but often result in an overly constrained design space, creating uninteresting or non-challenging design problems. We are developing a design problem within a novel virtual environment (i.e.a game) that provides first-year engineering undergraduates with a more authentic engineering design experience and a more complete and accurate understanding of the engineering profession. The design problem presented challenges students to incorporate carbon nanotubes and chemical surfactants into a hemodialysis ultrafiltration unit. our approach seeks to provide students with experience in the skills, knowledge, values, identity, and epistemology of the engineering profession, which is the epistemic frame of the profession. The virtual environment also provides a uniquely comprehensive platform for assessing the students’ epistemic frame development over time. We anticipate that this approach will be highly engaging to first-year undergraduate engineering students and will help engineering instructors understand how engineers-in-training learn to become engineers.

http://epistemicgames.org/eg/wp-content/uploads/Bagley-ICLS2010-submitted.doc

Preparing citizens to address the complex problems inherent in cities requires our changing society to embrace a new kind of education. One way to train people to think about complex problems is to identify and study how professionals who think in those ways develop their epistemic frame. In this paper, we examine one of the ways urban planners master and appropriate relevant expertise through an ethnographic study of an urban planning practicum. Specifically, we use a new method called epistemic network analysis to look at presentation feedback sessions during two weeks of the practicum to explore emergent relationships between the teacher’s planning expertise and the students’ expertise. The results of this study indicate that epistemic network analysis offers a technique for analyzing the kinds of situated understanding that result from sociocultural learning and for observing the translation of pedagogy into practice in various types of learning environments.

Epistemic games have been developed to help players develop domain-specific expertise that characterizes how professionals in a particular domain reason, communicate, and act (Shaffer, 2006; Shaffer & Bagley, 2009). Grounded in a sociocultural and sociocognitive approach to learning, epistemic games are designed to foster situated learning that leads to data structures with high levels of dependencies. As one might expect, traditional measurement models struggle to accommodate such contextual dependencies, especially when data are collected at smaller scales and epistemic network analysis (ENA) has been developed to provide a practically feasible modeling alternative (e.g., Rupp et al., 2009; Shaffer et al., in press). In this paper, we describe a research program that addresses key statistical considerations for modeling data from epistemic games using ENA with an eye toward representing different learning progressions of players within such games. Current approaches for representing learning progressions using ENA are juxtaposed with approaches for simulating such data using particular statistical constraints.