The test fails the test
A couple of months ago a study by the Intercollegiate Studies Institute (article here) reported:
US elected officials scored abysmally on a test measuring their civic knowledge, with an average grade of just 44 percent
That was even lower that the score of 49 percent for Americans overall who took the test.
So, being a curious fellow, I took the test myself, and got a score of 93.94 percent, and I was told that the monthly average when I took it was 78.1 percent. Apparently, Americans got 59.3 percent smarter since the test came out. Hmmm.
This got me thinking: Is my civic literacy really twice that of the average elected official in the US? That seems like a pretty odd thing to conclude, since I’m almost certain sure that I would be pretty lousy as a dog catcher, much less a high elected official. If a test that shows that I am more qualified to be an elected official than many elected officials, then surely there is something wrong with the test, right?
A quick look at the questions on the test sheds some interesting light:
Thirteen of the 33 knowledge questions are taken from previous ISI surveys developed by ISI faculty advisors from universities around the country. Nine of the civic knowledge questions are taken from the U.S. Department of Education’s 12th grade NAEP test, and six from the U.S. naturalization exam. Two new knowledge questions were developed especially for this new survey and three are drawn from an “American History 101” exam posted online by www.InfoPlease.com.
Just to be sure we have this right: Elected officials don’t do well on questions from a 12th grade standardized test and on some other standardized tests, therefore we conclude they don’t know enough civics.
Now I’m not about to defend everything every elected official does in this country, but I don’t think that because I can pass this civic literacy exam I could do a better job myself.
Here’s the problem: The point of those tests was to try to predict, based on a very brief set of simple questions, how well someone would do in the much more complex, real world. But when we find out that the test doesn’t predict well, we don’t throw away the test. We assume that people in the real world must not really know what they are doing.
They fail the test. The test can’t be wrong.
This is, of course, one of the most fundamental fallacies of our regime of high-stakes tests: we take performance on the test as an end in itself, never bothering to ask–much less test–whether the exams actually tell us anything useful about whether students can really do anything useful in the world.
Actually, that is not quite fair. Those studies have been done, and they routinely show that exams don’t give us much useful information. As I discuss in How Computer Games Help Children Learn:
Even students who do well on school tests cannot apply their knowledge to real-world problem solving. For example, one classic set of studies shows that students who have passed a physics course and can write Newton’s Laws of Motion down on a piece of paper still can’t answer even simple problems like “If you flip a coin into the air, how many forces are acting on it at the top of its trajectory?” Which is, of course, a problem that can be solved using Newton’s Laws.
The technical term here is validity. What makes a test valid? How do we know it is testing what we really want it to test?
In the case of this civics test, presumably we’d want to know that there is some relationship between how well someone does on the test and whether or not they really are a good public official, or a good citizen.
But we don’t do that. Passing the test has become the measure that matters, and the results we get are predictably absurd. Like the idea that because I can “name two countries that were our enemies in World War II”, I would be a good elected official. Or rather that because elected officials can’t pass a standardized test, they must not know what they are doing.
“…fundamental fallacies of our regime of high-stakes tests: we take performance on the test as an end in itself, never bothering to ask–much less test–whether the exams actually tell us anything useful about whether students can really do anything useful in the world.”
I agree, however, forgive me if I am wrong but I get the sense that proponents of epistemic games tend to assume that valid relationships, e.g., a person’s potential to be or act in certain ways, such as being a good public official or citizen, is somehow more discernible or assessable through, or in some other way more valid in an epistemic game. But surely a poorly researched, designed, implemented, and assessed epistemic game would not be more valid for determining correlationships than an excellently researched, designed, implemented, and assessed test just because it is an epistemic game. They are both human instruments and subject to the same potentials and flaws. Various epistemic games would have various levels of validity just as various tests would, and both require asking and testing the validity of the proposed relationships.
It seems to me that a valid design must start with the designer’s epistemology and assumptions and level of awareness. How does the designer determine if the frames they seek or have found are actually connected to the type of success that they intend to capture? Who defines the criteria for success? And how does the designer know that they have actually captured the epistemic frames they seek?
I am a sincere supporter of the potential of epistemic games and am currently preparing a research proposal to look at epistemic frames in the game development industry through cross analysis of individual game development companies. I would like to maximize my potential to capture relevant, valid, and as generalizable as possible epistemic frames relative to long term success in the game development industry.
Is there a recognized “best practice” methodology for uncovering and defining epistemic frames? Can anyone recommend any methodologies that have worked well for them? Or tools or strategies? This seems to me a preliminary step to model an epistemic game, which I hope to subsequently work on.
Of course a valid reason why a kid who has passed the exam but cannot actually really pass a simple problem is because we do not let them to think. We just give them the formula and thats it. But if we practice the kid to have some simple solving problems or just let them play word games like puzzle game their brain will start to develop and enhance their vocabulary learnings.