 |
 |
|
|
|
Pro/Con - Should We Question Questionnaires? |
|
|
 |
|
|
|
Survey use is on the rise as schools scramble to meet Engineering Criteria 2000. Can these assessment tools be mined for valuable data or do they yield mostly fool's gold? |
|
|
||||||||||||||||
|
|
|
|
||||||||||||||||||||||||||||||||||||||||
|
|
|
To prepare a valid questionnaire, one must first ensure that the survey seeks to determine information about something that actually exists. This facetious-sounding statement is neither flip nor trivial, and was first asserted by the mathematician Weierstrass more than 100 years ago. Indeed, many questionnaires seek answers to questions about things that do not exist. Specifically, the implied goal of the 10 steps presented in Vern Johnson's article is determining students' collective preferences. Almost always, however, the students will not collectively exhibit preferences that lead to a meaningful conclusion. In this case, the students' individual preferences cannot be combined to form a group preference, as the group preference does not exist-an idea advanced in Arrow's Theorem, for which Kenneth Arrow shared the 1972 Nobel Prize in economics, and which invalidates the results of most questionnaires. More recently, researchers such as Donald Saari have shown that pursuing answers to invalid questions can provide chaotic, albeit convincing, results of precisely the wrong conclusions. Second, there is great danger in the arbitrary imposition of scoring systems to the results of a questionnaire. Indeed, it is easy to show that alternative scoring methods lead to quite different conclusions from the same data, even when the questions are validly posed and the data are valid (see "Voodoo Mathematics" at Work, below). In short, questionnaires are very dangerous things, and those who have not been trained in the proper methods and mathematics would be well advised to avoid them. Unfortunately, this includes nearly everyone involved in the current assessment craze. -George Hazelrigg |
|
|
|
|
|
The quality movement has experienced much success in the industrial world, where results are paramount. Since quality is defined by customer satisfaction, customer surveys are fundamental to doing business. Though there is constant debate about whether students are "customers," my experience is that if surveys are carefully designed and administered, they are extremely useful as guides for improvement efforts. I believe that this is how ABET intends their use. |
|
|
|
|
||
|
"Voodoo Mathematics" at Work |
||
|
|
|
A professor decides to survey students to determine the instructional technique that they prefer be emphasized. Three options are considered: A) greater explanation of theory; B) more example problems; and C) more real-world case studies. Fifteen students provide their preferences: six students select A, five choose B, and four opt for C. Obviously, A is the preferred alternative, followed by B, and then C. Or is this the case? To be certain, the professor devises a scoring method and asks students to give scores from one (lowest) to five (highest). The results are as follows:
Satisfied, the professor spends more time on theory-to the dismay of the class. What went wrong? Taking a closer look at the students scores shows that six students preferred A to C to B, five preferred B to C to A, and four preferred C to B to A. So, asked to choose between A and B, they would vote 9 to 6 for B. Asked to compare A to C, they would select C by a 9 to 6 margin. And by a 10 to 5 vote they would prefer C to B. In reality, then, the students' clear preference is for option C, followed by B and then A-precisely the opposite ordering given by both the simple vote and the scoring system! Further, in this case the group actually had a clear preference-in most cases the situation is even worse, because the group does not have preferences that can be transitively ordered, and no scoring system would be valid.-G.H. |
|
|
The study of human preferences has been the subject of much attention in the fields of mathematics and economics for more than 200 years, and the mathematics of preference theory has been rather well developed. These fields must be studied by anyone who wishes to do a valid survey.
Johnson responds: