bigblue

Pro/Con - Should We Question Questionnaires?

Survey use is on the rise as schools scramble to meet Engineering Criteria 2000.  Can these assessment tools be mined for valuable data or do they yield mostly fool's gold?

Prism Forum

What do YOU think?
Share your opinion with us on this issue and suggest other Pro/Con topics for PRISM. See
www.asee.org/forum  or e-mail us at  prism@asee.org

In the October 1998 issue of PRISM, Vern R. Johnson's article "Ask and Ye Can Assess" offered a blueprint for developing questionnaires to use as assessment instruments. The guide suggested a 10-step process that included, among other things, deciding which attributes to assess, selecting a scoring system for the responses, and using the interpreted data to justify changes in courses or programs.

George Hazelrigg, a mechanical and aerospace engineer who has taught at several engineering schools, took issue with the methods described in Johnson's article. The following are Hazelrigg's concerns and Johnson's response.

redtri

Lies, Damned Lies, and Questionnaires

Illustration by Ken DubrowskiThe study of human preferences has been the subject of much attention in the fields of mathematics and economics for more than 200 years, and the mathematics of preference theory has been rather well developed. These fields must be studied by anyone who wishes to do a valid survey.

To prepare a valid questionnaire, one must first ensure that the survey seeks to determine information about something that actually exists. This facetious-sounding statement is neither flip nor trivial, and was first asserted by the mathematician Weierstrass more than 100 years ago. Indeed, many questionnaires seek answers to questions about things that do not exist. Specifically, the implied goal of the 10 steps presented in Vern Johnson's article is determining students' collective preferences. Almost always, however, the students will not collectively exhibit preferences that lead to a meaningful conclusion.

In this case, the students' individual preferences cannot be combined to form a group preference, as the group preference does not exist-an idea advanced in Arrow's Theorem, for which Kenneth Arrow shared the 1972 Nobel Prize in economics, and which invalidates the results of most questionnaires. More recently, researchers such as Donald Saari have shown that pursuing answers to invalid questions can provide chaotic, albeit convincing, results of precisely the wrong conclusions.

Second, there is great danger in the arbitrary imposition of scoring systems to the results of a questionnaire. Indeed, it is easy to show that alternative scoring methods lead to quite different conclusions from the same data, even when the questions are validly posed and the data are valid (see "Voodoo Mathematics" at Work, below).

In short, questionnaires are very dangerous things, and those who have not been trained in the proper methods and mathematics would be well advised to avoid them. Unfortunately, this includes nearly everyone involved in the current assessment craze.

-George Hazelrigg

redtri

Numbers Are Not the Whole Story

Illustration by Ken DubrowskiJohnson responds:
Arguments about the applicability and accuracy of surveys are a never-ending journey for many engineers and physical scientists.  Surveys measure people's attitudes and opinions-"soft" qualities-and one must constantly worry about what survey results mean and about their accuracy.  This softness is a particular problem for people schooled in physical science fields, where meters and gauges are available to make measurements.

 

The quality movement has experienced much success in the industrial world, where results are paramount. Since quality is defined by customer satisfaction, customer surveys are fundamental to doing business. Though there is constant debate about whether students are "customers," my experience is that if surveys are carefully designed and administered, they are extremely useful as guides for improvement efforts. I believe that this is how ABET intends their use.

"Voodoo Mathematics" at Work

A professor decides to survey students to determine the instructional technique that they prefer be emphasized. Three options are considered: A) greater explanation of theory; B) more example problems; and C) more real-world case studies. Fifteen students provide their preferences: six students select A, five choose B, and four opt for C. Obviously, A is the preferred alternative, followed by B, and then C. Or is this the case?

To be certain, the professor devises a scoring method and asks students to give scores from one (lowest) to five (highest). The results are as follows:

  • Six students:A-5,B-1,C-2
  • Four students:A-1,B-4,C-2
  • One student:A-1,B-5,C-2
  • Four students:A-1,B-2,C-3
  • Totals: A-39 pts, B-35 pts, C-34 pts

Satisfied, the professor spends more time on theory-to the dismay of the class. What went wrong?

Taking a closer look at the students scores shows that six students preferred A to C to B, five preferred B to C to A, and four preferred C to B to A. So, asked to choose between A and B, they would vote 9 to 6 for B. Asked to compare A to C, they would select C by a 9 to 6 margin. And by a 10 to 5 vote they would prefer C to B. In reality, then, the students' clear preference is for option C, followed by B and then A-precisely the opposite ordering given by both the simple vote and the scoring system!

Further, in this case the group actually had a clear preference-in most cases the situation is even worse, because the group does not have preferences that can be transitively ordered, and no scoring system would be valid.-G.H.



return to PRISM online

pointer