In 'Voodoo Mathematics at Work,'  ("Should We Question Questionnaires?" February 1999, p. 14) George Hazelrigg presents an alternative understanding of class preferences.

However, his alternative treats the preferences as if they were strictly a rank ordering of the three choices. In that case, the choices would be 1, 2, or 3 (with "3" being the most preferred) and the totals would have been A-27, B-29, C-34, giving the ordering C B A, as Mr. Hazelrigg notes in his analysis.

However, as he presented the example, the students did not just rank the three alternative instructional techniques. They rated each technique separately from 1 (not desirable) to 5 (very desirable), which adds a sense of passion of preference. The original total scores take this passion into consideration, as well as the implicit ranking of the techniques. These "passion quotients" (one score minus another) for the example given would result in scores of +4 for A-B, +5 for A-C, and +1 for B-C, which are exactly the differences among the original total scores.

Using the original total scores to select the preferred class technique would have, to varying degrees, upset nine students mildly. However, using Mr. Hazelrigg's alternative would have greatly upset six students. The best solution to the lack of a true group preference would be, if possible, to divide the class into two separate sections (theoretical vs. applied).

Mr. Hazelrigg says, "Second, there is great danger in the arbitrary imposition of scoring systems to the results of a questionnaire." I fully concur. That is why the scoring system is best developed along with the questionnaire itself so that one is certain what the responses do and do not say. In Mr. Hazelrigg's example, the original scoring method gives one perspective on the problem-but not all perspectives. Similarly, his alternative scoring method gives another perspective on the problem, but ignores the perspective of the original method. Therefore, it is sometimes useful to employ both methods together to get a better understanding of the group of respondents-rank order the choices and rate each one individually.

Kevin D. Shell
Purdue University

We may all agree with George Hazelrigg that questionnaires should be questioned. But when it comes to educational assessment, what types of questions should we be asking about questionnaires? Hazelrigg's comments on Vern Johnson's article raise our consciousness about one type of question: How do scoring systems unwittingly affect our understanding of group analysis of individual preferences?

But two other questions about questionnaires should be highlighted. First: Is tracking the effects of scoring systems on collective outcomes an insurmountable problem? No. Software now exists that allows an analyst to review the results under different scoring systems to detect intransitivity and preference reversals. In fact, the software can indicate if the results from a default scoring system represent a strong or weak consensus.

And second, what types of data (besides preferences) can be (or should be)

analyzed from Web-based questionnaires? Unfortunately, in this regard, most engineering assessment work seems to be concerned with taking what has been done traditionally in a paper-and-pencil world and converting to electronic form in an unimaginative way. At Stevens Institute of Technology, we are designing and implementing an assessment system based on strategic and tactical data collection via the Web that promotes continuous quality improvement by students, teachers, and administrators.

Arnold B. Urken
Stevens Institute of Technology

In his Voodoo Mathematics example, George Hazelrigg uses a five-point scale when there are only three items to be evaluated by preferential ballot. I think this is the cause of the strange result he gets. If one uses a three-point scale then the "correct" result (compared to his pair-wise matching) is obtained.

I'm no expert in the relevant mathematics, but it appears that for preferential balloting of this type to be correct, the range of the ranking scale must not exceed the number of items being ranked. (That is, if "n" items are to be ranked then the ranking scale used must be equal to or less than "n.")

Norman Fortenberry
Rensselaer Polytechnic Institute

Thanks for your discussion of questionnaires in the February 1999 issue of PRISM. My only complaint is the profanity used in the title. Why? Is our vocabulary so limited that we need return to the language we gave up in junior high school?

Leo Setian
John Brown University

The "profanity" that Setian refers to is the subhead  "Lies, Damned Lies, and Questionnaires," which was derived from 19th-century British statesman Benjamin Disraeli's famous utterance about the sometimes dubious uses of statistics.
  -Ed.

Letters Via E-mail
Send letters to prism@asee.org .

Because of space limitations, not all letters can be published, and those that are may be abridged.

return to PRISM online