CHAPTER 1
Why concern yourself?
A PLEA FOR THE IMPORTANCE OF ASKING GOOD QUESTIONS
What is the need for this first book on question wording? No one else has considered it necessary to devote a whole book to the subject. A chapter or two has always seemed enough before. Articles of several pages frequently appear in professional journals here and there. What more can a book do for question wording? If it all boils down to the familiar platitudes about using simple, understandable, biasfree, non-irritating wordings, all of us recognize these obvious requirements anyway. Why say more?
Oblivious of the obvious
One reason for elaborating on the subject is that all of us, from time to time, forget these requirements. Like some church-goers, we appear to worship the great truths only one day a week and to ignore them on working days. Or we remember a certain example, but fail to see how it applies to other situations. In combatting our very human frailty, a more provocative set of examples and a detailed list of points to consider may be more helpful than the isolated examples and the broad generalities which we now so often disregard.
The import IS important
Another reason for separate treatment is to help us realize how basic the phrasing of questions is to worthwhile research. As it is, we may look upon the few available examples of differences wrought by wording as mere freak occurrences. There is no danger of our making such weird mistakes, we think. Besides, if the question "works," it must be a good question.
By now, we should realize that the fact that something "works" does not mean that it works correctly. The Literary Digest poll seemed to "work" all right until 1936. The methods of subsequent election polls seemed to "work" all right until 1948. In both of these cases, attention has concentrated on sampling difficulties as causes of the wide margins of error. In the case of the Digest the faulty sample was no doubt the prime contributor to the error. At least one expert had predicted how and why this faulty sample would lead to error (1). In the more recent case, sampling is only one of the many possible causes, according to the Social Science Research Council committee (2).
Be that as it may, people — laymen and qualified experts alike — are impressed that sampling is an important feature of the survey method. Even if we agree that they are right, however, let us not forget that the Gallup Poll in 1948 overestimated the Dewey vote by only 4.4 percentage points, and the Digest poll in its 1936 prediction of the Landon vote was only 19 percentage points too high.
I use the word "only" advisedly because, as we shall see, survey differences resulting from changes in words or phrases sometimes amount to considerably more than a 19 point error! Perhaps, if we grant that by emphasizing the importance of sampling the election upsets have contributed to advances in sampling techniques, we should deplore the fact that nothing similar has happened to concentrate attention on the importance of question wording.
Tens versus tenths
At the present stage of development of the survey method, improvements in question wording and in other phases can contribute far more to accuracy than further improvements in sampling methods can. I don't mean that the sampling experts should stop seeking further improvements, trying to knock a few more tenths of a percent off the statistical error. But, while they are laboring with tenths of a per cent, the rest of us are letting tens of per cents slip through our fingers. As Frederick Stephan of Princeton University has remarked, "It's like using a surgeon's scalpel in a butcher shop."
The experts said so
Even as far back as 1936, however, when the Literary Digest was riding to its fall, a group of experts called question wording the Number One problem. Howard T. Hovde asked a sample of researchers what they saw as the principal defects of commercial research (3). Here are their most frequently mentioned criticisms:
Improperly worded questionnaires 74%
Faulty interpretations 58
Inadequacy of samples 52
Improper statistical methods 44
Presentation of results without
supporting data 41
Since three experts in every four pointed their fingers at question wording, it seems that the subject should have been worthy of concentrated attention. The specialists in statistics and sample theory certainly didn't let these expert opinions dampen their efforts. It's too bad that question worders weren't more stimulated by Hovde's report.
Samuel A. Stouffer and his collaborators in their recent monumental work on the American soldier (4) came to a similar conclusion: "To many who worked in the Research Branch it soon became evident that error or bias attributable to sampling and to methods of questionnaire administration were relatively small as compared with other types of variation — especially variation attributable to different ways of wording questions."
Jack-of-all-trades
Probably the reason that the question worder hasn't done more to advance his phase of research is that he just doesn't exist, at least not as a specialist. The statistician is the only one among us who has a specialty. All the rest of the work comes under the jurisdiction of a jack-of-all-trades. This man's job is to develop the questionnaire, pretest and revise it, have it printed, select, train, and supervise the interviewers, conduct the survey, analyze the results, write the report, and present the findings. His attention is necessarily divided. Question phrasing is but one part of the complex machinery which he must put together and operate. Small wonder that this all-around type of researcher has not had much time to formulate detailed statements on question wording.
Not special pleading
Having said that sample theory has moved ahead through specialization, I do not mean to imply that question wording needs also to be treated as a specialty. In actual practice, wording cannot be thought of as being in a vacuum apart from other phases of the survey method. It would be extremely naive to expect a single question to provide "some magic way of reducing a complex matter of people's attitudes, wishes, and aspirations to some simple wording which will not bias the returns." Most researchers will agree with Daniel Katz (5) that the solution usually requires "an integrated questionnaire which explores the problem comprehensively from many angles."
The fact that this book deals with the subject of question wording by itself and usually in terms of improving a single question does not mean that the other phases of the work can be slighted. The steel square is a useful tool to the carpenter and...
