POLI 300 PROBLEM SET #1B 02/11/09

QUESTION DESIGN: ANSWERS & DISCUSSION

Note 1. Some of these answers are considerably more detailed than those you were expected to provide. This will be typical of many of the Answers and Discussion that will be attached to your returned Problems Sets. You should always read these attachments with care, even if you did fine on the problem set; other possible answers may be noted and supplementary course material may be introduced. You should regard the Answers and Discussion attachments as basic course material — similar to the Course Pack, other handouts, and assigned readings. For this reason you should make sure to get a copy of each A&D even if you did not turn in the corresponding problem set.

Note 2. Unlike many (but not all) subsequent Problem Sets, these problems do not have clearly “right” or “wrong” answers. The point of the problem set is mostly to invite you to think about certain methodological problems.

Note 3. Remember that survey questions are always embedded in a broader questionnaire that may include more general questions. In ANES and similar surveys, a “filter question” often precedes an opinion question, e.g., “Do you have an opinion about [named issue] or is this something you really haven’t thought about much?” Then only people who answer “Yes, I have an opinion” are asked what their opinion is. Surveys can also have “intensity probes” following an opinion question (e.g., “Is this a strong opinion that you have or not so strong?”) or other types of follow-up questions. Such devices address many of the concerns that a number of students mentioned.

1. Probably Question (b) would elicit less apparent support for tax cuts and more for government spending, because its phrasing calls attention to lots of relatively uncontroversial “good things” that the spending could be used for. (The effect would be the opposite if “foreign aid,” “welfare,” higher salaries for government works, etc, were mentioned.)

2. Probably Question (a) would elicit less support for “sending American troops to assist the NATO mission in Kosovo” than (c), because (c) tells the respondent that this proposal is supported by “many Americans.” Question (b) really asks a different question: whether the respondent would endorse this policy if President Clinton decides to adopt it. Whether (b) would elicit more support than (c) or even (a) probably depends a lot on what the respondent thinks about President Clinton more generally (e.g., whether the respondent “approves of” the President, which in large part depends in turn on whether the respondent is a Democrat or Republican party identifier).

3. Some respondents may react to Question (b) as if it asks them of whether they approve of “communism” (etc.), so quite a few may respond negatively. Question (a) asks whether Congress should pass a law that abridges freedom of speech by forbidding such speeches. In one pair of actual polls, 39% or responded answered (a) positively (i.e., such speeches should be “forbidden”), while at the same time 56% answered (b) negatively (i.e., such speeches should be “no allowed”). Of course, any question asking Americans whether they support “freedom of speech” in general would get almost unanimous support, which drops off dramatically when the question (like both of these) gives examples of unpopular speech.

4. Though almost all Americans would endorse freedom of the press in general, quite a few might answer (a) negatively if it were asked before (b). But if (b) is asked first (which almost everyone would answer positively) changes the context of (a) and probably would increase positive responses to (a).

5. (a) This question seems to be straightforward and reasonably unslanted. Note that a purely factual question (reporting behavior) is being asked — not one about attitudes or opinions. However, since some stigma may attach to the use of food-stamps (as a form of “welfare”), some respondents who actually use food stamps may be reluctant to acknowledge this fact to an interviewer (again the “social desirability” factor). It may help to introduce the question with a preface that legitimizes a (truthful) “yes” answer, e.g., “In talking with people, we find that quite a few families from time to time need to use food stamps. How about you — does your family use food stamps?” (Compare with #1 above.)

There are at least two ambiguities in the question: first, it should specify “do you at present use . . . ” or “have you ever used. . . .” (according to whatever the researchers intend); second, the word “family” might be interpreted by some respondents as (in effect) “extended family,” so the question might better refer to “your household” (assuming that is what the researchers intend). Some students assumed the researchers were really interested in the respondent’s financial status and concluded that the question should ask about financial status more directly. But the researchers may well be interested in food stamp usage per se.

(b) Many students noted that this closed-form question is deficient because it provides only two relatively “extreme” options. In principle, this is OK because the stem of the question (like the general instructions for the Student Survey) asks only “which of the following best represents your opinion . . . ?” The trouble is that Option (1) is an extreme and (arguably) unconstitutional option stated in an extreme fashion. Option (2) invokes (some of the) constitutional language (of the Second Amendment) and in fact may allow for many (but not all) kinds of gun regulations. Thus, while some students said that the options were too “black and white,” the problem is more that the second option is “gray” and therefore more appealing. A poll that gave respondents these two choices would (I expect) produce results that show that a very large proportion of the population prefers option (2). It therefore could be (misleadingly) claimed that a large majority of the public is “against gun control,” making the survey a “push poll” or “propaganda poll.” In a scientific survey, respondents should be presented with a more balanced format of options.

(i) it includes a number of “big” and somewhat technical words that some respondents may not fully understand;

(ii) the preface of the question pushes respondents to favor [any and all] environmental measures; and

(iii) it is “double-barreled” (see Weisberg et al., pp. 87-88), asking two questions at once: (a) “should there be more recycling?” and (b) “are economic incentives the best way to encourage this?”

Questions (b) and (c) might be employed in “push” or “propaganda” polls that are not so much trying to measure the opinions of respondent (or even to change their opinions) as to provide rhetorical ammunition in public policy debates.

6. “Health” and “education” are viewed in (almost) universally positive light (social desirability once again). The direct and “unbalanced” question in the form of (a) would undoubtedly produce high levels of support, which would be considerably reduced when a “balanced format” question such as (b) or (c) is used (see Weisberg et al., p. 86). The choice between (b) and (c) raises the question of whether it makes a difference which of the two balanced views is stated first and which second. One might conjecture that the first view might get slightly more support in a written (self-administered) questionnaire (there is evidence that the candidate listed first for a particular office on a ballot gets a slight advantage, especially the candidates are not well-known and their party affiliations are not shown [see #2 above]), while the second view might get slightly more support in an oral (telephone or personal) interview (since some respondents may have forgotten the first view when the time comes to answer). (See Weisberg et al., pp. 83-84.) Some students suggested that “some people” suggests a majority and “other people” suggests a minority and that there would be some bias in favor of the apparently majority opinion.

Concluding Note. All these remarks (including student remarks) about how different ways of phrasing questions may produce different distribution of responses are essentially speculative. Any such speculations about question design can be tested by using a survey experiment: use a random mechanism to split the sample of respondents into two groups, give one group the question(s) excluding, and the other group the question(s) including, party affiliation, and see whether there is a significant difference between the two groups. (There will be a small difference in any case due to sampling error, as has since been discussed in the handout on Random Sampling and in class. The question will be whether the differing questions produce different responses to a greater degree than we would expect on the basis of sampling error alone.)