Jump to:
1. The Art of the Implausible | 2. Assertions, or Arguments?
3. A Problem of Implementation | 4. Extremely Extremist Extremism
5. Opinion Polls Make You Dumb

5. Opinion Polls Make You Dumb

What is it about the survey that carries the veneer of respectability? It seems that all people have to do is present a pie chart and say that it is the results of a “study” of Afghans or Pakistanis and suddenly they act like they’ve gained some kind of window into the minds of an entire population.

I bring this up because a presenter this afternoon showed slide after slide of pie charts supposedly indicating what Pakistanis think about various subjects. I get frustrated when people do that sort of thing. Here are the reasons:

1. Unrepresentative Findings. The presenter introduced the survey findings by calling them the results of a study of Pakistani university students. No mention of how many people participated in the survey–the presenter just gave percentages. I thought that would be an obvious detail to mention, as most everyone (I thought) knows that you can’t generalize to a population based on a dinky sample. But there was also no mention of whether the results were based on a random sample. When you’re trying to generalize percentages to the entire population of Pakistan, as this speaker seemed to be doing, getting a non-random sample of 5000 people isn’t really any different from getting a non-random sample of 100 people. There’s no way to know if the findings actually said anything about Pakistanis at all.
2. Biased/Ambiguous Questions. One of the first sets of results presented was the responses to the question “What threatens the Muslim Community?” The options, apparently, were “America,” “the West,” “India,” “Israel,” and “other.” If those really were the options, then the results are useless because it automatically told the respondents how the question was supposed to be answered. If the questions was just open ended, with no options to choose from, then the results are equally useless because there’s no way to know if people answered differently because they really think differently, or because it just didn’t occur to them to answer the same.
3. Un-Interpretability of Findings. Let’s make some really unrealistic assumptions and choose to believe that the previous question actually contained a whole list of possible answers, was asked to a large, random sample of people, and produced the five responses listed. What do those results indicate? When someone says America poses a threat to the Islamic community, what does that mean? Are we talking about a security threat? A non-military threat to their way of life? A threat to economic stability? Something else? When we say the Islamic community is threatened, who belongs to that community? The question assumed that the terms used were clear enough for all respondents to understand them in roughly the same way. There’s no reason to believe that assumption.
4. Irrelevance of Findings. Even if none of the previously-listed problems existed, the polls still tell us nothing. Several presenters talked about the fact that Afghans and Pakistanis can rattle off a whole litany of grievances that supposedly motivate them to do various things. Nearly every speaker talked about the need to address grievances. I think there’s an entire panel devoted to the subject tomorrow. Even if grievances are as important as people say they are, what does that have to do with actual operations or policies in the Afghanistan-Pakistan region? If you’re mad about U.S. foreign policy in the Middle East, does that mean you join the insurgents? Help the insurgents? Run for office? Blog? Complain to your neighbors? Join the military? Sit around and be angry? Reconcile yourself to the situation? There are hundreds of different actions that could spring from a single grievance. Knowing that people think about certain grievances more than others brings us no closer to actually being able to do anything to resolve conflicts.

Most opinion polls aren’t worth the time it takes to read the results, but the military seems to be fascinated with them. It’s like they suspect, deep in their heart of hearts, that if we could just poll people on the right questions we would suddenly understand this conflict well enough to win it. In fact, didn’t MG Flynn of ISAF write a report on this subject a few months ago, where he basically said that getting good intelligence in Afghanistan meant getting more opinion measures? There seems to be this absolute disconnect between the proposed courses of action for the military the desired end states that are supposed to result from those courses of action.

Argh. I’m done for the day.

Jump to:
1. The Art of the Implausible | 2. Assertions, or Arguments?
3. A Problem of Implementation | 4. Extremely Extremist Extremism
5. Opinion Polls Make You Dumb

Joshua Foust spends much of his time discussing the social and cultural aspects of American foreign policy. He works closely with Global Voices Online and other organizations to promote citizen media in underserved areas. He has lived in Kazakhstan and Afghanistan, and has extensive experience working on Post-Soviet Central Asia. He writes about the region at Registan.net, which is quoted routinely by the New York Times, the Washington Post, the Christian Science Monitor, Reuters, and World Politics Review. He is a regular contributor to the Columbia Journalism Review and World Politics Review, and contributes to Need to Know on PBS.

Source: Registan.net [All URLs lead to Error 404 -- these posts appear to have been removed]

See also:

• The CENTCOM Af-Pak Conference [UPDATED] [Removed at author's request]
• Dispatches from the CENTCOM AfPak Conference: Talking in Circles and Rushing to Nowhere [Google cache]
• Dispatches from the CENTCOM AfPak Conference: Assumptions Versus Argument [Leads to: Error 404]
• The Art of the Possible – Towards Afghanistan’s Political Settlement
• CENTCOM hosting AFG-PAK conference in June