As most BCC readers will know, I have fielded a pilot survey on people’s experiences of the Book of Mormon over the last several days. Thanks to all who have participated!
I want to comment briefly on an unexpected aspect of this survey. I’ve administered a number of surveys before, all in South America. For each of those experiences, very few respondents contact the project administration team to ask for additional details or to complain about the survey instrument. For this survey, by contrast, a little over 5% of respondents have emailed me, usually to complain.
Also surprising to me is the tone of the complaints. A large proportion of email correspondents have called into question my motives for carrying out the survey. One category of emails operates under the assumption that the survey assumes the Book of Mormon to be genuinely divine and ancient, and often advances the hypothesis that I am an operative of some unspecified plan conceived in the Church Office Building in Salt Lake City. Another category works from the premise that the survey is intended to distinguish “intellectual” Mormons from allegedly under-informed rank-and-file Mormons, and develops the theory that I intend to use the results to belittle the latter category of my coreligionists.
Just for the sake of clarification, let me state that my motive in doing this survey is pure intellectual curiosity. I want to know more about how people think and feel regarding the Book of Mormon. That desire, in conjunction with the fact that internet surveys are almost free, is the entire causal account of how this survey came to be. I have no connection with Powers that Be, and I don’t intend to belittle any category of respondent in analyzing and discussing the results of this survey. Indeed, my primary attitude toward all the respondents is one of gratitude.
Why, then, the suspicion? Why the aggressive tone that often accompanies it? Let me offer a few snippets from emails I’ve received calling into question my motives. “This survey is probably an unwitting tool of the adversary.” “Your survey can’t prove the Book of Mormon is true, so grow up.” “I won’t have you sneering at the good members of my church.” Such sentiments flabbergast me.
Another set of emails was less surprising, although I certainly expected fewer of them than I received. These messages do not address my motives, but simply call into question my competence. Many emails complain that the questions I ask in the survey are not specific enough to capture their particular experience of the Book of Mormon. To this complaint, I respond that I agree. In trying to learn about the experiences of a broad collection of people, it is probably inevitable that there will be some simplifications that distort (in small and large ways) the experiences and beliefs of many respondents. Indeed, I feel that the survey doesn’t fully capture my own perspectives. Nonetheless, it asks a collection of questions that will hopefully help us to at least sketch the broad details of the experiences of a very diverse set of people. Carrying out a survey, I think, entails a decision to sacrifice depth for breadth.
Other messages complain about specific questions, claiming that they are problematic, misleading, or invalid. Most of the time, these messages contain explanations of how the respondent answered and why they felt dissatisfied with the question. Ironically, most of these explanations indicated that the items in question were functioning as I intended them to do — my correspondents had simply misinterpreted the intentions of the questions. These complaints, thus, serve as valuable qualitative evidence of the validity of the indicators, their authors’ intentions notwithstanding.
A number of complaints center around one particular question, which may be problematic when applied to the body of all respondents. This question asks if people agree that, even if they don’t know where the Book of Mormon comes from, they are convinced that the LDS church is true. For respondents in general, this question contains two distinct attitude objects (Book of Mormon, LDS church) and is therefore double-barreled. It will be used analytically only for respondents who have clearly indicated uncertainty about the origins of the Book of Mormon on other questions, removing the double-barreled nature of the question. I asked the question of other respondents simply because (a) there’s no real cost to adding it in, and (b) I’m curious to learn about how it will perform empirically among people who aren’t uncertain about the origins of the Book of Mormon; it’s a matter of substantive interest to me whether the Book of Mormon attitude object or the LDS church attitude object will predominate in such answers.
More generally, I am willing to agree to a blanket statement that any of the questions asked in the survey may not work as intended. Fortunately, I have a plan! A major analytic goal for this survey is to put together empirical evidence regarding the validity and reliability of each question asked; we lack a validated set of measures for most attitudes regarding the Book of Mormon, and I hope this survey will be able to provide some. The analysis will not provide definitive information about the quality of each question, but it will produce some hints about which indicators may not be performing as expected. Those will be thrown away, and the most useful will be retained for more substantive analysis of this survey and for use in future surveys.
While this category of complaints is clearly legitimate, and the emails they produce often helpful, I was once again surprised by the vehemence and belittling character of many of the messages I received. Several correspondents called into question my competence as a survey researcher and, at times it seemed, a human being. “Did you have a professor look over your survey before putting it online?” “Think twice before messing up like this again!” “At least you spelled most of the words right.” (I should note that these comments were at least partially offset by a collection of gratifying compliments by Mormon academics and professionals with various connections to survey research.)
I very much appreciate the degree of engagement that these messages reflect. Nonetheless, it is worth noting that none of my past survey experiences have entailed such combative correspondence with respondents. What is it about this survey and these respondents that produced such intensity and negativity?
Could it be the medium? My past survey work was done through face-to-face interviews, rather than over the internet. The net makes response to the survey’s creator simpler and more immediate, and it has the well-known effect of increasing verbal aggression by eliminating visual and auditory cues. Perhaps all web surveys result in such active and, ahem, vigorous engagement among respondents.
Could it be the subject matter? Certainly attitudes regarding the Book of Mormon are salient and divisive among the population that I surveyed. Yet it is not clear to me that they are distinctively salient and divisive in comparison with other work that I have done. For example, I administered a survey regarding political participation and attitudes in Venezuela in early 2003, in the middle of a massive countrywide political showdown between supporters and opponents of Venezuelan president Hugo Chavez. Could political issues and engagement, in that place and at that time, really be seen as much less salient and divisive than beliefs about the Book of Mormon among people with ties to the Latter Day Saint movement?
Or could it be the population? Are people in the Mormon world just, um, special?