The Risks of Being a Questioning Mormon

As most BCC readers will know, I have fielded a pilot survey on people’s experiences of the Book of Mormon over the last several days. Thanks to all who have participated!

I want to comment briefly on an unexpected aspect of this survey. I’ve administered a number of surveys before, all in South America. For each of those experiences, very few respondents contact the project administration team to ask for additional details or to complain about the survey instrument. For this survey, by contrast, a little over 5% of respondents have emailed me, usually to complain.

Also surprising to me is the tone of the complaints. A large proportion of email correspondents have called into question my motives for carrying out the survey. One category of emails operates under the assumption that the survey assumes the Book of Mormon to be genuinely divine and ancient, and often advances the hypothesis that I am an operative of some unspecified plan conceived in the Church Office Building in Salt Lake City. Another category works from the premise that the survey is intended to distinguish “intellectual” Mormons from allegedly under-informed rank-and-file Mormons, and develops the theory that I intend to use the results to belittle the latter category of my coreligionists.

Just for the sake of clarification, let me state that my motive in doing this survey is pure intellectual curiosity. I want to know more about how people think and feel regarding the Book of Mormon. That desire, in conjunction with the fact that internet surveys are almost free, is the entire causal account of how this survey came to be. I have no connection with Powers that Be, and I don’t intend to belittle any category of respondent in analyzing and discussing the results of this survey. Indeed, my primary attitude toward all the respondents is one of gratitude.

Why, then, the suspicion? Why the aggressive tone that often accompanies it? Let me offer a few snippets from emails I’ve received calling into question my motives. “This survey is probably an unwitting tool of the adversary.” “Your survey can’t prove the Book of Mormon is true, so grow up.” “I won’t have you sneering at the good members of my church.” Such sentiments flabbergast me.

Another set of emails was less surprising, although I certainly expected fewer of them than I received. These messages do not address my motives, but simply call into question my competence. Many emails complain that the questions I ask in the survey are not specific enough to capture their particular experience of the Book of Mormon. To this complaint, I respond that I agree. In trying to learn about the experiences of a broad collection of people, it is probably inevitable that there will be some simplifications that distort (in small and large ways) the experiences and beliefs of many respondents. Indeed, I feel that the survey doesn’t fully capture my own perspectives. Nonetheless, it asks a collection of questions that will hopefully help us to at least sketch the broad details of the experiences of a very diverse set of people. Carrying out a survey, I think, entails a decision to sacrifice depth for breadth.

Other messages complain about specific questions, claiming that they are problematic, misleading, or invalid. Most of the time, these messages contain explanations of how the respondent answered and why they felt dissatisfied with the question. Ironically, most of these explanations indicated that the items in question were functioning as I intended them to do — my correspondents had simply misinterpreted the intentions of the questions. These complaints, thus, serve as valuable qualitative evidence of the validity of the indicators, their authors’ intentions notwithstanding.

A number of complaints center around one particular question, which may be problematic when applied to the body of all respondents. This question asks if people agree that, even if they don’t know where the Book of Mormon comes from, they are convinced that the LDS church is true. For respondents in general, this question contains two distinct attitude objects (Book of Mormon, LDS church) and is therefore double-barreled. It will be used analytically only for respondents who have clearly indicated uncertainty about the origins of the Book of Mormon on other questions, removing the double-barreled nature of the question. I asked the question of other respondents simply because (a) there’s no real cost to adding it in, and (b) I’m curious to learn about how it will perform empirically among people who aren’t uncertain about the origins of the Book of Mormon; it’s a matter of substantive interest to me whether the Book of Mormon attitude object or the LDS church attitude object will predominate in such answers.

More generally, I am willing to agree to a blanket statement that any of the questions asked in the survey may not work as intended. Fortunately, I have a plan! A major analytic goal for this survey is to put together empirical evidence regarding the validity and reliability of each question asked; we lack a validated set of measures for most attitudes regarding the Book of Mormon, and I hope this survey will be able to provide some. The analysis will not provide definitive information about the quality of each question, but it will produce some hints about which indicators may not be performing as expected. Those will be thrown away, and the most useful will be retained for more substantive analysis of this survey and for use in future surveys.

While this category of complaints is clearly legitimate, and the emails they produce often helpful, I was once again surprised by the vehemence and belittling character of many of the messages I received. Several correspondents called into question my competence as a survey researcher and, at times it seemed, a human being. “Did you have a professor look over your survey before putting it online?” “Think twice before messing up like this again!” “At least you spelled most of the words right.” (I should note that these comments were at least partially offset by a collection of gratifying compliments by Mormon academics and professionals with various connections to survey research.)

I very much appreciate the degree of engagement that these messages reflect. Nonetheless, it is worth noting that none of my past survey experiences have entailed such combative correspondence with respondents. What is it about this survey and these respondents that produced such intensity and negativity?

Could it be the medium? My past survey work was done through face-to-face interviews, rather than over the internet. The net makes response to the survey’s creator simpler and more immediate, and it has the well-known effect of increasing verbal aggression by eliminating visual and auditory cues. Perhaps all web surveys result in such active and, ahem, vigorous engagement among respondents.

Could it be the subject matter? Certainly attitudes regarding the Book of Mormon are salient and divisive among the population that I surveyed. Yet it is not clear to me that they are distinctively salient and divisive in comparison with other work that I have done. For example, I administered a survey regarding political participation and attitudes in Venezuela in early 2003, in the middle of a massive countrywide political showdown between supporters and opponents of Venezuelan president Hugo Chavez. Could political issues and engagement, in that place and at that time, really be seen as much less salient and divisive than beliefs about the Book of Mormon among people with ties to the Latter Day Saint movement?

Or could it be the population? Are people in the Mormon world just, um, special?

Comments

  1. Can I see the survey? Or better yet, can I take it?
    I’m sorry to hear that you’ve had such bad responses.
    Perhaps you made your email address to easily accessible :)

  2. It is amazing to me that Mormons can be as impossibly rude online as they are. We fail tremendously in this respect. I know this because I’m among the worst offenders. Still, the comments and questions you received seem over the top. Yet if you were to print out what some of these people wrote and meet them face-to-face to talk about it, I’d wager that most of them would be sheepishly embarrassed about it all.

    At least, that would be my hope.

  3. sister blah 2 says:

    Thanks for the follow-up JNS. As someone who is doing some survey work in connection with my dissertation, I got a nice chuckle out of this post. None of my survey participants have ever emailed me to complain! Though I do occasionally get both substantive and nit-picky feedback from colleagues at conferences.

    I took your survey and was also a little bit bugged by the “even if they don’t know where the Book of Mormon comes from, they are convinced that the LDS church is true” question, though I didn’t email you. But hey as long as I’m commenting here, I’ll throw in another thought. In the “which book of scripture is the quote from” section, I think the “I don’t know” option is too easy an out. I could have made some educated guesses for some of them, but since I wasn’t at all sure, it was tempting to say “I don’t know.” I’m not sure what your intended/preferred behavior would be in that scenario, but that’s my 2cents.

    Carry on. :-)

  4. Jessawhy, the link to the survey is in this post. I should note that the respondents have answered the questions just as I asked them to do; even those who were a bit unkind afterward have done me a service, and I’m grateful to them.

    Steve, I hope so, too.

    SB2, I think feedback is great; I really did appreciate the emails that were substantive and not aggressive. I’m hoping that the clarification above helps on the Book of Mormon/LDS church question. On the identify-the-book section, I wanted the “don’t know” option as an easy out. It’s not too difficult to guess correctly when there are only 4 options, especially since the Old Testament is often an implausible choice. So for people tempted to guess, the “don’t know” option should reduce their propensity to do so. Solid majorities got each of these questions right, in any case.

  5. Mark IV says:

    Jessaawhy,

    You can take the survey here.

    Steve, to be fair to yourself, I don’t think you have ever called J. an unwitting tool of the adversary.

  6. Mark, to be fair to my anonymous correspondent, I obviously wouldn’t be in any position to judge whether I’m an unwitting tool of the adversary. Being unwitting and all.

  7. Oooh! Oooh! Jessawhy used a wrong variant of to/too/two! Now we can mock and belittle her and question her competence to post on the internet!

    Who wants to go first?

  8. Latter-day Guy says:

    I think that the aggression is not only due to the issue being a divisive one. It is possible that the importance of apologist writing in Mormon thought makes LDS members immediately defensive when it comes to the Book of Mormon.

    I am intrigued, when I interact with members of other faiths, with their approach to scripture. Catholics (whose scholarly interpretations I generally favor) tend to worry about the meaning of scripture far more then they worry about its historicity. Meanwhile, in some Evangelical Protestant groups, there is greater focus on proving things like the Ark/universal flood, crossing the Red Sea, and literal readings of the creation account.

    In Mormon writing, the ratio of apologetic to analytical/interpretive work is greater than in, say Catholicism. Indeed, much writing that intends to interpret ends up spending a great deal of space trying to prove (or at least show as plausible) the Book of Mormon as an ancient record. This is not necessarily a bad thing, but it does color our approach to scripture study, and tends (I feel) to lessen the depth of our scriptural interpretation/application.

  9. Other messages complain about specific questions, claiming that they are problematic, misleading, or invalid. Most of the time, these messages contain explanations of how the respondent answered and why they felt dissatisfied with the question. Ironically, most of these explanations indicated that the items in question were functioning as I intended them to do — my correspondents had simply misinterpreted the intentions of the questions.

    I think the question is what the intention of the question was. While I was quite pleased and interested with the survey I have to confess there was one question which was very confusing to me. (The counterfactual one) Could you clarify what you were attempting to do with that question?

    Are you saying it was just a marker question so there would be at least a one poor question the survey to compare the success of other questions against?

  10. Wait, Mark, I thought we had agreed — J. is a _witless_ tool of the adversary, not _unwitting_. It’s not quite the same thing (though there may be significant overlap).

  11. Left Field says:

    I’m curious about your intent regarding the question about the percentage of indigenous Americans descended from Lehi. Assuming Lehi existed, and left descendants in America, it is likely that Lehi would be a very tiny proportion of the ancestry of Native Americans, and yet be an ancestor of nearly all of them. A question asking what proportion of Native American ancestry is Lehite would likely give a very different result than the question you asked. When I took the survey, I couldn’t help but think the second question might be the one you intended to ask. But perhaps not.

  12. Feedback I got from some educated, orthodox LDS on your survey was that they suspected that it might be an attempt to cross-examine them, so to speak. Many of them ditched the survey once they got to the final pages and realized that questions were being posed about their level of education and activity in the Church. This was because of the scripture passages section and the “name the speaker” section. The reasoning was that they couldn’t decide if one particular passage was in the New Testament, the D&C, or the Book of Mormon. So they guessed one of them but then once they got to the end of the survey, they felt like their incorrect answer would be used against the body of the Church in the following manner:

    X number of respondents said they had graduate or undergraduate degrees and were very active members of the Church in terms of Church attendance but X% of these same respondents wrongly attributed X passage to the Book of Mormon; ergo, even the most educated, active LDS don’t know very much about the Book of Mormon doctrinally; ergo, the responses of the same respondents on other matters such as the percentage of native Americans who are descendants of Lehi, the percentage of text quoted from the KJV or relating to origins of the Book of Mormon are or can be fodder for ridiculing the body of the Church and its beliefs. (All of these issues being the hot button topics of anti-Mormons.)

    As I said, I know some who simply exited the survey right at the end without finishing and submitting the result based on reasoning similar to this (perhaps not the exact reasoning I have presented here since I am trying to reconstruct a general line of reasoning that is plausible).

    I should note that if orthodox LDS are exhibiting this level is distrust at a survey like this, it might seem uncharitable to criticize them for it. This isn’t coming out of thin air. What is causing these people to have such a reaction? Perhaps they are victims of such treatment frequently enough on the internet that they suspect the same from the survey. If they are victims, perhaps their defensive response is not so difficult to understand.

  13. Latterday guy I think that’s because Mormons tend to adopt a very realist approach to spiritual phenomena. Thus the meaning of scriptures (I sense) is less the narrative value than the reality behind which generates the narrative. That’s not to neglect the narrative. But I think there’s a sense where we all adopt a basic American stance towards stories.

    This leads folks (IMO) to initial try to defend traditional readings where often the miraculous is more defended and then if it can’t be defended the story tends to be devalued a lot or at least recontextualized.

    I think that most apologetics is of this movement. (IMO)

    While I think we are in dangerous territory if we neglect narrative too much I think seeing the scriptures as primarily literary with the truths being literary truths is quite disturbing to many Mormons.

  14. I thought it was a thought-provoking survey (and I confess, I guessed on the Name-the-Speaker portions) I hope you’ll blog about the data results of the query and not just keep it a personal intellectual journey. =)

    The question that startled me to see was the one about Joseph being a fallen prophet, and something like this: “I think these people might be onto something.” The wording was amusing. =)

  15. Steve Evans says:

    John, the survey didn’t come out of thin air, either. How did they hear about it? If someone came to BCC, saw what we were about and saw a link to the survey and still didn’t trust it, then I’d say there is something going on.

  16. Steve,

    But it got passed on to various other outlets, posted at MAD boards, and so on. So not everyone came to the survey via BCC.

  17. How most of the people I know who saw it/took it got it was that it was forwarded to them by email and the original source was unknown to them.

    I got my own from Ronan who emailed the EMSA list with it. If someone on the EMSA list emailed their gmail contacts with it, 99% of them wouldn’t know who Ronan was.

  18. Left Field, I thought the same thing when answering that question.

    John F, I think that’s a good point I’d never thought about. I think one problem is that so many surveys don’t quite map on to LDS belief which ends up distorting it. (I’m thinking of some of the questions of the Pew survey that’s been discussed at many blogs the past week or so) There’s a sense that surveys often distort as much as they inform.

    Perhaps some of us are a tad too touchy about all this. (I’m thinking here of my own feedback to Sean Carroll over at Cosmic Variance to his unfair generalization about why Mormons aren’t respected within the academcy)

  19. Steve Evans says:

    John, Kaimi, fair enough. I guess I can understand some of the mistrust, but like Clark I do think we are too touchy about all this.

  20. Latter-day Guy says:

    Oh, yes, Clark. I agree that an exclusively, or primarily literary approach to the BOM would bother a lot of people. I think that it is something of a catch 22. If we don’t concern ourselves with defending historicity, then (for many) the value of the book is lowered. OTOH, if we do spend time defending historicity, we spend less time teaching the truths IN the book (as opposed to the truth OF the book).

    They aren’t necessarily mutually exclusive, but given that we can only spend a limited amount of time in study/discussion, we have to choose our focus, and I think the focus chosen in many cases is the less helpful of the two.

  21. Vesper Holly says:

    I was emailed the survey by a friend (who was emailed it by Ronan, to my suprise). I filled it out and my only question is: When and where can we see the results?

  22. Clark, if the counterfactual question you have in mind is the “even if I don’t know where the Book of Mormon came from” indicator mentioned above, this question isn’t counterfactual for people who indicate (on the other questions in the survey) that they don’t know where the Book of Mormon came from. I only intend to use this as a serious indicator, rather than as an object of study, for such folks.

    John F., I saw a lot of such theories in my email, as well. Such theories were balanced by equal and opposite theories among those who are disaffected — the survey will show that even really educated disaffected people don’t know anything about Mormonism, so why should they be taken seriously? In fact, I don’t intend to belittle anyone in this way. The education and reading variables are there to separate demographic groups for various purposes. The textual familiarity questions are there to see whether people who have spent more time with the text are distinctive in the structure of their attitudes toward it.

    I can understand some degree of suspicion on these points. Even so, I’ve never encountered an equivalent degree of concern, even though I’ve interviewed a lot of people in very difficult circumstances and about quite controversial matters.

    FHL and Vesper Holly, watch this space! I’ll be blogging about some of the fun, simple results in the weeks to come. The more in-depth analysis won’t be possible in a blog format, but I promise that BCC will always have links to everything that comes out of this project.

  23. Gee Ann (7) that seems a bit to harsh. Theirs no reason to get all mean about it.

  24. J. When do we get our grades?

  25. Left Field says:

    Most of the passages we were asked to identify were obscure enough that even most reasonably well-read believers might be unsure exactly where they came from. Perhaps that may have led some to think they were being set up. However, the exercise could be seen as more problematic for nonbelievers. For Mormons, the standard works are all equally authoritative, and it’s really not that big of a deal if we forget the reference. However, confusing a Book of Mormon passage with a biblical one would be more problematic for those who believe the Bible to be inerrant scripture, and the Book of Mormon to be of the Devil.

  26. Jami, you get an A+.

    Left Field, we’ll talk a bit more about the identification questions in a post some time next week.

  27. I don’t think anything in our history is as toxic to our image as our constant vigilance over our collective reputation, and our often misguided, sometimes embarrassingly uninformed defensiveness.

    Btw,I took the survey on BCC and also received it from a friend. So yes, it did get around.

  28. Left Field, they weren’t common passages but I’d hardly call any of them obscure. Now if you want obscure I could come up with a bunch… (Say compare and contrast Isaiah quotes and BoM paraphrases of Isaiah)

  29. Latter-day Guy says:

    Oh dear… just got an email from someone I sent the survey to. She said that she sent you a note, JNS. I suspect I already know what it says. [guilty shrug]

  30. Eric Russell says:

    Margaret, it seems to me that other groups have had some fairly positive effects as a result of a constant vigilance over their collective reputation. The NAACP, ADL and ACLU, for example, do a pretty good job of defending their respective groups.

    I wonder if the church could learn something from these folks – perhaps if the church upped its defensiveness and outrage, it would start to become less socially acceptable to speak so negatively about the church in public.

  31. FWIW, one commenter at MAD boards asked if it was a trick from Shades (a well-known internet church critic). See http://www.mormonapologetics.org/index.php?showtopic=36246 . (Others at MAD pointed out that J is a reputable blogger.)

  32. You’ll probably need to segregate your results from pre and post this followon discussion.

    i suspect part of the reason for the responses are
    1) people believe they are expert in Mormonism and therefore see themselves as peers with you on a survey on Mormonism.
    2) people are heavily invested in being sure that the church has a good name in public and worry about repercussions of getting things “wrong”

    I agree that the ahistoricity and faith question appears to be proselytizing one particular view just by token of being posed, though I’m sympathetic to the impulse behind the inquiry.

    and what a filthy survey. I mean really. and to suggest that’s it’s a hypothesis-generating convenience sample. how dare you?

  33. Left Field says:

    Clark, I didn’t really say the passages were obscure in an absolute sense, just that they were sufficiently obscure that a lot of LDS may not immediately know the origin of a lot of them. As for me, they all felt familiar, but there were some that I couldn’t quite place with certainty, though I could make a reasonable guess. It did seem like they were all passages that believably could have come from the Bible, Book of Mormon, or the Doctrine and Covenants. None of them had obvious clues as to their source.

    If one wanted, it would be easy enough to pick passages that nobody could identify, that everybody could identify, or that only Latter-day Saints could identify. I think it might be more of a challenge to find passages that would befuddle Mormons that did not equally befuddle Gentiles. For that reason (among others), it would seem unlikely that the questions were chosen specifically to trip up Mormons for the purpose of making them look bad.

    As I went through the questions, I had no sense that I was was being “set up” to expose my ignorance, but I knew enough about JNS to know that he wouldn’t be doing that. However, it did occur to me at the time that one could choose Book of Mormon passages designed to fool non-LDS Christians into thinking the verses were Biblical.

    Apparently, JNS will explain in a later post why he picked the particular passages he did and what information he intended to derive from it.

  34. smb, at this point the response rate is down to a bare trickle, mostly consisting of people who report that they never read Mormon-themed blogs. It’s easy to divide pre- and post-, and to test differences in patterns before and after this discussion. Yet there’s really little point; the sample size of 2200 at the time I posted this little essay is large enough for any reasonable purpose.

    I guess most of the questions in the survey proselytize for a given point of view, but the respondent is always free to press “strongly disagree.”

    Left Field, the survey really went to people with connections to the Mormon community, not to pure outsiders. The overwhelming majority of respondents are either current (about 85%) or former (about 8%) members of the LDS Church. So there’s no non-LDS Christian sample to compare with here.

    The idea was to pick passages that might plausibly be identifiable for people who spend a lot of time with the Book of Mormon text — but that aren’t all entirely obvious. (One was pretty nearly entirely obvious, though, based on the responses.) My first substantive post on this will discuss people’s success rates at identifying passages and speakers, and give a couple of interesting comparisons involving those data.

  35. JNS, I thought the passages you chose were reasonable – not obvious, perhaps, but not obscure at all.

    I always have a hard time with “strongly agree to strongly disagree” questions. They generally are worded such that I could answer either side of middle, based on how I choose to interpret the implications of the questions. This was no exception with some of the questions.

    Overall, however, I thought it was an excellent survey.

  36. StillConfused says:

    I am surprised that people complained about the questions in the survey. it is not like they were getting a grade or something. I took the survey (I love taking surveys) and it never occurred to me to complain about the questions.

  37. My comment about obscurity was supposed to be a joke. I guess I’m not nearly as funny as I want to be. (grin)

  38. Sterling says:

    JNS: I admired the survey. I felt like it was, in some small way, a fulfillment of what Pres. Benson envisioned when he talked about flooding the earth with the Book of Mormon. Keep up the great work.

  39. Brett Williams says:

    I filled ‘er out, J your questions were as good as any I’ve ever read from the scions of the COB.

    Mormons on the internet are unusually caustic. Far less than nerd forums, but still unnecessarily caustic. It’s an outgrowth of the rather nasty relations that Mormons have reciprocated throughout their short history. If anyone has read some of Joseph Fielding Smith’s anti-RLDS pamphlets you know he’d make a great blogger. Maybe we can hook Connor up…LOL

    Will you publish the results online? I’m looking forward to it.

  40. I enjoyed taking it, and never in a million years would I have thought to email you to complain.

    It came to me via John Dehlin, which is funny, because I’ve only met him once.

  41. Chelseabug says:

    I took the survey and do not see how anyone… especially (or even?) strong members of the church could be offended by it. Maybe people don’t understand the purpose of a survey. Or maybe they don’t realize that they have the free agency to um… not take it. Why do people purposefully put themselves in a situation to be offended?

  42. Chris P. says:

    I have come across too many people saying that questioning your beliefs, or doubting what a scripture says, or prophet said is bad, or a sin. There is no sin in wanting to know truth. Nephi prayed about his father, Lehi, the prophet’s dream about the tree of life. And if I remember correctly, he was not struck down for it. I think the big thing concerning questioning what is taugh is whether it is for your benifit in understanding the gospel, or for your benifit in getting away with more things.
    For example, questioning whether or not it is revelation given to the prophet that we should only have one pair of earrings, I think is a legitimate question. (I do not think it is a commandment). But when you question whether or not it is ok to drink beer because in the D&C it says Barley is ok, and you are looking for a way around the commandment then there is a problem.
    I am not saying that these questions shouldn’t be asked by any given person, but it is the attitude in which they are being asked. Someone who may be considering joining the church should ask if beer is ok or not.
    I also believe that different people will get different answers. My wife and I on our honeymoon went to vegas, and thought it would be fun to try our luck at the slot machines. We thought it would be fun, we weren’t trying to get rich. (Although it would have been nice, and I admittidly will say that I thought about it while pulling the lever). I did not feel bad about it, and she did not either, and I am sure that if we prayed about it we would have felt ok about it, becasue we both knew when to quit. The Lord knew our hearts and that is why I think he would not have cared. However, there are some people that this could be a huge problem. And so if they were to pray about it, I am sure that the Lord would give them a strict NO. He knows our hearts. And he knows how to treat each of his children.
    Questioning isn’t bad. It is the intent of the question that we need to watch.

  43. Jonathan Green says:

    I read through the survey and recorded my subjective reactions as I answered the questions, which I provide below with the hope that it will help you understand the reactions of survey participants. It’s a bit of a concern to me that you, as the survey author, are here casting yourself as an uninhibited thinker being victimized by Mormon close-mindedness (“risks of being a questioning Mormon”). If a survey goes badly, you might want to ask yourself if you might just have designed the survey poorly in some respects.

    —–

    Book of Mormon graphic on first page: not a familiar cover, and I’ve seen a lot of them. Immediately alienating. Is this an old edition? An RLDS edition?

    Some questions should use a scale of “frequently” to “never,” rather than “agree” to “disagree,” for example whether I try to read the Book of Mormon once a year, or whether I read the Book of Mormon when I feel depressed to feel better. I want to answer neutrally about my actual behavior, rather state a belief. There’s a frustrating confusion between the questions about my behavior, and a scale asking about belief, and I feel like I’m disagreeing with the idea that I should read the Book of Mormon yearly. Also applies to question 16 (opening up the Book of Mormon at random for guidance).

    Question 14: “Some people say… Joseph Smith… became a fallen prophet…. I think these people might have a point.”
    Confusion! There are too many subject positions in this sentence. I have to say whether or not I agree or disagree with my own thinking about other people thinking about Joseph Smith. Disagreeing with this question means disagreeing with myself (“I think these people might have a point”).

    Huge problem in 24: “Even if I feel uncertain about where the Book of Mormon comes from, I feel that the Church of Jesus Christ of Latter-day Saints is God’s church.” Am I agreeing that I’m uncertain about the Book of Mormon? I understand what the question is asking, but answering the question requires me to adopt uncertainty about the Book of Mormon.
    [I see that you discussed this question in your post. Whether or not the question does what you want it to do, it still affects how survey respondents feel about the rest of the survey, which will color all their answers.]

    How often I read from the Book of Mormon, question 25. I just finished reading it by reading daily, but since then I haven’t read in it at all. How do I answer?

    Question 27, concerning the percentage of quotation from the KJV. I don’t know what standard of textual citation you’re using. Do verses have to be word-for-word identical? Mostly the same? I’ve never seen this statistic in any church source, so I’m wondering to whom you’ll be turning to for an answer.

    Question 28, about the percentage of biological descendants of Lehi, is phrased as a factual question, as if I should know the right answer, but it’s purely a question about my beliefs, or guesses, or assumptions.

    Question 29, the first scripture ID question. I’m really concerned why you want to know. What does scripture memorization have to do with anything? The scripture in question talks about “not laying again the foundation,” which sounds like something an anti-Mormon ministry might quote about why there was no need for a Restoration. If you’re only interested in testing familiarity with exact wording of various scriptures, this verse seems highly problematic. If it turns out to be from the Book of Mormon, only more so: “the Mormons don’t even know their own book!!”.

    Question 31. That’s three questions in a row with a very theological or proof-text feel, rather than stories. Why do you keep asking me to distinguish works of scripture by their theology, rather than by who is doing what? Most of us, I think, remember what people do in the various works, and not isolated paragraphs of theological statement.

    Question 32 includes the word “shewed,” which seems too archaic for the Book of Mormon, so I say KJV/NT.

    Question 34, the first “who said this in the Book of Mormon” question: No anxiety, but I’m still mystified why you’d want to know. This is a type of work with the text that we never do, so I’ll probably stink at it. What will it end up proving?

    Question 36 is clear enough. Anyone who has been paying attention will know. Question 37 is a reasonably good question about whether I’ve heard the first thing about scholarly Mormon Studies. Later questions about educational attainment are all pretty standard survey fare, so no problems.

    Question 40, “Have you ever been a member,” raises big red flags. The use of the perfect verbal mood suggests that church membership normally has a beginning and end, which is not how Mormons typically consider their membership. The option about “another church in the LDS movement” suggests an equivalence that is similarly controversial. “Movement” seems like a particularly unwise word choice, as it suggests a coherency that is vehemently disputed by LDS church leaders. This question should have been worded more carefully and neutrally.

  44. JG (#43) wrote

    Huge problem in 24: “Even if I feel uncertain about where the Book of Mormon comes from, I feel that the Church of Jesus Christ of Latter-day Saints is God’s church.” Am I agreeing that I’m uncertain about the Book of Mormon? I understand what the question is asking, but answering the question requires me to adopt uncertainty about the Book of Mormon.

    This question, more than any other, is what made orthodox Latter-day Saints suspicious of what the point of the survey was. If they agreed, then they have to adopt uncertainty about the Book of Mormon, which many orthodox Latter-day Saints feel that they do not have, even if empirically and if in an environment of mutual support and understanding they might admit that such uncertainty exists based on the current state of what we understand of the Meso-American archeological record (although the same cannot be said, objectively speaking, of the Old World archeological record). If they disagreed because of not wanting to admit uncertainty about Book of Mormon origins, then it would seem like they are saying they don’t believe the Church is true. So they felt forced to press “agree” and thereby appear as though they are conceding uncertainty about BoM origins. This naturally lead them to wonder why a survey would force them to admit uncertainty about BoM origins and they naturally questioned the agenda of the survey. Paranoid? Sure, but perhaps not without plenty of background and reason to be so.

    I thought the conditional aspect of the question alleviated the problem that others described in it but even in discussions with people about it, they didn’t see it my way. (I made a point that you weren’t trying anything subversive with your survey and that you were a personal friend of mine.)

    But I can see why people viewed it as the “Have you stopped beating your wife?” question.

    JG wrote

    Question 29, the first scripture ID question. I’m really concerned why you want to know. What does scripture memorization have to do with anything? The scripture in question talks about “not laying again the foundation,” which sounds like something an anti-Mormon ministry might quote about why there was no need for a Restoration. If you’re only interested in testing familiarity with exact wording of various scriptures, this verse seems highly problematic. If it turns out to be from the Book of Mormon, only more so: “the Mormons don’t even know their own book!!”.

    Although I did not speak with Jonathan about this survey and he recorded these impressions at the time he took the survey, presumably before I made my comment # 12 on this thread, it appears that he also had a similar reaction to this scriptural passage and the possible motivations for including it.

    Another sticking point was the question relating to the percentage of Native Americans whose ancestor was Lehi. Most educated Mormons are highly aware of the DNA issues and how anti-Mormons and others take the DNA arguments as absolute proof that the BoM is fraudulent and the beliefs of Mormons are ridiculous. Seeing the question on the survey caused people to try to figure out who was giving this survey and why such a question would be posed. How should they answer? Since they probably haven’t thought about such percentages before, they simply guess.

  45. Ross Eldridge says:

    Another great discussion. Seems to me that the survey, in places, should have had a choice that read “I don’t know”. How many people got confused, felt some guilt about the confusion, and dawdled and guessed at Lehi’s descendents today? Better give the biggest number, or I’m not true-blue LDS. Surely that will skew the results, people giving the responses they think they should, rather than what they honestly know or believe.

    I was interested in the question about Joseph translating the Book of Mormon honestly, but then going awry and becoming a fallen prophet … This is my opinion, but leaving the Book of Mormon out of things completely, the Joseph Smith Canon is more interesting, more compelling, and did more to convert me that the (rather tedious) Book of Mormon.

    The King Follett Discourse is brilliant. It is, I believe, as read today, something of an amalgam of what listeners recalled Joseph saying. But, even if they had guessed a bit they did use Joseph’s teachings and intent. However, I suppose the missionaries (assuming they’ve even heard of the King Follett Discourse) would be hard-pressed to teach it to young people in Africa and Asia. Easier to preach something that is just a shade off a famous text like the Bible.

    The missionary discussions, in 1973 at least, had questions asked of the investigator that resulted in a lot of “Yes! You are quite right!” replies from the missionaries. The Book of Mormon was made to appear as needed and correct, based on a few ideas and quotations, before you had read it. So, even today, you feel you NEED to give the right answers, or appear a fool.

    Out of this complicated mish-mash, I feel it IS possible to question the Book of Mormon, but still support the Church on many (perhaps most) issues.

    Has anyone suggested that the Book of Mormon is a “story book” … With so many examples of things that might happen and what they might lead to …

    Who amongst you “believes” the story of Jonah and the Whale?

  46. Jonathan, thanks for your reactions. I don’t see anything in your account that raises issues for the survey, although I will confess that I often have similar feelings of uncertainty and ambiguity when answering a survey. But on most of these points, you seem to be answering using the thoughts that I would have hoped you would have in mind.

    Some of your concerns are of the kind that are interesting to me — the kind that I never anticipated and that in my experience don’t usually come up in talking about surveys with other kinds of target populations. This survey did go to a lot of people who had once been Mormon and was also intended to be answerable for Community of Christ members and members of other churches that use the Book of Mormon. Is it really so controversial to us to pose a question in a way that allows such people to be included? If so, what does that say about us?

    Your point on Question 27 assumes that there’s a true-false answer here. That’s not necessarily how the item will be used. Even if it were, though, is it really a problem simply because it’s a hard question?

    I agree that Question 24 didn’t necessarily work well, at least for respondents in general. For the intended subset of respondents, it seems to be quite reliable and to have predictive power. Still, I appreciate people’s expressions of concern about this question.

    But the concerns over the passage ID questions are of the kind that has me flabbergasted. I do admire the creativity involved in finding patterns to worry about, though. The passages weren’t chosen because of theological content. Rather, they were chosen because they were mostly hard but potentially identifiable. If I chose easy passages, such as narrative accounts, everyone would get them right and they’d contain no statistical information.

    Finally, your opening paragraph rather missed the point. The survey absolutely didn’t go badly; it’s worked quite well, and initial analysis suggests that nearly all the items are empirically useful. Furthermore, I’m not “casting [myself] as an uninhibited thinker being victimized by Mormon close-mindedness.” This is taking the post title far too seriously. The title is a joke, playing off the fact that a “questioning Mormon” is a good way of describing me as a Mormon asking people survey questions. And let me reiterate that the aggressive emails I received came just as often from disaffected or former Mormons as from active believers.

  47. On the descendants-of-Lehi question, the distinction of primary analytic interest is between respondents who said 0% and respondents who gave some other answer. The full range of responses is available to allow some comparison between those with higher belief ranges and those with lower belief ranges for secondary purposes. But there’s no particular reason that the precise details of people’s answers will be either important or problematic. As long as people managed to place themselves in a percentile range that feels approximately comfortable to them, all is well.

  48. Left Field says:

    JNS, that’s really exactly what I thought about the Lehi question. It seems good for determining if the person thinks Lehi existed, but doesn’t really say anything about where they come down on the “hemispheric, principal ancestors” model vs. the “local geography” model. Supporters of both models would likely be included in the higher percentage categories. Since the limited geography model is a commonly raised issue, it’s not unreasonable for the respondent to think you might be trying to address that. However, I’m not really sure what you mean by “higher belief ranges.”

    It does seem like a lot of the questions (including my own) and discomfort with the survey are the result of a concern that the surveyor may interpret the questions and hence the answers in a way that was not intended by the respondent. I do know that sometimes survey questions are deliberately designed to get information that is not necessarily obvious to the respondent. Can it become a problem if some respondents try to outsmart the questions by answering what they think was intended, rather than what was actually asked?

  49. “Educated” respondents of this survey will definitely have done some thinking on the descendants of Lehi question, likely believing that it was a trap relating to hemispheric model vs. limited geography model, although some might have had an additional layer of confusion if they have come away from reading the apologetic/polemic back and forth on the DNA theories with the idea that if Lehi really existed at all and had children then everyone alive would be a descendant of him today, statistically speaking.

  50. Left Field, it can indeed be a problem if people try to outsmart the questions. To the extent that different people try to outsmart them in completely contrasting ways — which absolutely seems to be the case here, to the extent that people are doing this — they should cancel each other out, leaving more meaningful information behind. But if large numbers of people game the survey in the same direction, there can be trouble. A famous example is that, in surveys, people dramatically over-report having an active library card. I think it tells you a lot about people if you think about why this might be the case…

  51. Interestingly, a solid respondents chose values toward the middle of the range on the descendants-of-Lehi question. So the math model suggesting that everyone is a descendant of every single person from a few thousand years ago or more that has any descendant at all doesn’t seem to have played too much of a role for most respondents.

    The descendants question, of course, doesn’t tap the issue of different Book of Mormon geographies. It might be useful to develop a battery of questions to address that set of beliefs, but that task struck me as a second-order one in comparison with understanding what appear to me as more personal aspects of people’s Book of Mormon experience.

  52. Matt W. says:

    I liked the survey, I had an issue with the question that was addressed so I answered in the middle, (I think) because It was confusing, and that is my survey response to confusing questions. (I meantioned this over at Clark’s Blog.) Outside of that, I thought it was a fun survey that will bring fun results.

    I am interested to see the results, and hope you will make the raw data and some clustering analysis available.

    I got the survey from multiple sources, including my wife’s MTC and one of the most psychotically orthodox women I’ve ever met. My wife forwarded it on to everyone she knows (I don’t forward things) so I’d be interest in what your response rate is.

    I can’t recall, but I wish there was a way to break down the responses by geographic location.

  53. I should say that I also liked the survey and had fun with the scripture passage ID and character ID questions! It was like quiz night at the local pub, only Mormon-themed.

  54. Mormon night at the local pub

    I wonder how long that Marketing Manager kept his job?

  55. JNS,

    You probably have a pretty good idea on my views after all these years in the bloggernaccle. You can also ask your SP about me. Use my last name.

    I did not find your survey offensive in the least. Some people simply like to take offense these are also the same people that will summon enough energy to send you a nasty email about the survey.

    The only reason I can think of that would make an Orthodox reader suspect is the “Academic tone” of the survey. If you are unfamiliar with this manner of questioning you may find it a bit off-putting regarding sacred beliefs.

  56. Jonathan Green says:

    I’m glad to hear the title was not intended 100% seriously. I hope you understood my comments as my immediate reactions, rather than as carefully considered responses.

    You might want to think about some of the assumptions in the scripture ID questions. Outside of weekly church attenders, lots of people have little idea what’s actually in the Bible, such as if Jonah is in the OT or NT. Including the names “Shadrach” or the story of Pentecost wouldn’t be a giveaway for most people. The level of scriptural expertise you’re trying to distinguish begins at a very high baseline.

  57. Jonathan, let me reiterate that a lot of respondents got the scripture ID questions as posed correct. So they worked fine. Also, I think it might be more accurate to say that the title was intended 100% not seriously.

    bbell, thanks for the comment. I agree that there are lots of people who just like to take offense, and I’m very glad to talk with those friendly folks like you who aren’t in that category.

    Matt W., I do have some information on geography based on one question about continent of residence and the probable location of people’s IP addresses. It’s something that I’ll look at.

    I’m not sure that Chris P. read the post or the comments.

  58. John Hamer says:

    J: When I reviewed the survey, I assumed your motive was to gather data. The survey left me feeling curious about the results. Thanks for undertaking it.

    I think a lot of the folks who wrote you and complained that you were attempting promote an agenda may have been telling you how they themselves operate: find conclusion first, then produce data that appears to justify that conclusion. Just because they do that, doesn’t mean that you’re doing that. But it’s natural for people to assume that everyone else acts as they themselves act.

  59. JH, just because they are suspicious of surveys dealing with their religion doesn’t mean they’re doing what you said. If they have a persecution complex, perhaps it’s because they’ve been persecuted? No, probably not.

  60. That said, I should note that I fully agree with others on this thread that we need to be a lot less touchy and tune our anti-Mormon detectors/meters way down.

    Some of my comments have been an attempt to elicit some understanding for some of the reactions from orthodox members that the survey provoked. That doesn’t mean I think that people who responded that way behaved in the ideal manner. Far from it — I am sorry that JNS apparently got some angry emails.

  61. John Hamer says:

    John F (#59): That’s what I get for responding w/o reading all the comments. I meant to refer to J’s intial post that he was attacked by extremists on both sides: critics who felt he must be shilling for the LDS Church, and apologists who felt he must be planning to attack the LDS Church.

    He separated those complaints out from those who have more technical concerns about the questions and the potential for problematic results given some of the phrasing.

  62. Could the difficulties with question #24 just be a simple failure of imagination?

    They all seemed clear to me. Maybe there should have been a time limit so that people didn’t over-think it.

    On the other hand, over-thinking issues does not appear to be much of a problem for a lot of folks.

  63. Jay in Phoenix says:

    J: I took the survey a few days ago after a friend forwarded it to me, and then he forwarded me a link to this post. Previous to this I have never seen or heard of BCC before. Just some background to give my comments context.

    It seems you had people who brought up technical issues and yet others who brought up personal issues. I would like to comment on both.

    As for the technical problems with the survey, I suggest people get over it. Surveys have inherent limitations and this one is no different than most in that regard, but there was nothing GROSSLY problematic if you ask me. Take a class in survey design, and you will find like all people who do surveys that it is as much an art as a science. And the interpretation of the results are also equal parts art and science. This is always the case when we are dealing with subjective responses and questions that speak to deeply held views and people’s sense of identity. In other words, if you don’t like this survey, you shouldn’t like ANY survey, because most critiques of this one would apply equally well to virtually all surveys. That’s a generalization to be sure, but I think it’s a pretty fair one.

    As for the more emotional/personal responses, what surprises me is that you were so surprised by them. My experience has been that if a survey or study or article or commentary is biased pro-Mormon, then for the most part we Saints are OK with it, but if it neutral or anti-Mormon, we lash out. (I am using “we” loosely here. I know we do not ALL lash out, of course.) And we should not be surprised by the extreme nature of some of the responses. Those who are the most emotionally invested in this topic (for whatever reason) are the ones who are most likely to take the extra step to respond directly. It was only 5% who bothered to do so, after all. Maybe it will lift our spirits a bit to recognize that the vast majority of participants – 95% according to your post – didn’t care enough to do anything more than just participate. We can’t let the outliers have undue influence on our interpretations. BEcause I suspect that we are not as collectively nutty and rude as this may suggest.

    Just my take on it, anyways.

  64. Was anybody else tempted to look up the speaker ID quotes before answering them? I thought about it, but decided that since it was anonymous, I didn’t really have any motivation to look better than I am.

  65. Steve Jones says:

    I do not think the passages were obscure or uncommon, just the opposite. Some could have come from several sources depending of the exact verbage and with out looking at the source I was left to make an educated guess of which was the precise passage. It reminds me of my mission. We not only were required to memorized numerous scriptures but to know numerous chapters on specific topics and I ended up cutting up the scripture chapters verse by verse and then testing myself on whether it was a specific chapter on the atonement for example

  66. #57: Last sentence – You think? That one jarred as I was reading through the comments, but you beat me to the punch.

    Having said that, I think that one is indicative of quite a bit of what I see on other sites. Not so much on this one, but even here it happens frequently enough to not surprise me anymore. It might be directly correlated to the percent of those who will complain about the survey – those who go into it with a preconceived idea of what is going to be said and don’t take the time to see what really is being said/asked.

  67. I thought it was grossly unfair that you gave us four potential scripture sources and four questions — I felt compelled to shoehorn the four into producing four different answers. I was also more than a little sulky when the quiz failed to tell me whether I got the right answers or not (and the fact of the matter is that I will wonder about how good my spelling was for the rest of my life!)

    But other than that, it was fun. And I totally forgot about the whole “ancestor of everyone” thing and instead focused on Eskimos walking across a land bridge ten thousand years ago, from my 5th grade science lessons about plate tectonics.

    I feel bad for having failed to experience the appropriately vitriolic reaction to your survey, basically. You’re so fired, Bro. Nelson-Seawright. =P

  68. JNS,

    I enjoyed the survey (as I do almost all surveys) and it made me wish I was trained to conduct my own surveys and analyze their results. I’m looking forward to your analysis of the results.

    The hardest question for me was the Lehi descendents question. I wanted to answer 100% due to the reasoning you mentioned in #51, but feared that would be interpreted as a naive assumption that all Native Americans are “Nephites.”

  69. By the way, I got a good laugh out of the “Did you have your professor look over the survey before putting it online” comment. Hilarious.

  70. Oh, wait, I guess traditionally the Native Americans were thought to be Lamanites rather than Nephites. My bad.

  71. JNS,

    Thanks for doing the survey. I, like everybody else, look forward to the results as well as analysis. I think that I could tell generally why many questions were asked, but I think that I would enjoy a post on how you designed the survey and what you were targeting with the questions.

    Thanks again for your time and effort. I think it is definitely easier to pick at someone else’s design rather than to conceive your own.

  72. wry catcher says:

    Hi JNS/RT,

    I took your survey from the link posted on FLAK. I also saw all the semi-cranky low-grade complaints about the survey as well (sounds like they got worse in email). To be honest, I didn’t think much of the complaining because it is my experience as a sociologist and longtime researcher, people *always* complain about the way questions are worded, question your objectives (read: motives) and your sanity, and perhaps your parents’ child-rearing skills. To add religion into the mix, particularly something as touchy as BoM beliefs (and knowledge), well, lighted matches and piles of oily rags come to mind.

    FWIW, I thought the survey was quite well done, thoughtful and well-written and clearly meant to be a very broad first test of some of the approaches and questions. There is nothing like a survey to get people’s suspicions raised, though. I recently got three pages of single-spaced, extremely detailed feedback on a 30-question canned proprietary survey (which was noted when it was sent out — we had no power to change the questions, they are an existing battery we bought in order to use the large database of norms they had at the country level) regarding people’s management style. This particular respondent was JUST SURE we were tracing their responses and comparing them to existing feedback mechanisms so that we could build a case against them…and this was the 538 reasons why the survey was not valid as a means to doing so.

    Anyhow, I ramble. Good survey, nice first shot out of the gate, and I hope you’ll share all the findings.

  73. I think its worth noting here at the Correlation Committee and CES sent out surveys to hundreds of Seminary and Institute teachers last fall. It was extremely detailed and focused primarily on what the teachers thought of the CES “teaching emphasis.” The problematic part of the survey being that my wife, who has taught Institute for 4-5 years now, had no idea what the teaching emphasis was. So strike one for survey design.

  74. Little Sister says:

    I believe they’ve done studies on standardized tests showing that women are significantly less likely than men to guess on a question if there is a penalty for wrong answers. As I was taking the survey, I felt there would be a very large penalty (making my Church membership look stupid) for wrong answers so I put “I don’t know” for most of them, even if I had a pretty good idea. (Sorry, I did feel like I was being set up, even after being linked there from BCC). That could throw off your results some. I saw in the comments that “sister blah 2″ did that as well.

    Very good and interesting survey though! I am looking forward to the results.

  75. sister blah 2 says:

    ’tis true, I put “I don’t know” for almost everything, even when I had a pretty good idea.

  76. Good luck with survey. I asked my Gospel Doctrine class to close their eyes before I took a poll. With their eyes closed, I asked them how many believe the Book of Mormon to be a word for word translation. Out of 52 in the class 43 believe it is word for word. After class, the only comment I heard was “I can’t believe 9 people don’t believe it is a perfect translation.”

Follow

Get every new post delivered to your Inbox.

Join 9,516 other followers