Alex, thanks for replying to our report. You had some good points. However, this comment reflects what I think is a misunderstanding of the Pew report:
"Mathematically for the Protestant average to be 15 points higher, several individual denominations must have significantly higher retention than LDS. Grouping them by fragments makes a misleading comparison.”
If you go down two pages in the report you will see a table with all of the retention rates for the individual Protestant denominations as well as the average rate of 44%. It seems you assumed the 70% was an average of the individuals rates, but it is not. Rather, the 70% seems to count switching from one Protestant denomination to another as retention within Protestantism, if that makes sense.
Hi Sam, I appreciate you responding. I did have a misunderstanding of the data in this particular point. Most people stay within the protestant tradition (70%), but the number for switching from one protestant denomination to another is different with an average rate of 44%.
The question then becomes is it better to compare protestantism as a whole or by individual denomination? I think its fair to look at individual protestant denominations, but acknowledging the whole protestant number would be important context and may help in future readers in accurately reconciling the pew chart and the charts in this report. Thanks for clarifying this and your work in this project!
The 44% is just anyone in any of those specific denominations leaving for anything else, another Protestant one or otherwise. And yes it's debatable which comparisons is more appropriate or meaningful, with each providing a different angle. We are updating the report based on many of your comments, so appreciate another set of eyes.
This brings up some really interesting questions about the right approach to analyzing protestant data in general as Protestant isn’t really a religion at all but a vague grouping of religions.
I think you are being over-critical. (I read the article but not the report itself.) The article does address the decline in retention, mentioning the headwinds. It’s balanced and not all roses. Their attendance-weighted retention metric may not be of interest to all faiths, but it does provide a window into the data that is relevant from a certain perspective, and says something useful about the data, so I think it’s fair. They were clear about what they were doing and didn’t just try to sneak in a new definition of “retention”, for example, which would have been unfair. On the clustering with a small sample size - at least they do mention the sample size rather than bury it, and I’m glad they went ahead despite a small sample, as the results do shed important light on an interesting phenomenon. (Same story as your analysis.) I wasn’t bothered by the concerns you mentioned.
Great report on this analysis, and thank you. I read your work often, and I am certain many others do for additional interpretation. What may further add to your reports is short commentary on the possible theoretical nexus as to the why, and the what that data shows in relation to social trends. You touched on it briefly, I believe, in mention of women factors and retention changes. Having not read the original article myself, I am unsure if the authors included a theoretical framework within their methodology. Writing towards a possible “why,” even with low sample sizes, offers us as readers greater understanding and helps connect the dots.
I appreciate this feedback, Keith. I think you are right with this public facing work it is more memorable and impactful in the context of a story (explaining the "whys"). I think I can be hesitant to elaborate too much as to not lead people astray mistaking a theory as definitive fact! But, I think you are right that I could be better about giving more theoretical reasons with hedging where necessary.
Alex, thanks for replying to our report. You had some good points. However, this comment reflects what I think is a misunderstanding of the Pew report:
"Mathematically for the Protestant average to be 15 points higher, several individual denominations must have significantly higher retention than LDS. Grouping them by fragments makes a misleading comparison.”
If you go down two pages in the report you will see a table with all of the retention rates for the individual Protestant denominations as well as the average rate of 44%. It seems you assumed the 70% was an average of the individuals rates, but it is not. Rather, the 70% seems to count switching from one Protestant denomination to another as retention within Protestantism, if that makes sense.
Hi Sam, I appreciate you responding. I did have a misunderstanding of the data in this particular point. Most people stay within the protestant tradition (70%), but the number for switching from one protestant denomination to another is different with an average rate of 44%.
The question then becomes is it better to compare protestantism as a whole or by individual denomination? I think its fair to look at individual protestant denominations, but acknowledging the whole protestant number would be important context and may help in future readers in accurately reconciling the pew chart and the charts in this report. Thanks for clarifying this and your work in this project!
The 44% is just anyone in any of those specific denominations leaving for anything else, another Protestant one or otherwise. And yes it's debatable which comparisons is more appropriate or meaningful, with each providing a different angle. We are updating the report based on many of your comments, so appreciate another set of eyes.
This brings up some really interesting questions about the right approach to analyzing protestant data in general as Protestant isn’t really a religion at all but a vague grouping of religions.
I think you are being over-critical. (I read the article but not the report itself.) The article does address the decline in retention, mentioning the headwinds. It’s balanced and not all roses. Their attendance-weighted retention metric may not be of interest to all faiths, but it does provide a window into the data that is relevant from a certain perspective, and says something useful about the data, so I think it’s fair. They were clear about what they were doing and didn’t just try to sneak in a new definition of “retention”, for example, which would have been unfair. On the clustering with a small sample size - at least they do mention the sample size rather than bury it, and I’m glad they went ahead despite a small sample, as the results do shed important light on an interesting phenomenon. (Same story as your analysis.) I wasn’t bothered by the concerns you mentioned.
Thanks, Stephen, appreciate your take.
Thanks for this. Some great research here.
Great report on this analysis, and thank you. I read your work often, and I am certain many others do for additional interpretation. What may further add to your reports is short commentary on the possible theoretical nexus as to the why, and the what that data shows in relation to social trends. You touched on it briefly, I believe, in mention of women factors and retention changes. Having not read the original article myself, I am unsure if the authors included a theoretical framework within their methodology. Writing towards a possible “why,” even with low sample sizes, offers us as readers greater understanding and helps connect the dots.
I appreciate this feedback, Keith. I think you are right with this public facing work it is more memorable and impactful in the context of a story (explaining the "whys"). I think I can be hesitant to elaborate too much as to not lead people astray mistaking a theory as definitive fact! But, I think you are right that I could be better about giving more theoretical reasons with hedging where necessary.
I always appreciate and enjoy your posts, Alex! Thank you for sharing.
My thought? A religion that has to do a study to prove it’s not as bad as other religions in maintaining its membership, is worried.