There has been a lot of recent interest in the media about a study which claimed to show that modern religious societies were more socially dysfunctional than non-religious ones. (See e.g. George Monbiot). The full text of the study (Gregory S. Paul, Cross-national correlations of quantifiable societal health with popular religiosity and secularism in the prosperous democracies, Journal of Religion and Society 7 (2005)) is available on the web (see http://moses.creighton.edu/JRS/2005/2005-11.html). I thought Id had a look.
Bloggers have already been suggesting its a sloppy/flawed study (see e.g. http://www.tpmcafe.com/story/2005/10/4/17430/4632). Id go further. Its a study with a lot of flaws, most of which mysteriously seem to help prop up Pauls argument: in other words, its very biased. Here is a selection of the problems, as an example of how not to do research. Pauls hypothesis (BTW) as set out in s9 is to investigate whether faith in a creator or disbelief in evolution improves or degrades societal conditions.
First of all, theres the problem with the countries he chooses to look at. For example, he includes Japan, but its not at all clear that any of the measures of religiosity he uses are suitable for countries that have traditionally followed a non-Christian religion. One of them, take the Bible literally, is clearly not. Why does he take England to mean Great Britain excluding Northern Ireland? Also, he includes one second world European democracy: Portugal. He doesnt explain why Portugal is second world, (in contrast to Spain, for example), or even whether he thinks or he has evidence to believe its typical of second world countries. (The suspicion that Portugal has been included because it supports his thesis is high, since he doesnt actually include data from it in most of the graphs, as far as I can make out).
Secondly, theres the question of the measures he chooses to measure social dysfunction. These are: homicides (fig 2), youth suicide rates (fig 3), under-5 mortality (fig 4), life expectancy (fig 5), teen and adult gonorrhoea rates (fig 6), teen and adult syphilis rates (fig 7), teen abortions (fig 8) and teen pregnancies/births (fig 9). He doesnt give any indications of why these particular measures have been chosen, which makes one suspicious that the choice was intended to provide him with good statistics. In particular, I dont see the logic of having 2 indicators of STDs, while not including any on divorce rates or illegitimacy or children being brought up in one-parent families. Similarly, while the use of life expectancy as a measure might be justified by claims that being religious makes you live longer, why include child mortality statistics? That surely measures the effectiveness of health systems far more than social dysfunction. Is it widely claimed by religious leaders (even US ones) that if a nation becomes more Christian fewer children will die?
Thirdly, theres the fact that he doesnt actually do any regression analysis, or even provide the statistics so someone interested could do them themselves. (I suppose you could use non-parametric ranking if you were really keen). So his ideas of correlation often rely on the eye of faith, as it were. I would say myself that some of the figures in which he sees correlation look very weakly correlated at best: for example fig 2 (on homicide) (where the correlation is improved considerably by the inclusion of Portugal) and figs 6-7 on STDs.
Finally, some of the discussions in the article suggest a willingness to blur facts to suit his views. For example, at s15 he says: A few hundred years ago rates of homicide were astronomical in Christian Europe and the American colonies. In all secular developed democracies a centuries long-term trend has seen homicide rates drop to historical lows. I dont know the historical trends for other countries but that is very misleading as a description of the UK. One recent study of secularisation (Callum Brown, The Death of Christian Britain) argues convincingly that secularisation in Britain wasnt a long process, but actually happened relatively rapidly (in the 1960s). If you look at UK historical statistics on homicide rates, (see http://www.parliament.uk/commons/lib/research/rp99/rp99-111.pdf) they declined from 1990-1960 and have since then been largely climbing (and are now higher than in 1900). In other words, that data suggests a substantial fall while Britain was still a Christian country, followed by a substantial rise since.
Paul does manage to show that compared to other developed countries the US is unusually socially dysfunctional and also that the US is unusually religious. He also confirms that secularisation in Europe does not necessarily lead to social collapse. But his attempts to show any greater correlation between religion and social dysfunction are deeply flawed, and that isnt mitigated by claims that its a just an initial study. He could have done a good initial study, which raised questions; he chose instead to botch together statistics to support his views. That doesnt help anyone, except those who are opposed to high academic standards.