Rules of Comparative Research

Rules of Comparative Research

Comparative research is a research methodology in the social sciences that aims to make comparisons across different countries or cultures. The term comparative method refers to a specific kind of comparison – the comparison of large macro-social units. It can be seen as a way of bridging the qualitative and quantitative research traditions. I will in this article focus on comparative survey research.

There are many names for the type of research that we do: comparative research, comparative public opinion, cross-national public opinion, or even political behavior.

Roger Jowell (1998) in his article “How Comparative Is Comparative Research?” provides us with some rules of comparative research.

1)      Knowledge about country: Social scientists should undertake not to interpret survey data relating to a country about which they know little or nothing. This would tend to ensure cross-national collaboration in the interpretation as well as the design of comparative research.

2)      Limit the number of countries: Resist the temptation to compare too many countries at once. This is to avoid marginal countries being the primary focus (for example, choose OECD countries from the full range of WVS-countries). Emerging naturally from the six previous rules, cross-national surveys should ideally be confined to the smallest number of countries consistent with their aims, rather than celebrating as many nations as possible in their purview.

3)      Contextual variables matter as well: Cross-national surveys should pay as much attention to the choice and compilation of aggregate-level contextual variables, as they do to individual-level dependent and independent variables (relevant level-2 variables).

4)      Aware of limitations: Social scientists contemplating or engaged in cross-national studies should be as open about their limitations as they are enthusiastic about their explanatory powers. The fact is that only certain subjects, and only certain aspects of those subjects, can successfully be measured cross-nationally.

5)      Rules for methods: Stringent and well-policed ground rules for comparable survey methods should become much more common in comparative studies than they are now. To avoid infringing well-established cultural norms in one country or another, substantial national variations in methods are sometimes tolerated that should render comparisons invalid. To transform cross-national surveys from parallel exercises into joint ones, collective development work, experimentation, scale construction, and piloting should be undertaken in all participating nations. Routinely provide for secondary data analysts’ detailed methodological reports about each participating nation’s procedures, methods, and success rates, highlighting rather than suppressing variations. One should routinely include methodological experiments in cross-national research.

6)      Be critical of findings: Analysts of cross-national data should try to suspend initial belief in any major inter-country differences they discover. All too often, such unexpected differences turn out to be impostors – the result of a poor translation, a subtly different show card, a variation in sampling coverage, or a particular cultural cue that subtly alters the meaning of the variable in that country.

If these rules were even roughly adhered to, the situation would improve considerably. Indeed, any comparative data set that complied with these rules would immediately transform itself from being deeply suspect to just plainly problematical.