What is a General Population Experiment?

General population experiments allow investigators to assign representative subject populations to experimental conditions of their choosing. They use CATI and Internet technology to confront randomly selected respondents with experimental stimuli of some kind. These stimuli can take any form; thus far, general population experiments have used systematic variation in the information that is given to respondents, alterations in question wording, pictorial stimuli, and differing incentives and instructions, just to name a few possibilities. As TESS is dedicated to promoting innovative experiments, all proposals should include experimental or quasi-experimental design consistent with the definitions used by Campbell and Stanley in Experimental and Quasi-Experimental Designs for Research or Cook and Campbell in Quasi-Experimentation: Design and Analysis Issues for Field Settings.

For TESS proposals that have used this procedure, please click here. For other publications that include general population experiments, please click here

What are the advantages of General Population Experiments?

Some social scientists conduct experiments. Others do not. While most social scientists recognize the tremendous benefits of experimentation, the traditional laboratory context is not suitable for all important research questions (see, e.g., the range of methods described in Kagel and Roth 1995). For some researchers, survey methods provide an alternative means of data collection. There are, however, often substantial obstacles to drawing strong causal inferences from conventional survey data. Over the years, many have hoped that advances in statistical methods would help scholars use such survey data to "partial out" or control for all plausible rival interpretations of a potentially causal relationship (see Cook & Campbell 1979). Despite massive advances in statistical methods over the years, few people are as optimistic today that statistics can solve all such causal inference problems.

Of course, we can never be certain of what causes what. As King et al. (1994: 79) explain, "no matter how perfect the research design, no matter how much data we collect, no matter how perceptive the observers, no matter how diligent the research assistants, and no matter how much experimental control we have, we will never know a causal inference for certain." This impossibility stems from our inability to observe a counterfactual and is known as the fundamental problem of causal inference (Holland 1986; cf. Cook & Campbell 1979).

Throughout the natural and social sciences, researchers employ experimental designs in order to combat the challenges posed by the fundamental problem of causal inference. As interest in experimentation has grown, the kinds of experiments conducted have multiplied. While there is disagreement across fields about what constitutes an experiment (e.g., Friedman & Sunder 1994), there is also substantial common ground.

Across the social sciences, for example, the modal experimental design strategy entails creating a situation in which causal hypotheses are pitted against one another. A critical experiment is one in which certain experimental observations render impossible a causal hypothesis that was viewed as plausible before the experiment. Such experimental designs contain at least one of the two most commonly discussed adaptations to problems of causal inference. Researchers either evaluate a unit of analysis before and after a given intervention and draw inferences from these within-subject comparisons or they use a between-group design in which different subjects are randomly assigned to groups that receive different experimental treatments (Holland 1986: 947).

What is notable is that either of these basic techniques for strengthening causal inference, and even highly complex experimental designs, is easily implemented in the context of computer-assisted telephone interviewing (CATI) platforms and internet-based modules. By moving the possibilities for experimentation outside of the laboratory in this way, we can strengthen the internal validity of social science research and interest a much broader group of social scientists in the possibilities of experimentation.

While laboratory experiments provide strong tests of causal propositions, scientific audiences, policy makers, and the public sometimes request more than a causal demonstration. In many cases, science and society benefit from knowing that our laboratory observations survive exposure to myriad conditions outside of the lab. Moreover, some critics have questioned the extent to which the usual subjects in social science experiments resemble broader, more diverse populations (see, e.g., Sears 1986). General population experiments offer a powerful means for researchers to respond to such critiques.

General population experiments allow researchers to assign large subject populations to experimental conditions of their choosing. They use CATI and Internet technology to confront randomly selected respondents with randomly selected stimuli. General population experiments offer many advantages to social scientists. Laboratory experimenters, for example, can use general population experiments to show that observations generated in a laboratory can be replicated in very different conditions. They can also test new hypotheses that emerge from their work with smaller groups of subjects. Alternatively, experimenters who use the internet to run experiments on "knowledge networks" (e.g., markets, which aggregate diffuse individual behavior into prices and social systems that aggregate many individual characteristics into social hierarchies) can reinforce their research agendas by using general population experiments to evaluate individual-level hypotheses that inevitably follow from network-level observations (see, e.g., NetLab Workshop Report 1997). Likewise, scholars can use general population experiments to clarify the causal implications of findings from conventional surveys. As reviews of such experiments reveal (e.g., Sniderman and Grob 1996), such designs have generated important discoveries in several social sciences.

For a wide range of scholars, general population experiments offer new opportunities to innovate. A special advantage of general population experiments is the broad and diverse subject pools that they allow researchers to contact. Such experiments are particularly effective at documenting differences in the status of causal hypotheses between the type of people who are usually selected for laboratory experiments and those who are not. Though not all social scientists require large and diverse subject populations to accomplish their research goals, many do.

In addition to providing greater opportunities for experimentation on a variety of topics, TESS also provides opportunities to strengthen and improve a wide range of measurement issues. For example, those interested in how to assess race and ethnicity in an increasingly diverse society can use experimental methods to understand how the method of data collection affects response attributes (see Snipp 1986, 1998). Moreover, although the philosophy of early survey research was to attempt to create a social vacuum in which people could express their "true" beliefs and opinions, the more recent acknowledgment of attitudes, beliefs and preferences as a function of both the person and situation has led to an interest in the systematic study of how context alters the opinions and preferences that are expressed (e.g., Nisbett & Ross 1991, Tanur 1992, Britt 1993, Sudman, Bradburn, and Schwarz 1996). General population experiments are ideally suited to these purposes.

TESS offers a new opportunity to social scientists: the possibility of using the internet to reach a broad and diverse population that is likely to differ from the usual subject pool in laboratory experiments and that may differ from the typical phone-based subject pool. Moreover, they can increase the scientific and public benefits of all social science experiments by giving us more precise knowledge about what individuals are doing and thinking when they participate in face-to-face laboratory experiments, telephone-based general population experiments, or other internet based interactions. With time, technological advances promise to obliterate current distinctions between the Internet and the telephone, and may even allow a new form of face-to-face interviewing. Regardless of what direction new technology takes for these purposes, the process of interviewing will remain a fundamentally social interaction, one that needs to be studied to understand its particular dynamics and pressures (see, e.g., Hippler, Schwarz, and Sudman 1987, Jabine, Miron, Straf, Tanur, and Tourangeau 1984, Schwarz and Sudman 1986, Schwarz, Strack, and Mai 1991). For this reason we have made mode experiments a methodological priority and have incorporated, among our rank of Associate PIs, several of the country's leading experts on the psychology of human-computer interactions and their impact on mode of interview effects.