Field Experiments
Show Less

Field Experiments

John A. List and Anya Samek

This research review discusses the most critical and influential articles that utilise field experimentation to answer questions of economic importance. Field experiments have gained popularity in recent years, allowing researchers to infer causal effects of different market environments, policies and interventions. The articles analysed here provide insights into market functioning and individual and group decision-making across a wide range of domains, including marketplace transactions, labor decisions, charitable giving, financial planning, and education and health-related decision-making. This research review will be an important resource for students new to the methodology and applications of field experiments and academics alike.
Buy Book in Print
Show Summary Details
This content is available to you

Review Article: Field Experiments

John A. List and Anya Samek

Introduction

John A. List and Anya Samek

Recent years have seen an enormous increase and interest in research using field experimental methods to address questions across a wide range of domains. We are now at a point where field experiments have touched nearly all sectors of the economy, and are a common tool to test scientific theories. For example, a large literature studies marketplace behavior in both face-to-face and online settings. There is a stream of work using field experiments in firms, including studying how to increase productivity of workers through incentives and framing manipulations and exploring why consumers behave as they do. All of this occurs as an effort to open the black box we commonly call “the firm.”

Researchers have also worked with charities using field experiments to provide rigorous tests of different solicitation practices. Another group of studies focuses on behavior related to consumer financial decision-making and financial planning. Another strand uses field experiments to explore environmental concerns. Yet another group of researchers partners with schools with the goal of using field experiments to improve academic achievement. And, finally, field experiments have touched the health domain as well, where a strand of literature uses this method to study how to promote healthy eating and physical activity.

In this research collection, we have assembled a series of articles that utilize field experimentation to answer questions of economic importance – for purely scientific purposes, for policy purposes, and to inform practice. While there is by now a large literature using field experiments, we have limited our book to include select studies that represent some of the breakthroughs in using the field experimental approach across the main sectors of the economy discussed above. We would be remiss not to note that there are hundreds of other studies that we could have chosen. For this we beg forgiveness from those omitted authors.

Many of the selected papers use natural field experiments, which combine an attractive element of the traditional laboratory model – randomization – with an attractive element of field data – realism. And, these are all conducted with experimental participants who are making decisions in natural environments, and who are unaware that they are part of an experiment (Harrison and List, 2004). In this way, natural field experiments combine the most attractive elements of the experimental method and naturally occurring data: randomization and realism. Some other papers in this compilation use framed field experiments, where participants are recruited into a decision setting but the natural decision context is intact; or artefactual field experiments, where non-student subsets of the population are used in a laboratory setting (Harrison and List, 2004).

As noted by Harrison and List (2004), in some sense, every empirical researcher searching for causality is an experimentalist. Field experiments represent a useful tool for such scholars for answering economic questions. A major benefit of field experiments relative to naturally occurring data is the ability to infer causality through randomization, which is much more difficult with naturally occurring data. For instance, while naturally occurring data can measure if men and women are treated differently in markets, it is very difficult to go beyond said measurement and decipher the “whys” behind the differences. Field experiments can provide such insights.

A growing discussion exists regarding the benefits and drawbacks of the field experimental method relative to laboratory experiments, which are a complementary approach. Laboratory experiments differ from field experiments in that the population under study is usually university students, the study occurs in a laboratory on a college campus, and the context of the decision is removed. Laboratory experiments and field experiments have several commonalities, including the reliance on randomization (typically).

One main theme that emerges when scholars inevitable compare the two (complementary) approaches is that field experiments may provide more generalizability (that is, by letting the researcher infer results on the population or context of interest) but come with a loss of control (for example, because the practitioner partner has constraints on what can be done or has control of the implementation and not the researcher). Yet, this trade-off is a false one, and it is not always as clear as one initially imagines (see Al-Ubaydli and List, 2015). As Al-Ubaydli and List (2015) show, contrary to conventional wisdom, natural field experiments can offer experimenters more control than laboratory experiments, specifically in terms of the participation decision. Readers interested in a more thorough discussion are advised to read reviews that provide summaries (Harrison and List, 2004; Ortmann, 2005; Carpenter et al., 2005; List, 2007; List, 2008; Levitt and List, 2009; Roe and Just, 2009; List and Reiley, 2010; List, 2011; Charness et al., 2013; Al-Ubaydli and List, 2015; Samek, 2018).

Field experimentation is still a relatively new area of study – many of the most influential papers in the field have been published over the last decade. In this compilation, we include works that have pushed the boundaries of field experimentation. We include some of the earliest influential works that began in early 2000, as well as more recent promising studies. One area that we exclude is environmental and energy economics. This is mainly because we cover it by itself in a related volume (see List and Price, 2013).

We start with market environments since the market is the cornerstone of economic decision-making. Some of the earliest papers in this section are List and Lucking-Reiley (2000, Chapter 1 this volume) and List (2004, Chapter 2 this volume) who visited naturally occurring markets (the sports card and collector pin market) to test market theories, including auction theory and neoclassical competitive theory. Tests of this type are important because they help us understand the predictive power of economic theory to real-world contexts. Another set of papers focused on testing economic theories on Internet markets, including the value of reputation on eBay (Resnick et al., 2006, Chapter 3 this volume) and the design of knowledge markets on Google Answers (Chen et al., 2010, Chapter 6 this volume). The remaining two papers in this section studied the rapidly expanding microcredit market (de Mel et al., 2008, Chapter 4 this volume; Karlan and Zinman, 2009, Chapter 5 this volume).

In Part II, we assemble field experiments in firms. Fehr and List (2004, Chapter 7 this volume) study the behavior of Chief Executive Officers (CEOs) in situations requiring trust and trustworthiness, comparing this behavior to university students in a laboratory. Several papers in this part focus on the effect of different incentives on worker effort (Gneezy and List, 2006, Chapter 8 this volume; Bandiera et al., 2007, Chapter 9 this volume; Fehr and Goette, 2007, Chapter 10 this volume). And one paper explores whether simple framing manipulations can similarly increase worker effort and productivity (Hossain and List, 2012, Chapter 11 this volume).

Part III of the compilation is one of the largest, reflecting the inroads that field experiments have made in the charitable giving literature. Research in charitable giving is attractive for several reasons. First, the field is large – in the U.S., nearly $300 billion is contributed to charity annually. Second, studying charitable giving behavior allows for tests of various social preference theories, which are of interest in behavioral economics. The papers in this section test the impact of different solicitation strategies, such as seed money (List and Lucking-Reiley, 2002, Chapter 12 this volume), recognition (Soetevent, 2005, Chapter 13 this volume), lotteries (Landry et al., 2006, Chapter 14 this volume), unconditional gifts (Falk, 2007, Chapter 15 this volume), social information (Shang and Croson, 2009, Chapter 16 this volume; Chen et al., 2010, Chapter 17 this volume), and matching gifts (Huck and Rasul, 2011, Chapter 19 this volume). One paper also investigates pay-what-you-want pricing (Gneezy et al., 2010, Chapter 18 this volume). Finally, DellaVigna et al. (2012, Chapter 20 this volume) use structural estimation to understand the welfare costs of a solicitation on potential donors.

Part IV focuses on financial decision-making. This includes interventions to encourage better savings behavior (Duflo et al., 2006, Chapter 21 this volume) and tax compliance (Hallsworth et al., 2017, Chapter 25 this volume). A strand of literature has also studied present-biased and time preferences to learn how they are associated with real-world financial behaviors (the papers represented here include Tanaka et al., 2010, Chapter 23 this volume, and Meier and Sprenger, 2010, Chapter 24 this volume). Finally, Bertrand et al. (2010, Chapter 22 this volume) study the impact of advertising content on consumer credit take-up.

Part V provides a selection of research articles focusing on using field experiments to address environmental concerns. Understanding whether people overstate their preferences for a good when asked a hypothetical question is important for environmental policy. List and Shogren (1998, Chapter 26 this volume) use a field experiment to study the difference between actual and hypothetical valuations in an auction. Stoop et al. (2012, Chapter 27 this volume) study cooperation among fishermen in social dilemmas.

Part VI includes a growing literature that harnesses field experiments to understand how to improve education and reduce the academic achievement gap. Several papers study the impact of financial and non-financial incentives on student performance (Angrist and Lavy, 2009, Chapter 30 this volume; Fryer, 2011, Chapter 31 this volume; Levitt et al., 2012, Chapter 34 this volume). Others look at the impact of teachers on educational attainment and aspirations (Banerjee et al., 2007, Chapter 28 this volume; Dee, 2004, Chapter 29 this volume; Beaman et al., 2012, Chapter 33 this volume). Finally, one paper considers the association of children’s time preferences with educational outcomes (Castillo et al., 2011, Chapter 32 this volume).

Part VII focuses on studies that use field experiments to improve health outcomes. Charness and Gneezy (2009, Chapter 35 this volume) study the impact of incentives on exercise. Wisdom et al. (2010, Chapter 37 this volume) focus on studying the impact of information and convenience on food choice in a fast food restaurant. List and Samek (2015, Chapter 39 this volume) study the impact of information and incentives on child food choice in school. Harrison et al. (2010, Chapter 38 this volume) study the association between time preferences and smoking behavior. Finally, Björkman and Svensson (2009, Chapter 36 this volume) explore the impact of community monitoring on health care providers in a developing country.

The papers in this compilation address several cross-cutting themes, which are reflective of the broader literature that uses field experimentations. Most of the papers focus on using interventions to improve market or societal outcomes (e.g., Resnick et al., 2006, Chapter 3 this volume; Chen et al., 2010, Chapter 6 this volume; Gneezy and List, 2006, Chapter 8 this volume; Bandiera et al., 2007, Chapter 9 this volume; Fehr and Goette, 2007, Chapter 10 this volume; Duflo et al., 2006, Chapter 21 this volume; Halsworth et al., 2017, Chapter 25 this volume; Bertrand et al., 2010, Chapter 22 this volume; and most papers in Parts III to VI). These papers also provide insights into economic theory. Several others are aimed more directly at testing theory (List and Lucking-Reiley, 2000, Chapter 1 this volume; List, 2004, Chapter 2 this volume). Others focus on associating economic preferences with real-world outcomes (Tanaka et al., 2010, Chapter 23 this volume; Meier and Sprenger, 2010, Chapter 24 this volume; Castillo et al., 2011, Chapter 32 this volume). Finally, Fehr and List (2004, Chapter 7 this volume) compare the economic behavior of professionals and university students.

As can be inferred by reading through the papers in this compilation, conducting field experiments – especially natural field experiments – can be a daunting task. Researchers new to the method who would like to learn more about practicalities are advised to read List (2011). Carroll and Samek (2017) also provide a guide focused on experiments in grocery stores.

This compilation is aimed at faculty, graduate students, and researchers seeking to understand what field experiments can teach us – both practically and theoretically. It can also be used in graduate courses on field experiments to provide a guided reading of recent key papers in the field. We hope you enjoy reading the papers in this compilation as much as we did.

References

  • Al-UbaydliOmar and John A. List“Do natural field experiments afford researchers more or less, control than laboratory experiments?” American Economic Review Papers and Proceedings 105 no. 5 (2015): 462466.

    • Search Google Scholar
    • Export Citation
  • CarpenterJeffrey P.Glenn W. Harrison and John A. List. “Field experiments in economics: An introduction.” In Field Experiments in Economics pp. 115. Emerald Group Publishing Limited2005.

    • Search Google Scholar
    • Export Citation
  • CarrollKathryn and Anya Samek. “Field Experiments on Food Choice in Grocery Stores: A ‘How-to’ Guide.” Work in progress 2017.

    • Search Google Scholar
    • Export Citation
  • CharnessGaryUri Gneezy and Michael A. Kuhn. “Experimental methods: Extra-laboratory experiments – extending the reach of experimental economics.” Journal of Economic Behavior & Organization 91 (2013): 93100.

    • Search Google Scholar
    • Export Citation
  • HarrisonGlenn W. and John A. List. “Field experiments.” Journal of Economic Literature 42 no. 4 (2004): 10091055.

  • LevittSteven D. and John A. List. “Field experiments in economics: The past, the present, and the future.” European Economic Review 53 no. 1 (2009): 118.

    • Search Google Scholar
    • Export Citation
  • ListJohn A. “Field experiments: a bridge between lab and naturally occurring data.” The BE Journal of Economic Analysis & Policy 5 no. 2 (2007).

    • Search Google Scholar
    • Export Citation
  • ListJohn A. “Homo experimentalis evolves.” Science 321 no. 5886 (2008): 207208.

  • ListJohn A. “Why economists should conduct field experiments and 14 tips for pulling one off.” The Journal of Economic Perspectives 25 no. 3 (2011): 315.

    • Search Google Scholar
    • Export Citation
  • ListJohn A. and Michael K. Price eds. Handbook on Experimental Economics and the Environment. Edward Elgar Publishing2013.

  • ListJohn A. and David Reiley. “Field experiments.” In Microeconometrics pp. 5358. Palgrave Macmillan UK2010.

  • OrtmannAndreas. “Field experiments in economics: Some methodological caveats.” In Field Experiments in Economics pp. 5170. Emerald Group Publishing Limited2005.

    • Search Google Scholar
    • Export Citation
  • RoeBrian E. and David R. Just. “Internal and external validity in economics research: Tradeoffs between experiments, field experiments, natural experiments, and field data.” American Journal of Agricultural Economics 91 no. 5 (2009): 12661271.

    • Search Google Scholar
    • Export Citation
  • SamekAnya. “Advantages and disadvantages of field experiments.” In Handbook of Research Methods and Applications in Experimental EconomicsArthur Schram and Aljaz Ule eds. Forthcoming 2018.

    • Search Google Scholar
    • Export Citation