On Balance: Two Decades of Benefits and Costs: Promise and Pitfalls
A new article in the Fall Issue of the Journal of Benefit-Cost Analysis (JBCA), “Some Pitfalls of Practical Benefit-Cost Analysis,” describes common pitfalls that well-meaning analysts fall into. Over the past 23 years I have worked on and supervised hundreds of benefit-cost analyses. Most of these analyses have dealt with public health regulations proposed by the Food and Drug Administration, although on occasion I have reviewed analyses from other government agencies and academia. I’ve seen these pitfalls occur many times and at one time or another I’ve been guilty of most of them.
Benefit-cost analysis can have great value in creating efficient regulations and in making the effects of regulations and policies accessible to policy makers and the public. From public investments to traffic control to educational choice, benefit-cost analysis has shown its value. Several articles in the JBCA over the years have highlighted “good practices” in specific areas where benefit-cost analysis is done, notably papers by Susan Dudley (with others), and by Scott Farrow and Kip Viscusi.
Much of the good – which far outweighs the bad – has come from better data and methods and the growth of the profession of benefit-cost analysis (evidenced by the creation of the Society for Benefit-Cost Analysis and the JBCA). These developments include the practical use of a wide array of methods to evoke revealed and stated preferences for non-standard goods and services, the vast output of data amenable to benefit-cost manipulations, and the external benefits of having so many skilled practitioners doing and reviewing work in the field.
The bad side of benefit and costs often comes from failure to follow best practices, and the inherent difficulty of estimating benefits and costs in the messy real world. In spite of the best training, guidance, and intentions, practitioners can stumble; practical benefit-cost analysis is hard and mistakes get made.
The common mistakes made by practitioners can be sorted into either “buckets” or pitfalls. These include difficulty identifying market and other failures, the potential of partial measures to be misleading, the dangers of stories without numbers, the failure to correctly identify causation or what leads to what, the confusion associated with separating opportunity costs from transfers and vice versa, the errors caused by incorrect baselines, and the many difficulties associated with identifying behavioral failures.
Avoiding these pitfalls can be challenging. One key is to recognize the distinction between the real world (where the policy operates) and the ideal world of economic modeling (where the analysts get their tools). Thus, the practitioner needs to ask (and answer) questions such as: “What is the problem the policy is trying to fix and how large is it?” “What will the world be like with and without the policy intervention?” “Will the policy work to change behavior in desired directions?” “Are we able to quantify the most important benefits and costs?”
The final pitfall I describe is the use of old analyses renovated to fit new questions. If a previous analysis dealt with a similar question, analysts may be tempted to use it again. Uncritically using a preceding analysis, however, can lead to problems if the old analysis embodies one or more pitfalls. Moreover, methods and measures change, sometimes as part of the normal course of events and sometimes because we get better at benefit-cost analysis.
The re-use of an old analysis also illustrates the larger problem of doing benefit-cost analysis by rote, without hard thinking about what the benefits and costs really are. Most of the pitfalls can be avoided by an awareness of what the questions are and careful thought about whether the measures being used in fact capture this reality.
The value of benefit-cost analysis depends on its being done honestly and well. The existence of pitfalls demonstrates the difficulty of doing benefit-cost analysis and the need to exercise care at each step of an analysis. The value and importance of good practical analysis demands that we exercise this care; the most important way to do this is awareness of what we are trying to do and what can go wrong.