On Balance: The Persistence of Appraisal Optimism in Benefit-Cost Analysis

In my paper for this journal earlier this year (Abelson, 2020), I discussed how seven official guides to benefit-cost analysis (BCA) and the leading international text on BCA (Boardman et al., 2018) deal with eight contentious issues: the issue of standing, core valuation principles, the scope of CBA, changes in real values over time, the marginal excess tax burden, the social discount rate, the use of benefit-cost ratios, and the treatment of risk. I did not discuss, however, arguably the most potent cause of poor BCA studies: appraisal optimism, which is sometimes referred to less courteously as appraisal bias. Indeed, appraisal optimism receives little attention in most BCA textbooks and official guides. I will attempt here a partial rectification of these omissions.

 

I start with some substantial evidence for appraisal optimism in project evaluations. I then discuss the alleged two main drivers: cognitive biases and incentives in government appraisal processes.  Finally, I discuss some remedies. But I am not optimistic that the virus will entirely disappear. 

Evidence of appraisal optimism

As UK Treasury (2013) states: “There is a demonstrated, systematic, tendency for project appraisers to be overly optimistic”. Many studies have reported project cost and schedule overruns, as well as over-estimation of project benefits though these effects tend to be less manifest. Here, I briefly cite a few reports and results across several economic sectors (the tip of a melting iceberg). 

Mott MacDonald (2002) examined 50 large public projects across all sectors in the UK and found an average capital cost overrun of 47%, operating cost overrun of 42% and schedule overrun of 17%. 

Flyvbjerg et al. (2003), reporting on 258 transport initiatives across 20 countries, found average capital cost overruns in major transportation projects of 20% for roads, 34% for large bridges and tunnels, and 45% for rail. 

Flyvbjerg et al. (2005), covering 210 projects in 20 countries, found passenger forecasts overestimated for 9 out of 10 rail projects, with average overestimation 106%. 

Budzier and Flyvbjerg (2011) found 107 percent overrun for major IT investments. 

Ansar et al. (2014) found an average 90 percent cost overrun for 245 large dams built in 65 countries. 

Zimbalist (2015) reports in detail both high cost overruns and large over-estimated benefits for several Olympic Games and Soccer World Cups. Flyvberg (2020): every Olympics since 1960 has run over budget, at an average of 172 percent in real terms. 

Liu et al. (2018). In a study of capital costs of 19 long bridges in China, 3 bridges were marginally under budget and 16 were over budget including 8 by more than 15%.

Terrill (2018) found that between 2001 and 2016, Australian governments spent $28 billion more on 836 transport projects than they told taxpayers they would. 

Flyvberg and Bester (Forthcoming).1 Based on a sample of 2062 infrastructure investments in 104 countries, covering dams, bus rapid transit, rail, tunnels, power plants, buildings, bridges and roads, the paper finds that average cost overruns were around 40% and benefit overestimates averaged about 10 per cent, but with some substantial variations. The average overestimates in ex-ante benefit-cost ratios varied from 50 to 200 percent, depending on investment type.

Is optimism a cognitive problem or a political one?  

Boardman et al. (2018, p.185) define cognitive bias as a tendency to making “overoptimistic predictions”. This view of decision making has a distinguished provenance. In 1979, Kahneman and Tversky identified a cognitive bias known as the planning fallacy. They defined this phenomenon as “the tendency to underestimate the amount of time needed to complete a future task, due in part to the reliance on overly optimistic performance scenarios”. 

And many professors of psychology believe in cognitive optimism. For example, in a much-cited book, neuroscientist Professor Sharot (2013) contends that that “our brains may be hardwired to look on the bright side. Both neuroscience and social science suggest that we are more optimistic than realistic. On average, we expect things to turn out better than they wind up being”. 
 
A political explanation would be that bureaucrats and consultants advising government have strong incentives to support and advance government projects, especially those already announced. The bureaucrat who advises his or her minister that benefits do not justify the costs for most of their projects is unlikely to get promoted. Consultants who cannot be trusted to support government projects are unlikely to be re-engaged. 

In this writer’s experience, the second explanation is more convincing. Over many years advising governments and sitting in meetings with public servants and consultants, I would not characterize many of these bureaucrats and consultants as natural optimists. 

More authoritatively, I would point to Little and Mirrlees (1991) classic explanation for poor cost-benefit studies over the previous 20 years in the World Bank. “The difficulty is that appraisers will be judged on the number of "good" projects they find relative to the numbers others find… Worse than this, project analysts would never get promoted if they were honestly compelled to report unfavorably on several projects.”

In the US, the President has long appointed the top officials in the major Executive Branch Agencies. And under President Trump, admittedly an unusual US President, loyalty to the President’s policies was a well-known pre-requisite for retaining a top appointment. A recent article in the New Yorker2 observed that the Treasury Secretary, Steven Mnuchin, was one of the few remaining original cabinet members under President Trump. “He has a reputation for unflinching loyalty… which serves as a kind of job insurance”. 

Moreover, at least in some countries, power has been shifting away from an independent public service and towards ministers’ private offices. In the UK, government under Boris Johnson has moved power towards private office advisors close to the Cabinet Office over independent, long-term, public servants.3  

In Australia, senior public positions are increasingly occupied by persons associated with the governing party. In a major speech in 2019, the Prime Minister (Scott Morrison) declared that responsibility for setting policy lies with those elected. It is the job of public servants to carry out these policies. Public servants must remember that “they are on tap, not on top”.4 

Thirty years on, Little and Mirrlees’ findings almost certainly extend well beyond the World Bank. 

What are the remedies?

A start is recognition of the problems. As advocated by Boardman et al (ibid, p.269), we should use expected values to take account of risks rather than most likely or median (P50) values. Expected values allow for the asymmetric distribution of many project costs, where costs may be underestimated by 100% but are rarely over-estimated by more than 10%. Of course, to avoid tail sensitivity, the outlying estimates should be included with appropriately low probabilities. 

Secondly, allowances for contingencies should be included based on evidence, drawing on comparable projects and their outcomes: sometimes described as “reference class forecasting”.  But some caution is needed here. Applying specific cost uplifts as recommended by UK Treasury (2018) may reinforce cost inflation rather than address it.   

Thirdly, benefit-cost reports should be transparent and testable. They should provide spreadsheets showing the main categories of costs and benefits and the key data inputs. In my experience, many reports provide discussions of methods and assumptions and then the results along with some mechanical sensitivity tests. But these headline results are not readily testable.  

Fourthly, for medium and large projects, independent experts should be engaged to review the forecast costs and benefits. Kahnemann (1994) described this as an "outsider's view". However, if the biases have political causes, the concept of an independent reviewer may be an oxymoron. 

Conclusions

Optimism bias is a chronic problem. Whether the drivers are cognitive bias or political, or both, there are no easy technical solutions. But we must keep trying to give frank and fearless advice. 

1The results in this paper may include cases previously reported by Flyvberg.
2 Kolhatkar, “Dollar for Dollar”, The New Yorker, July 2020.
3 https://www.instituteforgovernment.org.uk/explainers/special-advisers
4 https://theconversation.com/scott-morrison-tells-public-servants-keep-in-mind-the-bacon-and-eggs-principle-122021.

References
Abelson, P., 2020, “A Partial Review of Seven Official Guidelines for Cost-Benefit Analysis”, Journal of Benefit-Cost Analysis, 11, 2, 272-293.

Ansar, A., Flyvbjerg, B., Budzier, A., and D. Lunn, 2014, “Should we build more large dams? The actual costs of hydropower megaproject development”, Energy Policy, 69: 43-56. 

Boardman A., Greenberg, D., Vining, A. and D. Weimer, 2018, 5E, Cost-Benefit Analysis, Concepts and Practice, Prentice Hall. 

Budzier, A. and B. Flyvbjerg, B., (2011), “Double Whammy – How ICT Projects are Fooled by Randomness and Screwed by Political Intent”, Working Paper, Saïd Business School, University of Oxford.

Flyvbjerg, B., Holm, M. and S. Buhl, 2003, “How common and how large are cost overruns in transport infrastructure projects?”, Transportation Reviews, 23, 71–88. 

Flyvberg, B., Mette, K., Skamris, H. and S. Buhl, 2005, “How (In)accurate Are Demand Forecasts in Public Works Projects?: The Case of Transportation”, Journal of the American Planning Association, 71, 2, 131-146.

Flyvbjerg, B., Budzier, A. and D. Lunn, 2020, "Regression to the Tail: Why the Olympics Blow Up," Environment and Planning A: Economy and Space. On-line url: https://ssrn.com/abstract=3686009

Flyvbjerg, B and D. Bester, forthcoming, "How (In)Accurate Is Cost-Benefit Analysis? Data, Explanations, and Suggestions for Reform”, Chapter 10 in T. Gomez-Ibanez and Z. Liu (eds.), Infrastructure Economics and Policy: A Reader, Brookings Institution Press, Washington, D.C.

Kahneman, D. and A. Tversky, 1979, “Prospect Theory: An Analysis of Decision Making Under Risk.” Econometrica, 47, 2, 263–291.

Kahneman, D.,1994, “New challenges to the rationality assumption”, Journal of Institutional and Theoretical Economics, 150, 18-36. 

Little, I. and J. Mirrlees, 1991, “Project Appraisal and Planning Twenty Years On”, proceedings of the 1990 World Bank Annual Conference on Development Economics, World Bank, Washington.

Liu, H., Jiang, C., Liu. Y. Hertogh, M. and X. Lyu, 2018, “Optimism Bias Evaluation and Decision-Making Risk Forecast on Bridge Project Cost Based on Reference Class Forecasting: Evidence from China”, Sustainability, 10, 3981. 

Mott Macdonald, 2002, Review of Large Public Procurement, prepared for UK Treasury.

Sharot, T., 2011, The Optimism Bias: Why we're wired to look on the bright side, Little, Brown Book Group.

Terrill, M., 2018, Cost overruns in Transport, Grattan Institute, Melbourne.

UK Treasury, 2013, Supplementary Green Book Guidance – Optimism Bias.

UK Treasury, 2018, The Green Book Central Government Guidance on Appraisal and Evaluation

Zimbalist, A., 2015, Circus Maximus, Brookings Institution.

Share this post:

Comments on "On Balance: The Persistence of Appraisal Optimism in Benefit-Cost Analysis"

Comments 0-5 of 0

Please login to comment