On Balance: Professional Principles for Benefit-Cost Analysts
Benefit-cost analysts essentially ask for a lot of trust. They look to inform policy decision-making by using a tool that boils very complex choices down to a seemingly simple comparison of the relative values of benefits and costs. If an analyst’s work is to be taken seriously, the decision makers must have confidence that the analyst is objective and competent, that the results being provided accurately capture what is known about any choice’s implications. The decision makers may have their own biases and interests and many choices will require decisions from a wide array of people with varying perspectives. But through the whole policy-making process, decision makers do not want to have to worry about hidden agendas, skewed data, or sloppy analysis in the information intended to inform their decisions.
The Society for Benefit-Cost Analysis, like other professional associations for people who evaluate public policies, can help build and maintain trust by encouraging people in the field to adhere to a set of principles. Those principles have two uses: they offer a guide to practitioners and also provide criteria for judging work done in the field. Decision makers should be more confident with work done by people who are guided by clear principles than with the work of people who ignore those principles. Furthermore, decision makers can use the principles to assess the analysts and their work.
So, what would a set of principles for benefit-cost analysts look like? The Society for Benefit-Cost Analysis has not established formal principles for professional conduct, but two of our sister organizations have: The American Economic Association and the American Evaluation Association. Both organizations provide guidance to promote consistent high-quality work in their fields and to maintain a productive and open environment among practitioners. Given the substantial overlap among benefit-cost analysis, economics, and evaluation, the guidance developed by the “two AEAs” offers a lot of value to benefit-cost analysts as they try to inform decision makers who often operate in highly charged policy environments.
The American Economic Association adopted a Code of Professional Conduct on April 20, 2018. The American Evaluation Association has had a set of Guiding Principles since 1994 which they updated in 2018. In both cases, the principles were developed by committees of experts in the field and were designed to enhance the ethics, credibility and integrity of work done in their respective fields.
Both sets of principles address what it means to perform professional work. In particular, they focus on intellectual and professional integrity to build the trust policy makers need. The American Economic Association lays out broad guidance noting that integrity demands honesty, care, and transparency in conducting and presenting research; disinterested assessment of ideas; acknowledgement of limits of expertise; and disclosure of real and perceived conflicts of interest. The Association also describes provides companion material suggesting best practices for conducting research, serving as a colleague, working with students, and leading workplaces and departments.
The American Evaluation Association goes further with its principles, insisting that communications with clients and relevant stakeholders concerning all aspects of an evaluation are “truthful and open” and that authors address any concerns about approaches or methodologies that are likely to produce misleading evaluative information or conclusions.
The Evaluation Association goes on to offer specific guidance about conducting “systematic inquiry,” that is, executing evidence-based studies that are thorough, methodological, and contextually relevant. Among other things, they indicate that evaluators should do the following:
-
Adhere to the highest technical standards appropriate to the methods being used while attending to the evaluation's scale and available resources.
-
Engage with stakeholders to communicate findings accurately and to be clear about the limitations and strengths of the core evaluation’s questions and the approaches that might be used for answering those questions.
-
Discuss the values, assumptions, theories, methods, results, and analyses that significantly affect the evaluator's interpretations of the findings.
-
Provide skilled professional services to stakeholders by ensuring that their evaluation team possesses the education, abilities, skills, and experiences required to complete the evaluation competently.
-
Undertake relevant education, training, or supervised practice to learn new concepts, techniques, skills, and services necessary for competent evaluation practice.
Both professional associations also articulate principles for encouraging an open and vibrant professional community. These principles deal less with conducting research and more about building a strong community among professionals. The Economic Association does this by highlighting three goals for people building the professional community:
-
Freedom of discussion: Working to create an environment where all can freely participate, where each idea is considered on its own merits, and where discourse is civil and respectful in all forums, including those that allow confidential or anonymous participation.
-
Equal Opportunity: Creating a professional environment with equal opportunity and fair treatment for all economists, regardless of age, sex, gender identity and expression, race, ethnicity, national origin, religion, sexual orientation, disability, health condition, marital status, parental status, genetic information, political affiliation, professional status, or personal connections.
-
Individual and Collective Responsibility: Ensuring that people in the field take individual responsibility for their own conduct and work to promote a collective responsibility to promote professional conduct. These responsibilities include developing institutional arrangements and a professional environment that promote free expression concerning economics. These responsibilities also include supporting participation and advancement in the economics profession by individuals from all backgrounds, including particularly those that have been historically underrepresented.
The Evaluation Association has similar principles (although with slightly different wording) and goes on to urge evaluators strive to gain an understanding of and treat fairly the range of perspectives and interests that individuals and groups bring to the evaluation, including those that are not usually included or are oppositional.
The Evaluation Association also stresses that people in the field should strive to contribute to the “common good and advancement of an equitable and just society.” This principle is seen as recognizing and balancing the interests of the client, other stakeholders, and the common good while also protecting the integrity of the evaluation. The evaluators’ principles also ask practitioners to identify and make efforts to address the evaluation's potential threats to the common good, especially when specific stakeholder interests “conflict with the goals of a democratic, equitable, and just society.” Finally, the principles ask evaluators to mitigate the bias and potential power imbalances that can occur as a result of the evaluation's context, including a self-assessment of the evaluator’s own “privilege and positioning” within that context.
The standing and influence of the practice of benefit-cost analysis and our ability to be seen as credible suppliers of relevant evidence depends on our collective professional conduct. To apply these principles to our own field, benefit-cost analysts can focus on a few demonstrable applications. I suggest we start with transparency. The goal of benefit-cost analysis is to organize complex information succinctly. In many cases, we could give a single number that purports to capture the net value created by the actions and reactions of millions of people responding to complex changes in opportunities and incentives. That single number is very attractive to decision makers, but may be misleading if it hides a host of assumptions and estimates. Benefit-cost analysts therefore must find ways to explain critical underlying assumptions, estimates, and uncertainties without undermining the simplification that makes benefit-cost analysis so attractive in the first place.
Other ways to apply principles of professional analysis are to acknowledge limitations of studies, disclose real and perceived conflicts of interest, and situate each new study in a context created by past research. Not only should our own work follow these practices, but we should encourage policy makers to look for these hallmarks and to be skeptical of analysts and studies that are not transparent or frank in assessing their limitations. In essence, studies that fail to reflect these principles should be viewed with suspicion.
As a field, we should also strive to be inclusive and to constructively engage with diverse groups. As individuals and as a professional society, we should look to engage with the full range of people affected by the decisions we seek to inform. Getting a wide array of perspectives will strengthen the field’s research and help to ensure findings can be communicated widely. Inclusion in terms of personal characteristics is essential, and it is worth noting the Evaluation Association’s principle of including people with varying “political affiliation, professional status, or personal connections.”
The call for individual and collective responsibility seems particularly important for benefit-cost analysis. The trust decision makers place in benefit-cost analysis will be shaped by the relative frequency with which they encounter studies that reflect principles like those established by the two AEAs and those that fall short. If too many benefit-cost studies lack transparency or appear to have selected underlying assumptions to generate a specific desired outcome, then the credibility of our field is weakened. So, paying attention to the principles in our own work and helping decision makers distinguish principled work from unprincipled will help build the trust required for benefit-cost analysis to play an influential role in informing policy making.
The Society for Benefit-Cost Analysis can promote professional principles by encouraging its members, many of whom are economists or evaluators, to understand and follow the principles set by the organizations representing those professions. The Society could also establish a committee to look into enhancing those principles to reflect any special features of benefit-cost analysis. In particular, the Society should consider its professional obligations to honor the ways in which benefit-cost analysis has been built into formal public policy analysis, particularly though requirements for regulatory impact analyses and OMB benefit-cost guidelines.
Please comment on this blog if you have suggestions about how our Society and profession can best maintain trust among the decision makers and the broader public they represent.