mPremium content.
registered users Learn more

Opportunities and Challenges for the Use of Evidence

Alexander Sileo, Research Associate and Special Assistant, The Forum for Youth Investment

Friday, March 9, 2018

On February 28th, agency staff, academic researchers and nonprofit officials gathered at the Bipartisan Policy Center’s offices to discuss how evidence was used in the Clinton, George W. Bush and Obama administrations as well as what opportunities policymakers can take advantage of to spread the use of evidence in 2018. The event, co-sponsored by the Forum for Youth Investment and the Bipartisan Policy Center, featured a new technical paper from the Bipartisan Policy Center on evidence initiatives in the Bush and Obama administrations as well as a panel of former political appointees from the last three administrations moderated by the Forum’s co-founder and executive vice president, Merita Irby.

The event’s first panel featured Nick Hart, Director of the Evidence-Based Policymaking Initiative at the Bipartisan Policy Center, and Kathryn Newcomer, Director of the Trachtenberg School of Public Policy and Public Administration. The two panelists co-authored the technical paper and discussed their findings with the audience. Nick Hart noted that President George W. Bush’s management agenda integrated budget and performance through the PART initiative which sought to rate programs based on how they met specific goals. This led to new performance measures and evaluations across the federal government and led to some tough conversations among agency officials about what programs were trying to accomplish.

The Obama administration used the updated Government Performance and Results Act to set priority goals for agencies. They also proposed numerous new funding initiatives, set aside authorities, and new evaluation positions. These were unevenly supported by Congress over President Obama’s two terms in office with only some agencies choosing to utilize these ideas. Other initiatives like the My Brother’s Keeper Task Force sought to analyze how the government uses public data files and disaggregates data to explore potential racial gaps.

Kathryn Newcomer explained how some of these initiatives failed to provide as much change as expected. Many of these involved randomized controlled trials (RCT) which raised the stakes for agency officials and may have dampened the enthusiasm for evidence use. The Forum similarly documented this issue in our Managing for Success report where one federal official noted “we have a history in which negative results on an evaluation can mean that the money goes away. But the problem you are trying to solve has not gone away. The ‘fear factor’ is real for programs—if anyone fears that the information will be used to kill their program, the learning won’t happen.” Agencies may have perceived these efforts by the George W. Bush and Obama administrations to be focused solely on accountability and ignoring improvement. This perception can weaken enthusiasm for the use of evidence.

Both administrations also used websites to increase transparency around data and evidence, but Newcomer noted that these websites were not necessarily in a format that would reach key target audiences. Finally, there were silos between evaluation and performance management which meant that agency officials were not learning from each other to the greatest extent possible. These phenomena were also documented in the Forum’s report: “Instead of having a unified, cohesive infrastructure supporting all types of evidence, the federal government currently has what may be described as series of separate infrastructures. While there are plenty of exceptions, in general, there is one infrastructure to support statistics, a separate infrastructure set up to support performance management, and yet another infrastructure set up to support evaluation.”

The Bipartisan Policy Center’s paper recommended a number of ways to advance the building and use of evidence moving forward. The panelists suggested that agencies should prioritize learning agendas and building a portfolio of evidence. They also suggested that implementation and process evaluations could help support agency efforts in addition to the more common RCT studies. The paper recommended that officials coordinate across the evidence-building community to leverage the strengths of different silos. Policymakers should also spend more time focusing on building the audience for evidence and making the correct type of evidence available in order to increase engagement among evidence consumers.

The event’s second panel (moderated by the Forum’s co-founder Merita Irby) featured former political appointees from the Clinton, George W. Bush, and Obama administrations: John Kamensky (Deputy Director for Vice President Gore’s National Partnership for Reinventing Government); Marcus Peacock (Deputy Administrator for the U.S. Environmental Protection Agency during the George W. Bush administration); and Shelley Metzenbaum, (Associate Director for Performance and Personnel Management at the White House Office of Management and Budget during the Obama administration).

John Kamensky noted that a lot of the work from the Clinton administration focused on moving from measuring processes to results and outcomes. He specifically mentioned work by the U.S. Coast Guard involving the inspection of boats for safety issues as an example of where positive changes took place. Marcus Peacock focused on the need to make evidence initiatives bipartisan in order to increase buy-in. Labeling evidence initiatives as from a particular administration can create an “us vs. them” mentality between parties or between branches of government. Shelley Metzenbaum shared that the Obama administration had three objectives for its evidence activities: focus (by setting priority goals), communicate (by building the audience for evidence), and network (by creating learning groups within government).

The panelists had a number of suggestions for current policymakers. Shelley Metzenbaum emphasized the need to report information on trends relating to children and youth and to then have meetings across silos about these topics. This focus would lead to greater interagency work and focus policymakers on key problems. Marcus Peacock advocated for more Pay for Success initiatives as a potential bipartisan tool to spread the use of evidence. John Kamensky said agencies should focus on core capacities and step back from grand ideas.

A recording of the event is located here.