Program Evaluation and Performance Measurement at the EPA

Program evaluation and performance measurement work together to identify areas of improvement and focus for programs and determine progress towards identified goals or objectives. They serve different but complimentary functions as programs move through their work. Different sources of evidence hold distinct value.

For example, program evaluation and performance measurement are key tools for federal program management, but differ in the following ways:

	Program Evaluation	Performance Measurement
What drives it?	Theory of program change	Agency goals
What data it uses	Quantitative or qualitative	Typically quantitative
What frequency	Discrete	Ongoing
What it can tell	Whether a program is working and why	How well a program is performing

Program evaluations are individual systematic studies (measurement and analysis) that assess how well a program is achieving its outcomes and why. There are six types of evaluation commonly conducted, which are described below.

Performance measurement is an ongoing process that monitors and reports on the progress and accomplishments of a program by using pre-selected performance measures. This helps describe the “what” that is being achieved.

So, performance measurement data describes program achievement, and program evaluation explains why we see those results.

How is program evaluation different from performance measurement?

A program sets performance measures as a series of goals to meet over time. Measurement data can be used to identify/flag areas of increasing or decreasing performance that may warrant further investigation or evaluation. Program evaluations assess whether the program is meeting those performance measures but also look at why they are or are not meeting them. At EPA, the goal is to build robust and effective measures that support high quality evaluations. These measures may change over time as programs shift and mature and lessons learned and built back into programs.

For example, imagine you bought a new car that is supposed to get 30 miles per gallon. But say, you notice that you are only getting 20 miles per gallon. That's a performance measurement. You looked at whether your car was performing where it should be. So what do you do next? You would take it to a mechanic. The mechanic's analysis and recommendations would be the program evaluation because the mechanic would diagnose why the car is not performing as well as it should. You need performance measures to know whether your program (or car) is performing where it should be, and you do a program evaluation (or go to the mechanic) to find out the reason why it is not meeting those expectations.

How do program evaluation and performance measurement work together?

Performance measurement and program evaluation may be different, but both rely on one another to determine good measures. While performance measurement evaluates what has been achieved, an evaluation helps assess what can be fixed for the future. They are part of an overall process of Continuous Improvement that all programs at EPA strive towards.

What types of program evaluations are there?

Program evaluations can assess the performance of an EPA program at all stages of a program's development. The type of program evaluation conducted aligns with the program's maturity (e.g., developmental, implementation, or completion) and is driven by the purpose for conducting the evaluation and the questions that a program seeks to answer. The purpose of the program evaluation determines which type of evaluation is needed and the types are described below.

Design Evaluation

A design evaluation is conducted early in the planning stages or implementation of a program. It helps to define the scope of a program or project and to identify appropriate goals and objectives. Design evaluations can also be used to pre-test ideas and strategies.

Process Evaluation

A process evaluation assesses whether a program or process is implemented as designed or operating as intended and identifies opportunities for improvement. Process evaluations often begin with an analysis of how a program currently operates. Process evaluations may also assess whether program activities and outputs conform to statutory and regulatory requirements, EPA policies, program design or customer expectations.

Outcome Evaluations

Outcome evaluations examine the results of a program (intended or unintended) to determine the reasons why there are differences between the outcomes and the program's stated goals and objectives (e.g., why the number and quality of permits issued exceeded or fell short of the established goal?). Outcome evaluations sometimes examine program processes and activities to better understand how outcomes are achieved and how quality and productivity could be improved.

Impact Evaluation

An impact evaluation is a subset of an outcome evaluation. It assesses the causal links between program activities and outcomes. This is achieved by comparing the observed outcomes with an estimate of what would have happened if the program had not existed (e.g., would the water be swimmable if the program had not been instituted).

Cost-Effectiveness Evaluation

Cost-effectiveness evaluations identify program benefits, outputs or outcomes and compare them with the internal and external costs of the program.

Formative Evaluation

A Formative Evaluation assesses the feasibility of a program or policy before its full implementation. This type of evaluation can include process and outcome measures, but its overall goal is to assist with proactive learning and improvement.