MGT 324 Program Evaluation Discussion

1
Explain the different way that you consider it in evaluating a situation or program that you would
like to attend; provide your answer with example.
CHAPTER 8
Program Evaluation
In Chapter 8, students are presented with the fundamentals
of program evaluation. Upon reading this chapter, students
will understand what program evaluation is and how one goes
about completing an evaluation. This chapter discusses the
various techniques for collecting information (or data) and the
importance of stakeholders in this process. It also examines
the types of program evaluations and concludes with a
discussion of ethical conduct for program evaluators.
258
CHAPTER 8
“Management
the science
of which
“Evaluation
is theisprocess
of determining
organizations
are
butvalue
experiments.”
the
merit, worth,
and
of things.”
JOHN CONSTABLE
MICHAEL SCRIVEN
English Romantic Painter
Evaluation Theorist
(1776-1837)
A focus group can provide valuable feedback.
Postal Workers Mural
Program Evaluation
259
WHAT IS PROGRAM EVALUATION?
Throughout the late 1950s and early 1960s, the policy sciences emerged as a means
of studying and addressing some of the most pressing societal problems through the
use of highly quantitative and quasi-scientific approaches to social problem solving.
Examples of these approaches include operations research and planning programming budgeting systems (PPBS). Operations research entails the use of statistics and
mathematical modeling in decision making, while PPBS is the systematic comparison
of different programs with regard to costs and effectiveness. With the advent of the
antipoverty movement during the administrations of President John F. Kennedy and
President Lyndon Baines Johnson, the policy science community saw an opportunity to contribute in the area of policy formulation. Policy science contributions culminated with the Economic Opportunity Act of 1964, which
was the centerpiece of Johnson’s Great Society agenda. In
“Everything that
spite of a comprehensive legislative effort to combat poverty,
can be counted
there was little improvement in the lives of the nation’s poor.
does not
Johnson’s antipoverty programs were, by and large, unsucnecessarily count;
cessful, and this consequently altered the focus of the policy
everything that
science community, whereby program evaluation research
counts cannot
moved to the forefront (deLeon 1988).
necessarily be
counted.”
Program evaluation is the use of social science research
methods in an effort to determine whether a public program
ALBERT EINSTEIN
is worthwhile. Program evaluation is systematic—which is
Physicist;
Nobel Prize Winner
to say that it is a quasi-scientific process. Unlike more tra(1879–1955)
ditional academic research, program evaluation is more
client-centered. Evaluation research is often referred to as
applied research. There are two schools of thought regarding how program evaluations should be conducted. On the one hand, some would argue that program evaluations should follow social science research principles to the letter, whereby very
few, if any, concessions are made regarding the needs of the client. In other words,
program evaluations should differ very little from purely academic research that appears in scholarly publications. On the other hand, while program evaluations must
be grounded in social science research principles, it is necessary to take into account
the individual needs of the client (Rossi, Lipsey, and Freeman 2004).
There are several reasons why program directors initiate an outside evaluation. For
most program directors, the motivation for an evaluation is to gain knowledge and
improve some aspect of their program. For some others, however, political or public relations considerations serve as motivation. At times, program evaluations are
ritualistic endeavors that are meant to appease policymakers and/or advocacy
groups that are pressuring a government department or agency for more accountability or better results.
260
CHAPTER 8
For an evaluator, it is important to have an understanding as to what is motivating
an evaluation. Evaluations motivated by veiled agendas or politics raise the possibility that a program’s director may apply pressure on an evaluator to conduct an
evaluation that lacks necessary objectivity (that is, the absence of personal bias). If
the motivation of an evaluation is unclear, the evaluator may ask:
•
Why is there a need for this program to be evaluated?
•
What questions will this evaluation try to answer?
•
How will the research results and data be used?
How to Collect Empirical Data
Program evaluations are based on the collection of empirical data. Empirical means
observable through one’s senses. In other words, empirical data are seen or heard,
and they are typically collected by surveys, in-depth interviews, focus groups, field
observations, experimentation, and existing data (Singleton and Straits 2004). Surveys involve asking a significant number of individuals a set of structured questions.
These questions are usually close-ended, which means that the survey respondent
is presented with a question and a specific number of corresponding answer choices.
Some surveys use open-ended questions, whereby the respondents provide their
own answers. Examples of closed and open-ended survey questions might include:
Close-ended example:
How would you rate the quality of the after-school mathematics tutoring program?
A. Excellent
B. Above average
C. Average
D. Below average
E. Poor
F. Unsure
Open-ended example:
How would you rate the quality of the after-school mathematics tutoring program?
Close-ended survey questions are preferable for several reasons. First, the data collection process can be completed more quickly and efficiently, as survey respondents are not compelled to search for answers. Second, it is easier for an evaluator
to organize, summarize, and analyze data obtained via close-ended survey questions. Third, close-ended questions are easier for respondents, which reduces what
is known as survey response fatigue. Completing a survey requires you to think,
and as such, you will begin to tire at some point during the survey. Response fatigue
Program Evaluation
261
The Pros and Cons of Using Surveys
to Measure
Accurate measurement requires valid data. This requires reliable data collection procedures. Several methods of data collection are available, including written surveys,
telephone surveys, focus groups, interviews, intercepts, experiments and systematic
observation. Of these, written surveys are probably the most widely used and abused.
Because they can be written fairly easily, printed surveys are often used rather than
more in-depth, difficult and expensive techniques. However, surveys should only be
used when information from other methods is not available or to verify the results of
other methods. If questionnaires are written, tested and administered correctly, the
results can be most informative. If they are not, the results can be misleading and, if
used for policy decisions, potentially disastrous.
When Should I Use a Survey?
•
Physical Limitations. Sometimes it is physically impossible to interview members of a population. For example, the population of air travelers is large, ill-defined and in constant motion. They are not
predisposed to stop and submit to interviews or intercepts. However,
they might complete a survey after being boarded if one is included
with a boarding pass or with a complimentary beverage after being
seated.
•
Cost Limitations. The cost of interviewing a representative sample of a
population is often prohibitive. Staff and training costs are major considerations.
•
Time Limitations. Even if a large staff of interviewers could be funded,
assembled and trained, the time necessary to do this and then to interview a selection of a population is significant. If timeliness is critical,
this can be a determining factor.
•
Confirmation. Surveys are well suited to confirm the findings of other
types of data collection. They can be used as follow-up inquiries after
interviews and focus groups to concentrate narrowly on specific issues
uncovered by these prior methods.
Advantages of Written Surveys
•
262
Inexpensive. They are not as labor-intensive as interviews or focus
groups. A well-designed survey can generate a substantial amount of
data without requiring a significant investment of staff time.
CHAPTER 8
•
Easy to Administer. Since the questions are written, staff members do
not need to be trained in techniques of interviewing, systematic observation or focus group facilitation.
•
Anonymous. Respondents tend to be more frank and honest if they can
be assured that their answers are anonymous. Anonymity is more difficult to maintain in face-to-face encounters.
•
Easy to Process. Survey responses are quantitative and easily analyzed
via any number of computer-based statistical analysis packages. Other
forms of data collection tend to use open-ended questions requiring
more complex qualitative analysis procedures.
•
Reduced Time Pressure. Respondents can answer questionnaires at
their leisure and consider their responses. This is not the case in interview situations.
Disadvantages of Written Surveys
•
Impersonal and Structured. Written questions generally do not allow
respondents to clarify their answers. This makes data analysis relatively easy, but does not necessarily allow for a full range of responses.
•
Over-interpretation. Sometimes, respondents read meaning into a
question that is not there. They also occasionally try to guess how the
questioner would like them to answer.
•
Time-consuming. This is particularly true if the survey is sent through
the mail. Time must be allocated both for delivery and response.
•
Low Response Rates. This is common with mail surveys. Many people
simply throw away all mail surveys.
•
Distortion. Respondents who either strongly agree or strongly disagree
tend to respond to surveys in greater numbers than do those who have
moderate feelings.
Overcoming Disadvantages
•
The impersonal, structured nature of surveys can be tempered by giving clear oral instructions when possible and by using a friendly conversational tone in written instructions. The second person is preferred
in writing (e.g., “If you have any questions, please call me—.”). Over-interpretation can be reduced by instructing respondents to give their
(continued)
Program Evaluation
263
(continued)
first impressions and not to attempt to analyze the question.
•
A lack of timeliness can be addressed by administering the survey to
larger groups of people in one place (auditorium, gymnasium, etc.) and
by budgeting enough time to compensate for possible mail delays.
•
Low response rates can be improved by offering respondents a small
token of appreciation for their participation. One must be careful, however, not to bias the responses by “paying” people to complete the surveys. People who are in financial difficulties tend to be
over-represented in surveys that make this error.
•
Distortion can be overcome by conducting follow-up contacts with nonrespondents, thanking them for their participation and encouraging
them to return the survey if they have not yet done so. This may take
the form of postcards or telephone calls.
Conclusion
Surveys are “lagging indicators.” They measure what is in the past. Many things can
happen to change situations between the time respondents complete a survey and
the time the survey is fully analyzed and reported. To be truly proactive, we should
use “leading indicators” to measure what is occurring now and how well we are
achieving what we want in the future.
Source: H.L. Merritt. 2001. “On Measurement II.” Impact: The Official Newsletter of the South Carolina
State Government Improvement Network 10 (2).
increases the likelihood of satisficing, which occurs when a respondent manages to
answer survey questions without expending substantial effort. In other words, respondents that are satisficing do not think very much about the questions—they simply complete the task quickly and mechanically. In more extreme cases, satisficing
emerges in the form of random guessing and an inordinate number of “I don’t know”
responses. Response fatigue and satisficing can have a significant impact on the
quality of an evaluator’s data. Close-ended questions are much less prone to response
fatigue and satisficing compared to open-ended questions. Close-ended questions,
however, are disadvantageous because they limit the amount of information that
can be collected from the respondent; open-ended questions do not confine the respondent to a few answer choices, which allows an evaluator to collect richer data.
In addition to surveys, in-depth interviews and focus groups are an effective means of
collecting empirical data. In-depth interviews involve asking broad, open-ended ques264
CHAPTER 8
tions that elicit lengthy and detailed responses. They differ fundamentally from surveys in that they are less structured, include far fewer respondents, and produce a
tremendous amount of qualitative data. Focus groups are small groups, usually consisting of six to ten participants, the purpose of which is to
establish a dialogue about a specific aspect of a program.
“What gets
Much like in-depth interviews, focus groups use broad
measured gets
themes and open-ended questions to drive the dialogue. Indone.
depth interviews and focus groups often rely on the use of
If you don’t
follow-up (or probing) questions, the purpose of which is to
address inadequate answers, gain additional information, or
measure results,
clarify statements made by interviewees or focus group paryou can’t tell
ticipants (Camino, Zeldin, and Payne-Jackson 1995). Gensuccess from
eral follow-up questions may include:
failure.
•
Could you tell me more about that?
•
Could you give me some examples?
•
I am not quite sure I understand. Could you elaborate some more?
•
What are some of the reasons you feel that way?
If you can’t see
success, you can’t
reward it.
If you can’t reward
success, you’re
probably rewarding
failure.
Field observation is another tool of the program evaluator.
If you can’t see
There are two primary types of field observations: particisuccess, you can’t
pant observations and nonparticipant observations. With
learn from it.
participant observations, evaluators immerse themselves
into the program that is being evaluated, documenting what
If you can’t
they see or hear. Central to this is establishing and mainrecognize failure,
taining relationships with program managers and practiyou can’t correct it.
tioners, as well as participating in some aspect of a
If you can
program’s operations. Conversely, nonparticipant observademonstrate
tions assume complete detachment. The evaluator has no
results, you can
participatory role. For participant observers, the relationwin public
ships that are cultivated and experiences that occur as an
support.”
“insider” serve as sources of intimate knowledge about a
program’s structure and processes. It is possible, however,
DAVID OSBORNE
for a participant observer to become too close. In other
and
TED GAEBLER
words, the relationships and experiences cloud the evaluaArchitects of Reinventing
tor’s judgment and introduce subjective bias. From an anGovernment
thropological perspective, this is referred to as having gone
native. So, while more intimate data can be collected via
participant observations compared to nonparticipant observations, there is the risk
of biasing one’s research by becoming too immersed in the group. Which, then, is ultimately better—participant observations or nonparticipant observations? UnfortuProgram Evaluation
265
nately, there is no clear answer here, as this boils down to a judgment call. Participant observers have the luxury of getting intimately connected with the group, which
may yield a rich amount of data and tremendous insight—although some of this insight may be biased due to personal interactions and actually becoming part of the
group. Nonparticipant observations are safer in terms of limiting bias, but do not
yield as much data as to the personal dynamics of a group.
Social science methods of experimentation are quasi-scientific, as they follow the
logic of natural science experiments. Experimentation (whether scientific or social)
involves manipulation and control in an effort to test causal relationships. In the
medical sciences, the most recognizable example is the drug experiment, whereby
there are two groups: the first group receives the drug (treatment group), while the
second group receives a placebo (control group). The results of the treatment and
control groups are ultimately compared in order to determine the effect of the drug.
In the context of evaluation research, experiments are the primary way of assessing the impact of a program. Consider that we are interested in determining the effect of a peer-tutoring program on student performance in mathematics. As in the
case of the drug experiment, some students would partake in the peer-tutoring program (the treatment group), while other students would not (the control group).
Any differences between the treatment and control groups are considered to be a
function of the peer-tutoring program. A more detailed discussion of experimentation is forthcoming, when we address program impact assessment.
Often overlooked is the fact that evaluative data may already exist. Existing data includes any information that the program formally collects, or any information that
can be obtained through internal program documents such as reports or memoranda.
Before collecting any new data via surveys, interviews, focus groups, observations, or
experiments, an evaluator should first determine what data a program already possesses and review internal documents relevant to the scope of the evaluation.
Conducting Evaluations and
the Importance of Stakeholders
Key to the entire program evaluation process is the relationship and interactions
that the evaluator (or evaluation team) has with a program’s stakeholders. Stakeholders are individuals or groups that have an interest in how a program is performing. In simplest terms, these are people who, in one way or another, care about
a program. The process of identifying stakeholders is sometimes referred to as an
environmental scan. According to Rossi, Lipsey, and Freeman (2004) there are six
primary stakeholders. They include:
266
•
Policymakers
•
Program Sponsors
CHAPTER 8
TABLE 8.1 – OVERVIEW OF METHODS TO COLLECT INFORMATION
Method
Overall purpose
Advantages
Challenges
Questionnaires,
surveys,
checklists
When need to quickly
and/or easily get lots of
information from people in a nonthreatening
way
• Can complete
anonymously
• Inexpensive to administer
• Easy to compare and
analyze
• Administer to many people
• Can get lots of data
• Many sample questionnaires
already exist
• Might not get careful
feedback
• Wording can bias client’s
responses
• Are impersonal
• In surveys, may need
sampling expert
• Doesn’t get full story
Interviews
When want to fully
understand someone’s
impressions or
experiences, or learn
more about their
answers to
questionnaires
• Get full range and depth of
information
• Develops relationship with
client
• Can be flexible with client
• Can take much time
• Can be hard to analyze and
compare
• Can be costly
• Interviewer can bias client’s
responses
Documentation When want impression
review
of how program oper(existing data) ates without interrupting the program; is
from review of applications, finances, memos,
minutes, etc.
• Get comprehensive and historical information
• Doesn’t interrupt program
or client’s routine in program
• Information already exists
• Few biases about information
• Often takes much time
• Info may be incomplete
• Need to be quite clear about
what is being looked for
• Not flexible means to get
data; data restricted to what
already exists
Observation
To gather accurate information about how a
program actually operates, particularly about
processes
• View operations of a program as they are actually occurring
• Can adapt to events as they
occur
• Can be difficult to interpret
seen behaviors
• Can be complex to categorize observations
• Can influence behaviors of
program participants
• Can be expensive
Focus groups
Explore a topic in
depth through group
discussion, e.g., about
reactions to an experience or suggestion, understanding common
complaints, etc.; useful
in evaluation and marketing
• Quickly and reliably get
common impressions
• Can be efficient way to get
much range and depth of information in short time
• Can convey key information
about programs
• Can be hard to analyze responses
• Need good facilitator for
safety and closure
• Difficult to schedule 6–8
people together
Case studies
To fully understand or
depict client’s experiences in a program,
and conduct comprehensive examination
through cross comparison of cases
• Fully depicts client’s experience in program input,
process and results
• Powerful means to portray
program to outsiders
• Usually quite time consuming to collect, organize and
describe
• Represents depth of information, rather than breadth
Source: Carter McNamara, Authenticity Consulting, LLC.
Program Evaluation
267
•
Evaluation Sponsor
•
Program Managers and Practitioners
•
Program Targets
•
Other Related Stakeholders
Policymakers usually represent elected officials or high-level governmental appointees who determine whether a program is created. Program sponsors are responsible for a program’s funding. From a public-sector point of view, there is
significant overlap between policymakers and program sponsors. From a nonprofit
point of view, policymakers would represent those who initiate or develop the program, while the program sponsors would likely represent a philanthropic foundation (or individual) that financially supports the nonprofit program.
“However good our
futures research
may be, we shall
never be able to
escape from the
ultimate dilemma
that all our
knowledge is
about the past,
and all our
decisions are
about the future.”
IAN WILSON
Scenario Planning Expert;
Strategy Consultant
The evaluation sponsor initiates and/or authorizes the
evaluation. The sponsor serves as the conduit between the
evaluator and the program itself. When conducting a program evaluation, the evaluator will likely need access to
managers, key personnel, rank-and-file personnel, internal records and data, and so on. The evaluation sponsor is
responsible for getting the “things” that the evaluator needs
to conduct the evaluation. Therefore, it is important that
the evaluator maintain a good working relationship with
the evaluation sponsor. In some instances, the sponsor
may guide or shape the direction of the evaluation.
Program managers are responsible for directing or supervising some aspect of the program’s day-to-day operations,
while practitioners implement policies and administer a
program’s services.
Program targets are the direct recipients of a program’s
services. For example, if we were evaluating an after-school
tutoring program for secondary school students failing algebra, then the program
targets would be high school students failing algebra.
Finally, related stakeholders refer to all other stakeholders who have an “interest” in
how well a program performs. Returning to our high school tutoring program, some
related stakeholders may include parents, the board of education, the superintendent of schools, and so on. In terms of identifying related stakeholders, an evaluator
should start by asking the evaluation sponsor, program managers, and practitioners
to identify key outside parties that have an interest in the program’s performance.
This will enable the evaluator to generate a list of related stakeholders. The problem
here, however, is that a list generated solely by individuals within the program is
268
CHAPTER 8
likely to be biased, given the tendency to identify related stakeholders that have a
good working relationship with the program. As a result, the information that these
stakeholders provide may be biased in positive way. The evaluator must minimize
bias by ensuring a wide representation of related stakeholders. This can be accomplished through snowball sampling, which is predicated on one stakeholder referring the evaluator to another stakeholder, who, in turn, refers the evaluator to yet
another stakeholder, and so on and so forth. Thus, the original related stakeholder
list “snowballs” into a larger, more representative list.
Obtaining Stakeholder Input
Stakeholder input is obtained, by and large, through in-depth interviews or
through focus group sessions. Key to this process is getting stakeholders to talk
candidly about the program. A good starting point is asking stakeholders: Why do
you think that program x is conducting an evaluation? Other questions include:
What is your perception of program x? What do you feel are the broader goals
and objectives of the program? To what extent do you agree or disagree with
these goals and objectives?
Another approach is to design open-ended questions around a concept known as
SWOT analysis. SWOT stands for strengths, weaknesses, opportunities, and
threats. The evaluator essentially wants to ask: What does the program do well
FIGURE 8.1 – GENERAL LOGIC MODEL
Inputs
S
I
T
U
A
T
I
O
N
O
R
T
I
I
E
S
P
R
What we
invest
Outputs
Activities
Participation
What we do
who we reach
Conduct
Participants
Deliver
services
Develop
products,
cuniculum
resources
Train
Provide
counseling
Assess
Facilitate
Partner
Work with
media
Agencies
Decisionmakers
Staff
Volunteers
Time
Money
Research base
Materials
Equipment
Technology
Partners
Assumptions
Customers
Outcomes – Impact
ShortTerm
What the
short term
results are
Learning
Medium Term
LongTerm
What the
medium term
results are
What the
ultimate
impact(s) is
Action
Awareness
Behavior
Knowledge
Practice
Attitudes
Decisionmaking
Skills
Opinions
Policies
Aspirations
Social action
Conditions
Social
Economic
Civic
Environmental
Motivations
External Factors
Source: University of Wisconsin-Extension, Program Development and Evaluation. 2002.
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html.
Program Evaluation
269
(strengths)? What does the program need to improve upon (weaknesses)? Is there
anything that the program does not do that it should (opportunities)? Is there anything that could potentially be damaging to the program’s future (threats)? The idea
with these broad questions is to get the stakeholders thinking and talking about the
program. As the evaluator, your job is to document what is conveyed and try to
“make sense” of the information that you have gathered.
“Making sense” of interview and focus group data can be daunting because of the
large amount of information one can gather through just a few interviews and focus
group sessions. Interview and focus group transcripts can be dozens or even hundreds of pages long. A general rule of thumb is to look for recurring themes and
stories. When similar patterns or stories begin repeating, an evaluator knows that
he or she has uncovered important information.
Types of Program Evaluations
There are primarily five distinct but interrelated types of program evaluations
(Rossi, Lipsey, and Freeman 2004; Posavac and Carey 2007). They include:
•
Needs Assessment
•
Assessment of Program Theory
•
Assessment of Program Process
•
Program Impact Assessment
•
Program Efficiency Assessment
Needs Assessment
When conducting a needs assessment, an evaluator tries to determine if, and to what
extent, a social condition or problem exists. In other words, is there a “need” for a
program? Conducting a needs assessment is a fundamental step in developing a new
program or restructuring an existing one. A needs assessment consists of four steps:
1. Defining the social condition or problem.
2. Determining the scope of the social condition or problem.
3. Defining the target population.
4. Briefly describing the services that are needed.
When defining the social condition or problem, it is important to be as specific as
possible. For instance, it is not enough to say simply that student underachievement in mathematics is the problem. It is important to stipulate what is meant by
underachievement. A clearer definition would be: Underachievement in mathematics exists when students receive a grade of “D” or lower on two consecutive quarterly grade reports.
270
CHAPTER 8
Program Evaluation Questions
Needs Assessment
•
What is the problem?
•
What is the scope of the problem?
•
Where is the problem localized?
•
Who are the targets (those in need)?
•
What are the characteristics of the targets?
•
What are the target boundaries?
•
What services are needed?
Program Theory
•
What services will be provided?
•
What are the program’s goals and objectives?
•
How will services be delivered?
•
How will the program identify and sustain target participation?
•
What resources are needed, and how will they be organized?
•
What will the program look like in terms of organizational structure?
Program Process
•
How well are program services being delivered?
•
How many targets have been served in a given time period?
•
What proportion of the total number of eligible targets has been served in a
given time period?
•
Do “enough” targets participate?
•
To what extent are the targets satisfied with the program’s services
and/or staff?
•
How effectively do the program’s personnel work with one another or with
inter-related government programs, departments, and/or agencies?
Impact Assessment
•
Are program goals and objectives being achieved?
•
How do the services benefit the targets?
•
Are there any adverse or unintended consequences of the program’s services?
•
Has the social problem improved?
Efficiency Assessment
•
Do program benefits outweigh program costs?
•
Are the costs to achieve program goals and objectives reasonable compared
to similar programs?
Program Evaluation
271
Shifting gears to a more difficult population to identify, consider that the social condition or problem under investigation is homelessness, which is defined as “individuals who lack a fixed, regular, and adequate nighttime residence” (42 USC
[United States Code] 119). In trying to determine the scope of homelessness in a
particular neighborhood, an evaluator may: (1) Survey and/or interview neighborhood residents and business owners, asking these individuals their perceptions regarding the scope of homelessness; (2) interview key informants, such as social
workers, church leaders, advocacy groups, and law enforcement officers; (3) conduct a street-by-street count of homeless individuals throughout a census (or sample) of neighborhood blocks. There are a couple of ways in
which a street-by-street count can be done: there are sim“Evaluation…
ple street counts and street counts with a short interview
[is] careful
component. The nomadic nature of homeless individuals
retrospective
increases the likelihood of double counting. In other words,
assessment of the
a homeless person counted on Block A on Monday may be
merit, worth, and
on Block B on Tuesday and thus could be counted twice.
value of
This makes the street count with an interview component
administration,
the method of choice, assuming the researcher has the reoutput, and
sources and time, as this helps guard against double countoutcome of
ing. Also, counting estimates are further complicated given
government
that homeless individuals may not necessarily be easy to
interventions,
find—or our stereotypical impressions of what a homeless
which is intended
person “looks like” will cause us to undercount people who
to play a role in
do not fit the “profile.” Determining the scope of a homefuture, practical
less problem is particularly difficult, and in instances such
action situations.”
as these, it is important to rely on multiple methods to empirically evaluate whether there is need (or how large the
EVERT VEDUNG
need is).
Author, Public Policy and
Program Evaluation
In determining the scope of a social condition or problem,
we want to understand where the problem is and how large
it is. These determinations can be made by collecting either existing data or by collecting new data through interviews, surveys, and/or focus group sessions that show
that there is “need” for a program to address a specific problem. Consider, for instance, that there are anecdotal reports that the high school students in your locality are “falling behind” their peers at neighboring schools in terms of mathematics
and science achievement. An evaluator conducting a needs assessment would likely
collect data regarding student achievement on state standardized tests, which are
typically administered each academic year, in addition to interviewing and/or surveying key informants. Key informants are individuals who have an intimate knowledge regarding the needs of the targets. In this case, the school’s mathematics and
science teachers and students’ parents would serve as key informants—as they could
attest to how students or children are faring in school.
272
CHAPTER 8
EXERCISE 8.1
Which Are Outcomes?
Nutrition Education Programs
____ 1. Older adults increased the amount of calcium-rich foods they eat.
____ 2. A series of lessons on healthy eating was taught in collaboration with a
drug treatment program.
____ 3. Participants serve more than one kind of vegetable to their families every
day after participating.
____ 4. Participants report savings as a result of wiser spending at the
grocery store.
____ 5. 75 adults have consistently attended all the nutrition workshops.
Food Safety Programs
____ 1. The ServSafe education program is working with 80 percent of all food
service managers in the state.
____ 2. Food poisonings dropped from 677 incidents in 1996 to 225 in 1997.
____ 3. Food service workers reported increased knowledge of safe handling
practices.
____ 4. Food safety skills were taught to state fair food vendors and
restaurant workers.
____ 5. Food safety information in English and Spanish is available on the
University web site.
Small Business Development Programs
____ 1. The small business development network grew from 10 to 13 offices
in two years.
____ 2. Clients generated nearly $40 million in sales.
____ 3. Clients received 12,138 hours of counseling in 1999.
____ 4. 6,349 participants attended 380 seminars and workshops.
____ 5. Clients created and retained 681 jobs.
Youth Citizenship Programs
____ 1. 4-H groups in 45 counties participated in community service projects.
____ 2. Teens volunteered in community service an average of 10 hours over
the year.
____ 3. Teens reported increased ability to identify and help solve a community
need.
____ 4. Teens feel more engaged in and responsible for their community.
____ 5. A local industry contributed $1,500 to the 4-H community service project.
(continued)
Program Evaluation
273
EXERCISE 8.1
(continued)
Quality Assurance
____ 1. Producers decreased their use of medications and made biosecurity improvements to prevent health problems.
____ 2. 724 adults and 1026 youth participated in training sessions.
____ 3. Producers changed management practices because of what they learned.
____ 4. Veterinarians co-taught the sessions.
____ 5. Overall herd health helped to reduce production costs.
Answer key at the end of chapter.
Source: Developing a Logic Model: Teaching and Training Guide, February 29, 2008. Copyright © 2008
by the Board of Regents of the University of Wisconsin System. All rights reserved.
Defining the target population entails developing criteria that determine who is eligible to receive a program’s services. Therefore, if we have determined that there
is an empirically based need for a mathematics and science tutoring program, then
it is important to be clear as to which students will be offered such services. In other
words, benchmarks must be set, which in this context may be based on a student’s
grades, performance on standardized tests, and/or some other criterion.
After clearly defining the problem, determining empirically that there is a need, and
setting target boundaries, the final step of a needs assessment entails describing the
services that are needed; that is, what will the program do to improve the problem?
Assessment of Program Theory
Program theory refers to the conceptual design of a program. It centers on the way
in which a program is supposed to operate in a perfect world. There are three essential elements that comprise a program theory. They include: the program’s impact theory, organizational plan, and service delivery plan. Program impact theory
outlines assumptions regarding the impact of program “x” on social condition “y.”
Consider a high school tutoring program for students failing high school mathematics. In simplest terms, the impact theory for this program would be: students
that attend the tutoring program will improve their mathematics skills. An organizational plan involves resources and personnel, while a service delivery plan underscores how a program’s services will be delivered to its targets. A program’s
impact theory and service utilization plan coincide with a number of program goals
and objectives. Goals and objectives differ somewhat, even though the terms tend
to be used interchangeably. Goals signify a general direction. They are broadly
based. Objectives are narrow, specific, and measurable. Again, consider our high
school mathematics tutoring program. The goal of this program may be to improve
274
CHAPTER 8
FIGURE 8.2 – LOGIC MODEL FOR A HIGH SCHOOL MATHEMATICS TUTORING PROGRAM
INPUTS
• Money
• Teachers
• Teaching Aids
• Meeting space
• Transportation
ACTIVITIES
• Teacher-student
group instruction
• Peer instruction
OUTCOMES
OUTPUTS
• Number of targets
served
• Percentage of
total eligible
targets served
• Number of peer
and teacherstudent sessions
Short-term
• Grades improve
• Standardized test
scores improve
• Enjoyment of
mathematics
cultivated
Long-term
• Graduate from
high school
• Attend college
the mathematics skills of failing students. Objectives may include: increasing standardized test scores x percent in y time period; or improving mathematics course
grades so that a student attains a “C” or better. These objectives are specific and can
be measured simply by gathering data that are routinely collected by schools.
In order to effectively assess a program’s theory, the evaluator must have clear understanding of a program’s impact theory, organizational plan, and service delivery
plan. In terms of theory assessment, an evaluator relies on information provided
from stakeholders, key informants, and research on comparative program theories.
Logic Model
A logic model is a conceptual representation of a program’s theory that includes
information regarding a program’s resources, the services it provides, and the outcomes it hopes to produce. A logic model is organized in terms of program inputs,
activities, outputs, and outcomes. Inputs refer to a program’s resources (such as
money, staff, supplies, buildings, and the like). Activities refer to the services that
a program provides. Outputs refer to the “amount” of services delivered. Outputs
are workload measures. Outcomes refer to the anticipated short-term and longterm impacts of a program.
Assessment of Program Process
Program process assessment centers on the extent to which a program’s services are
reaching its targets. This is typically thought of in terms of coverage and bias. Coverage refers to the level of participation by eligible targets. Undercoverage is a significant concern, and it is typically the result of recruitment and retention problems or
inadequate awareness of a program and its services. Bias assumes that certain subgroups of the target population are participating with either greater or less frequency.
Program process analyses, by and large, focus on the number of targets served during
a specific time period, the proportion of the total number of eligible targets served in
Program Evaluation
275
EXERCISE 8.2
Logic Model Lingo
Place the appropriate number or letter code on each line. Be prepared to explain
your choices.
1 – Input
a – Short-Learning
2 – Activity
b – Medium-Action
3 – Output
c – Long-term–Ultimate benefit
4 – Outcome
0 – Cannot identify
____ a. Teens learned leadership skills.
____ b. A new curriculum was developed.
____ c. Students reported increased confidence in negotiation skills.
____ d. Training programs included seminars and workshops.
____ e. Parents from around the state attended.
____ f. Operators applied their new skills on the job.
____ g. Two agencies partnered to design the program.
____ h. Volunteers provided over 300 hours of support to the project.
____ i. 25 teen mentors were trained.
____ j. Owners learned how to develop a woodland management plan.
____ k. Sessions were held in 10 locations.
____ l. Reported cases of abuse declined.
____ m. Food safety skills were taught to food vendors and restaurant workers.
____ n. Books were distributed to children.
____ o. Parents increased their employment skills.
____ p. Increased numbers of high school students graduate.
____ q. We helped the community assess the needs of families.
____ r. Specialists educated owners about effective production methods.
____ s. Youth serving agencies increased their collaboration.
____ t. Teens established a teen court and hear cases monthly.
____ u. 3 two-day workshops were conducted in each region.
____ v. Newsletters are distributed in three languages.
____ w. 30 listeners per week tune into the radio broadcast.
____ x. Teens learned to counsel other teens on tobacco prevention.
____ y. Town enacted a policy for youth curfew.
____ z. More kids walk to school.
Answer key at the end of chapter.
Source: Adapted from Taylor-Powell, Ellen and Ellen Henert. 2008. Developing a Logic Model: Teaching
and Training Guide, February 29, Handout no. 22. Copyright © 2008 by the Board of Regents of the University of Wisconsin System, http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf.
276
CHAPTER 8
a specific time period, and/or the degree in which the targets are satisfied or dissatisfied with the program’s services and/or staff. It is necessary to establish achievement
levels or benchmarks, and in doing so, it is best to examine comparable programs and
use the professional judgment of the program’s personnel and related stakeholders.
Moreover, through the use of management information systems, organizations are
collecting program process data at regular intervals, which allows managers and evaluators to continuously monitor program service delivery effectiveness.
Program Impact Assessment
The first step to assessing a program’s impact is to identify measurable outcomes. An
outcome is an observable characteristic relating to the potential benefits of a program. In terms of identifying outcomes, examining a program’s logic model will prove
useful. It is important that one not confuse program outcome and program impact.
A program outcome refers to change in a characteristic of the target population, while
impact assumes that a change in a characteristic of the targets is a result of the program and its services. Outcomes must be made measurable, either though existing
data or records, or by collecting new data through surveys, interviews, or observations. If we revisit the logic model presented in Figure 8.1, we see that the short-term
outcomes for a remedial high school mathematics program include improving a student’s mathematical skills and course grades. Each of these outcomes is measurable
through quarterly grade reports and standardized proficiency tests.
FIGURE 8.3 – PROGRAM EVALUATION (ASSESSMENT AND MONITORING)
IN RELATION TO PROGRAM ELEMENTS
Impacts Assessment
Input:
Resources
Implementation
Monitoring
Processes
Outputs
Input:
Demands
Short-run
Impacts
Long-run
Impacts
Performance Monitoring
Productivity Monitoring
Source: R. Hendrick. 1994. “An Information Infrastructure for Innovative Management of Government.” Public
Administration Review 54 (6): 543–50. Reprinted in Holzer, M., and E. Charbonneau. 2008. Public Management & Administration Illustrated. Available online at http://unpan1.un.org/intradoc/groups/public/documents/aspa/unpan029896.pdf.
Program Evaluation
277
The simplicity of the before and after method of impact assessment makes this approach popular among stakeholders. With this method, target outcomes are measured at two points in time: before and after a program’s delivery of services. Consider,
for example, that there are 100 students who participated in the remedial mathematics program as outlined in Figure 8.1. Of those 100 students, 50 percent showed
improvement in terms of their mathematics course grades, and 35 percent showed
improvement on their standardized tests. The important questions here are whether
these observable outcomes are a result of the program itself, or are they merely the
result of some other external factor. In other words, how does the evaluator know
that the program, and not something else, is producing the desired results? When
using the before and after method, there is no way of really knowing. It is through experimentation that we are able to better determine a program’s impact.
Experimentation
There are two distinct experimentation methods. The first is the randomized field
experiment—the gold standard of experimental methods because it follows most
closely the classic scientific laboratory experiment featuring two groups: the treatment group and the control group. The treatment group, in the context of this discussion, would receive a program’s services, while the control group would not. The
key to this experimental technique is the random assignment of individuals into either the treatment or control group. Consider hypothetically that there are 200 students in high school x that are, based upon a needs assessment, eligible for remedial
mathematics instruction. Then, if we were going to conduct a randomized field experiment, 100 randomly chosen students would be assigned to the treatment group
and receive tutoring, while the remaining 100 would receive no tutoring.
The idea of random assignment needs clarification. Random does not mean hap-
EXERCISE 8.3
The Grantmaker: Inner City Youth Organizations Compete
for Funds (Simulation)
Students will read the case study and assume the role of the foundation program
director. As the program director, students will prepare a report for the foundation’s
board, recommending the funding allocations to the selected organizations. Students will compile this board report with concise and specific reasons as to why
they are recommending the funding allocation. All students will present their recommendations to the class (role-playing the board). The class will determine which recommendation to support.
Wood, Miriam, M. 1996. Nonprofit Boards and Leadership—Cases on Governance,
Change, and Board-Staff Dynamics. San Francisco, CA: Jossey-Bass, 62–82.
278
CHAPTER 8
hazard or chaotic. In fact, from a social science perspective, randomness is a mathematical distinction whereby each eligible unit has the same mathematical chance
of selection. In other words, we hypothetically have 200
total students eligible to participate, and a truly random as“Reform is always
signment means that each student has the same 50 percent
a work in progress.
chance of being selected into either the treatment or conSince the world is
trol group. Why does randomness matter? By randomly asa dynamic place
signing individuals, the evaluator can be reasonably certain
and conditions
that individual characteristics, experiences, and biases will
within
schools and
be equally distributed among the two groups. This is recommunities
ferred to as equivalence.
The second experimental technique is the quasi-experiment. The quasi-experiment differs from the randomized
field experiment in that the assignment of individuals into
the treatment and control groups is not done randomly, resulting in a lack of mathematical equivalence. In trying to
achieve near equivalence, it is important to minimize the
differences between the treatment and control groups. This
is done through matching on either an individual or an aggregate basis. Individual matching entails partnering; that
is, two individuals with similar demographic and experiential characteristics (characteristics that are most relevant
to the program’s outcomes) take part in the experiment,
with one participating in the treatment group and the other
in the control group. With aggregate matching, however,
there is no element of partnering. The evaluator tries to ensure that the treatment and control groups are aggregately
similar based upon relevant characteristics. Individual
matching is preferable.
change over time,
there is no
guarantee that a
strategy that works
today will work
equally well
tomorrow…
Evaluation can
help schools
determine how to
adjust the reform
process to meet
selected
objectives.”
U.S. DEPARTMENT OF
EDUCATION
Fitting the Pieces:
Education Reform That
Works, October 1996
After selecting and employing an experimental method, the
evaluator compares the observed outcomes of the treatment group with the observed outcomes of the control group. If there is a statistically significant outcome
difference, then the evaluator can be reasonably sure that the program has had
some measure of impact. Statistical significance means that the results (or outcome
differences of the two groups) are not likely to have occurred because of chance
and/or because of some other external factor.
Program Efficiency Assessment
The final type of program evaluation, efficiency assessment, centers on whether the
money and resources put toward a program are “well spent.” The two primary approaches to determining whether money and resources are well spent include costbenefit analysis and cost-effectiveness analysis. With cost-benefit analysis, program
Program Evaluation
279
costs are compared to tangible and intangible program benefits, which are expressed
in monetary terms. Results are typically expressed in terms of a benefit-cost ratio,
which is equal to the benefits of the program divided by its costs. If the benefit-cost
ratio is greater than one, it can be said that the benefits of a program outweigh its
costs. The challenging and sometimes controversial nature of cost-benefit analysis
stems from trying to place a monetary value on the “intangible” or “public good”
benefits of specific programs. A common way of determining the value of an intangible or public good benefit is through a technique known as contingent valuation.
Contingent valuation entails using surveys to ask people how much they would be
“willing to pay” for an intangible public good (Mitchell and Carson 1989).
From a practical perspective, Small and O’Connor (2006) provide a cogent illustration of a cost-benefit analysis of a Chicago preschool program known as the Child
Parent Center. In determining the costs and benefits of the Child Parent Center,
989 Center participants and 550 nonparticipants were tracked longitudinally from
preschool to the age of 21. Small and O’Connor underscore the outcome differences
of “Johnny” and “Ricky.” Johnny and Ricky are similar in that both come from impoverished, single-parent family households where the mother has little education.
There is also a history of neglect and criminal activity. At age three, Johnny is cared
for at home, while Ricky is enrolled at the Child Parent Center. At the Center, Ricky
is nurtured and stimulated educationally, while parental education is provided for
FIGURE 8.4 – PROGRAM EVALUATION LADDER
Efficiency Assessment
More Complex
Impact Assessment
Process Assessment
Theory Assessment
Needs Assessment
280
CHAPTER 8
More Foundational
Ricky’s mother. In addition, the Center provides home visits and other family services. Johnny is not afforded these services. The costs and benefits relevant to Ricky’s
participation in the Child Parent Center versus Johnny’s
nonparticipation are summarized below.
“What explains
•
At age three, Ricky is enrolled in the Child Parent
Center for two years at a total cost of $10,728.
•
At age nine, Johnny is enrolled in special education classes for four years, the cost of which is
$9,497 per year ($37,988 total).
•
At age 10, Johnny is abused. The costs of child
protective services filing a report and conducting
an investigation amount to $10,861.
•
At age 14, Johnny gets into trouble with the police. Juvenile justice costs amount to $16,690.
•
At age 18, Ricky graduates high school and enrolls in Chicago City College, of which the average taxpayer costs amount to $4,039 per year
($16,156).
this paradox of
successful
programs and
failing students?…
Despite many
reports of success,
we find few
objective
evaluations
conducted by
independent
investigators.”
HERBERT J. WALBERG
and
REBECCA C.
GREENBERG
“The Diogenes Factor,”
•
At adulthood, Johnny is consistently in trouble
Education Week,
April 8, 1998
with the law, the costs of which are $40,195 (the
average criminal justice system costs for career
adult criminals). Ricky graduates from college and gains employment. Small and O’Connor estimate that a college graduate’s lifetime
earnings exceed a high school graduate by $223,303. This translates
into $73,838 in additional tax revenues.
So, here is the bottom line:
Johnny
•
Special education costs = $37,988
•
Child protective service costs = $10,861
•
Juvenile justice costs = $16,690
•
Adult criminal justice costs = $40,195
•
Public benefits = $0 vs. public costs = $105,734
Ricky
•
Child Parent Center costs = $10,728
•
City College taxpayer costs = $16,156
Program Evaluation
281
•
Tax revenue benefits = $78,838
•
Public benefits = $78,838 vs. public costs = $26,488
The difficulty of accurately placing a monetary value on the intangible, public good
benefits of programs is why evaluation studies tend to rely more on cost-effectiveness rather than on cost-benefit analyses. Cost-effectiveness analysis entails estimating the costs of achieving a specific outcome. Consider, for instance, that three
mathematics teachers are hired to administer an after-school mathematics tutoring
program at your local high school. Let us assume that the per-pupil cost is $100
per tutoring session. Let us also assume that the program has resulted in a 20 percent increase in standardized test scores. Consider, however, that a similar program at a neighboring high school achieves the same result at a cost of only $75 per
pupil. This program would be considered more cost-effective because it achieves a
similar result for less money.
Each of the five program evaluations discussed here is interrelated hierarchically
and ordered with regard to complexity. A needs assessment represents the foundation, or the bottom rung of the evaluation ladder. As noted, this is the first step
in either developing a new program or restructuring an existing one. There is no
point in moving forward with a program theory assessment without first establishing that there is a need for a program. Similarly, one would not assess a program’s
processes or service delivery aspects without first establishing that the program’s
theory is sound. Moreover, an impact assessment cannot proceed until we establish
that the program is reaching “enough” targets. Finally, one would not embark on an
efficiency assessment unless it was clear that the program has had some positive impact upon the target population.
Ethical Concerns
Primary ethical concerns deal with the treatment of human subjects. First and foremost, an evaluator must take all necessary measures to protect his or her subjects
from harm. Second, obtaining informed consent is a way of protecting human subjects. When obtaining informed consent, an evaluator must: (1) Explain the nature
of the program evaluation, what it entails, and what the potential implications may
be. (2) Inform human subjects that their participation is completely voluntary. It is
important to stress that human subjects may decline to participate at any time during the evaluation. Subjects may decline to answer any questions or refuse to engage
in any activities that they feel may prove harmful either physically or emotionally.
(3) Inform human subjects that ever effort will be made to maintain the confidentiality of all information collected. In maintaining confidentiality, subjects are typically referred to by number rather than name, and their responses are kept in a
password-protected computer.
In addition to the treatment of human subjects, evaluators must be concerned about
282
CHAPTER 8
potential bias regarding the presentation of their data and findings. In order to conduct an evaluation properly, relationships must be cultivated with a program’s
stakeholders. Given the fact that evaluation research is dependent upon interpersonal relationships, there is always the potential for bias. Consider that evaluator
x encounters program managers and practitioners who are approachable, flexible,
and willing to work with the evaluation team. On the other hand, consider that evaluator y encounters managers and practitioners who are resistant, inflexible, and
suspicious of the entire process. Is it reasonable to assume the potential for bias
given that evaluator x has good working relationship with program stakeholders
while evaluator y does not? Is it fair to assume that evaluators having good working relationships with program stakeholders may be influenced to present their
data and findings in, let us say, a more positive light? This possibility alone underscores the importance of making every effort to keep personal feelings from clouding one’s objectivity.
The program evaluation process is client centered. External evaluators are compensated for providing a service to their clients. There is a financial relationship
between the evaluator and the program, which presupposes the potential for bias.
Given that money can be a tremendous source of influence, it is important that an
evaluator have a clear understanding of what is motivating an evaluation. Evaluation projects that are motivated by personal interest or politics, as opposed to more
sincere reasons, increase the likelihood that money could be used either explicitly
or implicitly as a means of pressuring an evaluator.
Program Evaluation
283
KEY TERMS
Activities
Nonparticipant observations
Before and after method
Open-ended
Bias
Operations research
Close-ended
Organizational plan
Contingent valuation
Outputs
Control group
Participant observations
Cost-benefit analysis
Planning programming budgeting
Cost-effectiveness
Coverage
Efficiency assessment
Empirical
systems (PPBS)
Program evaluation
Program process assessment
Program targets
Equivalence
Program theory assessment
Existing data
Quasi-experiment
Experimentation
Quasi-scientific
Field observation
Random assignment
Focus groups
Impact assessment
Impact theory
In-depth interview
Informed consent
Randomized field experiment
Satisficing
Service delivery plan
Snowball sampling
Inputs
Stakeholders
Key informants
Statistically significant
Logic model
Survey
Management information systems
Survey response fatigue
Matching
SWOT analysis
Near equivalence
Treatment group
Needs assessment
284
CHAPTER 8
REFERENCES
Berk, R.A., and P.H. Rossi. 1998. Thinking About Program Evaluation. Thousand Oaks, CA: Sage Publications.
Camino, L., S. Zeldin, and A. Payne-Jackson. 1995. Basics of Qualitative Interviews and Focus Groups. Washington, DC: Center for Youth Development and
Policy Research, Academy for Educational Development.
Chelimsky, E., and W.R. Shadish. 1997. Evaluation for the 21st Century: A
Handbook. Thousand Oaks, CA: Sage Publications.
deLeon, P. 1989. Advice and Consent: The Development of the Policy Sciences.
Russell Sage Foundation.
Hatry, H.P., R.E. Winnie, and D.M. Fisk. 1973. Practical Program Evaluation for
State and Local Government Officials. Washington, DC: The Urban Institute.
Mitchell, R. C. and R. T. Carson 1989. Using Surveys to Value Public Goods. The
Contingent Valuation Method. Washington DC: Resources for the Future.
Posavac, E.J. and R.G. Carey. 2007. Program Evaluations: Methods and Case
Studies, 7th ed. New Jersey: Prentice Hall.
Preskill, H., and R.T. Torres. 1998. Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage Publications.
Rossi, P.H., W.M. Lipsey, and H.E. Freeman. 2004. Evaluation: A Systematic
Approach, 7th ed. Thousand Oaks, CA: Sage Publications.
Scriven, M. 1991. Evaluation Thesaurus, 4th ed. Newbury Park, CA: Sage Publications.
Shadish, W.R., T.D. Cook, and L.C. Leviton. 1991. Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: Sage Publications.
Singleton, R., and B.C. Straits. 2004. Approaches to Social Research, 4th ed. Oxford, UK: Oxford University Press.
Small, S., and C. O’Connor. 2006. “Cost-Benefit Analysis.” University of Wisconsin–Extension EvalEXchange. http://whatworks.uwex.edu/attachment/whatworks_cost_benefit.pdf.
Taylor-Powell, E., and E. Henert. 2008. Developing a Logic Model: Teaching
and Training Guide. Madison: University of Wisconsin–Extension.
U.S. Code, Title 42, Chapter 119, Subchapter 1, § 11302. General definition of
homeless individual.
Program Evaluation
285
U.S. General Accounting Office (GAO), General Government Division. 1998. Program Evaluation: Agencies Challenged by New Demand for Information on
Program Results. Report to the U.S. Committee on Government Affairs, U.S.
Senate. April.
Wholey, J.S., H.A. Hatry, and K.E. Newcomer. 2004. Handbook of Practical Program Evaluation, 2d ed. Hoboken, NJ: Jossey-Bass.
SUPPLEMENTARY READINGS
Taylor-Powell, E., and C. Hermann. 2000. Collecting Evaluation Data: Surveys.
University of Wisconsin–Extension Cooperative Extension.
Taylor-Powell, E., and S. Steel. 1996. Collecting Evaluation Data: An Overview
of Sources and Methods. University of Wisconsin–Extension Cooperative Extension, Program Development and Evaluation Unit. http://learningstore.
uwex.edu/assets/pdfs/G3658–4.pdf
W.K. Kellogg Foundation. 1998. W.K. Kellogg Foundation Evaluation Handbook. Battle Creek, MI.
ELECTRONIC RESOURCES
American Evaluation Association: http://www.eval.org/
Australasian Evaluation Society: http://www.aes.asn.au/
Canadian Evaluation Society: http://www.evaluationcanada.ca/
Centers for Disease Control (CDC) Evaluation Working Group:
http://www.cdc.gov/eval/
The Centre for Program Evaluation (CPE) at the University of Melbourne Graduate School of Education, Melbourne, Australia:
http://www.edfac.unimelb.edu.au/cpe/
The Evaluation Center at Western Michigan University:
http://www.wmich.edu/evalctr/
The Evaluation Exchange at Harvard University: http://www.hfrp.org/evaluation/the-evaluation-exchange
Evaluation Handbook, Evaluation Assistance Center–Western Region, New Mexico Highlands University:
http://www.ncela.gwu.edu/files/rcd/BE020502/Evaluation_Handbook.pdf
Free Management Library, Basic Guide to Program Evaluation:
286
CHAPTER 8
http://www.managementhelp.org/evaluatn/fnl_eval.htm
Online Evaluation Resource Library (supported by the National Science Foundation [NSF]): http://oerl.sri.com/
Oregon Program Evaluators Network (OPEN): http://www.oregoneval.org/
Program Development and Evaluation, University of Wisconsin–Extension:
http://www.uwex.edu/ces/pdande/
U.S. Department of Justice, Office of Justice Programs, Bureau of Justice Assistance, Center for Program Evaluation and Performance Measurement:
http://www.ojp.usdoj.gov/BJA/evaluation/
U.S. Environmental Protection Agency, Program Evaluation Glossary:
http://www.epa.gov/evaluate/glossary.htm
The World Wide Evaluation Information Gateway: http://www.policy-evaluation.org/
EXERCISES ANSWER KEY
Exercise 8.1
Nutrition: 1, 3, 4
Food safety: 2, 3
Small business: 2, 5
Youth Citizenship: 3, 4
Quality Assurance: 1, 3, 5
Exercise 8.2
4a
2
4a
2
2
4b
1
1
3
a.
b.
c.
d.
e.
f.
g.
h.
i.
4a
2
4c
2
2
4a
4c
2
2
j.
k.
l.
m.
n.
o.
p.
q.
r.
4b
4b
3
2
3
4a
4b
4c
s.
t.
u.
v.
w.
x.
y.
z.
Note: Several of the above are debatable given the program goal that is assumed. Participants should
be able to explain and defend their choices. To test outcomes, ask “so what?”
Program Evaluation
287

Purchase answer to see full
attachment