Assignment: Health Communication Interventions

Permalink: https://collepals.com/assignment-evalu…on-interventions/

Assignment: Evaluating Outcomes of Health Communication Interventions

Chapter 14

Overview/Objectives

Why, What and How We Measure
Evaluation Models and Tips
Case Studies
Linking Outcomes to Strategic Communication Programs

Why, What and How We Measure

The Language of Evaluation

Evaluation

Program Assessment

Metrics

Return on Investment (ROI)

Outcomes

Impact

Social Change Indicators and Social Impact

Behavioral Impact

11/14/2007

Social Change and Behavioral Indicators

Why We Measure

Focus staff/partners/intended audiences on shared goals
Clarify program purposes, goals, and objectives
Identify and compare effective practices
Improve service delivery
Adjust program in progress by refining strategies/messages
Assess cost effectiveness of program
Assess program results/ROI
In public health communication, ROI = health or social behavior that leads to improved public health outcomes
Determine program reproducibility and sustainability/opportunities for scaling-up phase
Communicate results to key stakeholders/ audiences
Implement lessons learned from new models/strategies in future interventions
Compete for economic and human resources

11/14/2007

Drawbacks of Evaluation

Cost
Time
Chance of measuring wrong variables and indicators
Questionable accuracy for programs with limited scope, reach and duration
Potential bias in evaluation method or tool
Hard to do if not planned ahead
Results that may be affected by independent influences on key audiences and program’s outcomes

11/14/2007

Types of Evaluation

Formative
Before program development/implementation
Pre-testing
Process and Progress
Compares program’s implementation with original plan
Program reach
Progress monitoring/intermediate results
Summative (outcome or impact evaluation in different models)
Program’s impact in relation to program/outcome objectives (behavioral, social, organizational) and long-term communication objectives

Each type corresponds to a specific

research and monitoring phase

11/14/2007

What’s Being Measured?

Process Evaluation/Intermediate Results

Message delivery: depth/extent, tone, frequency/ repetition, reach
Message retention: at what time intervals after the event?
Participation in an event: influential advocates/ endorsers, target population
Distribution of materials; number of follow up requests
Endorsement from third-party/KOLs
Ability to create alliances

11/14/2007

What’s Being Measured?

Process Evaluation/Intermediate Results

Media reach: circulation, impressions
Inquiry for information: toll-free #, website visits
Ability to neutralize opposition
Short-term or intermediate changes in knowledge/ attitudes, skills, community participation levels, etc. (progress)

Summative Evaluation (also Outcome or Impact Evaluation)

Changes in attitude, knowledge, social norms, policies, health or social behavior
Refers to outcome and communication objectives

11/14/2007

Summative Evaluation

“Evaluates program’s efficacy in relation to outcome and communication objectives initially established by the program” (R. Schiavo, 2007, p. 326)
Recognizes current emphasis on strategic behavior communications/behavioral impact
Measures extent to which change occurs
“In health communication programs, the primary objective is usually a health-related behavior.”

J. Bertrand, John Hopkins Center for Health

Communication, The Drum Beat, CI, 2005

11/14/2007

Health Outcomes as a Complex and Multidimensional Construct

Evaluating Behavioral and Social Impact

Assessing changes in individual, community, or social behavior requires:

Intensive effort in concentrated geographic area/time period
Front/back market research against
Tracking surveys
Other qualitative/quantitative measurements
Significant financial commitment
Only long-term efforts can generate sustainable behavioral and social results!

11/14/2007

How We Measure

Qualitative Methods

In-depth interviews with members of intended audiences, program participants, or other key stakeholders before and after the program

Focus groups

Completion of evaluation forms after specific activities (for process evaluation only)

Evidence of endorsement, such as letters of support or actual program participation from key influentials (for formative and process evaluation only)

Panel studies, that is, pre- and post-intervention studies, which involve the same panel of key stakeholders or representatives of key groups and relevant communities

Community dialogue, public consultations, and other participatory research methods

Quantitative Methods

Pre- and post-surveys—to be implemented in-person, online, by telephone or other methods
Tracking surveys, which collect evaluation data at different time intervals
Control and intervention groups in which groups are either randomized or selected in a way that one group will be exposed to the health communication intervention and the other will not
Online surveys and opinion polls – to take the pulse of progress throughout program implementation and also to evaluate summative results

11/14/2007

Current Trends in Evaluation

Increasing number of publications/models on this topic
Reflect need for accountability/new parameters established by major donors
Extent of evaluation efforts depending on:
Size of organization
Optimal balance between evaluation needs, cost, other organizational or program-specific factors
Still, major communication programs with national scope should include all types of evaluation
Mixed methodologies preferred to assess “what” and “why”/link intervention to outcomes

11/14/2007

Current Trends in Evaluation

Behavioral indicators need to be strongly linked to overall program goal
“Increased immunization rates predict decreased child mortality. Increasing number of girls in school often cited as predictor of economic progress.”

Rockefeller Foundation Communication and

Social Change Network, CI

Still, fewer than 1/3 of campaigns express goals/ behavioral objectives in measurable terms

The Health Communication Unit, University

of Toronto

11/14/2007

Current Trends in Evaluation

Participatory evaluation
Mutually agreed upon evaluation parameters,
Include communication team, partners, representatives of key groups, communities, donors, other stakeholders
Logic models/other frameworks used to organize different program elements and related outcomes
Comprehensive evaluation reports
Increased focus on quantitative results/ outcomes (e.g., percentage of intended population where change occurred)
Measurement vis-à-vis outcome objectives (behavioral, social and organizational)
Emphasis on behavioral impact at multiple levels of society = social impact

11/14/2007

Looking at Evaluation Models and Tips

11/14/2007

Assessing Changes in Behavioral Intentions and Behavior

Theory of Reasoned Action (Ajzen and Fishbein, 1980)

Behavioral beliefs

Normative beliefs

Subjective norms

Behavioral

Intentions

Attitudes toward

behavior

Behavior

11/14/2007

Hicdarm and Behavior Adoption

First, we Hear about the new behavior

then, we become Informed about it

and later Convinced that it is worthwhile.

In time,

we make the Decision to do something about our

conviction

and later we take Action on the new behavior.

We next await Re-confirmation that our action was a

good one

and if all is well,

we Maintain the behavior!

Source: Hosein, E (1979) Communication Training Manual for IEC

(Information-Education-Communication) Officers of IPPF member

affiliates in the Caribbean. Presentation at 1979 International

Planned Parenthood Federation meeting, New York, NY. Used by

permission.

11/14/2007

Logic Model and Evaluation Design

Program’s Name:…..
Long-term Problem to Be Addressed:….
Long-term Objectives to be Achieved…

Source: U.S. DHHS Office of Minority Health, 2010

Logic Model and Evaluation Design

Source: Adapted model from U.S. DHHS Office of Minority Health, 2010

Contributing Factors	Strategies and Tactics	Outcomes and Impact
From Your Formative Research/Evaluation	How the program addresses the health issue/attempts to achieve outcome objectives (e.g. via community engagement, celebrity engagement, etc.) What kinds of activities the program implemented (tactics)? For example, workshops, PSAs, etc.	Short-term/intermediate /outcomes Summative/Behavioral/Long-term outcomes

“Essentials for Excellence” Model

Source: UNICEF, Essentials for Excellence, Nov. 2006, p. 3

Level	Research, Monitoring, Evaluation
Program Goal	Impact Evaluation
Program Objectives	Outcome Evaluation
Behavioral Objectives	Behavioral Monitoring, Significant Change Monitoring, Evaluation
Activities (inputs, outputs)	Implementation, Monitoring, Pre-testing

11/14/2007

Introducing Mutually Agreed Upon Evaluation Parameters

Set specific quantitative/qualitative goals per project prior to implementation

Written assessment upon conclusion

To date, primarily used for process evaluation/ assessment of intermediate changes/progress

Can be used as model to secure consensus on summative evaluation parameters/behavioral indicators

11/14/2007

Process and Progress Evaluation: Establishing Mutually Agreed Parameters. Pre-Implementation Sample Chart

Activity

Healthcare provider workshop series

Radio PSA

Anticipated Quantitative

and Qualitative Results

Reach audience of 30 – 100 depending on venue

80% positive response to evaluations
70-80% speaker endorsement of key messages
Message retention assessed via pre- and post-test market research with target healthcare providers

20-25 million radio impressions over one year period

10,000-15,000 requests for additional information

11/14/2007

Process and Progress Evaluation: Establishing Mutually Agreed Parameters. Pre-implementation Sample

Activity

2. School-based parental outreach

Anticipated Quantitative

and Qualitative Results

Program endorsement by 5-7 schools per city

Message retention assessed using pre- and post-test research with parents in target area
Increase by 15-30% parental awareness and suspicion of disease/ (assessed via pre-/post survey with healthcare providers attending families in target regions)
5,000-10,000 requests for materials fulfilled in each city after initial distribution

No Matter What Model You Decide to Use Always Consider Evaluation Parameters that Are Inclusive of Vulnerable and Underserved Populations

“Were vulnerable and underserved populations included?
Did they participate in intervention design and evaluation?
Do results among these groups compare to results within the rest of the population?
If these groups were not included, are results relevant?
What issues need to be addressed to ensure inclusion?
What kinds of organizations or stakeholders should be involved to increase cultural competence, use and access?”

Developing Evaluation Plans that Suit Your Budget/Program/Organizational Needs

Measurement of outcome indicators (behavioral, social, organizational) a must
Still, small organizations with limited budgets may consider including only 1 or 2 phases of evaluation
All 3 phases should be included for large and/or national programs
In presence of budget/organizational constraints small scale evaluation preferable to poor large scale evaluation
Organizational culture/key stakeholders agreement on parameters critical to successful evaluation
Evaluation budget/plan should be part of original program planning

References: R. Schiavo, 2007, 2013; NCI/NHI, 2002

Developing Evaluation Plans that Suit Your Budget/Program/Organizational Needs

Customize evaluation/monitoring tools/methods to meet culturally diverse needs
Consider that some groups may:
Feel uncomfortable providing negative answers in group settings
Have limited reading skills or lack familiarity with questionnaires
Be uneasy speaking with a person of different culture or participate in telephone survey
Develop and use evaluation reports to:
Broadcast results/share lessons learned
Refine program/focus on key strengths
Gain visibility for your program/organization
Advocate for health communication’s worth
Request additional funds

References R. Schiavo, 2007, 2013; NCI/NHI, 2002

11/14/2007

Developing Evaluation Plans that Suit Your Budget/Program/Organizational Needs

Cost-Effectiveness Tips

Consider using:
General questions from previous evaluation efforts/customize them for specific program
Involve audience members/program partners in data collection
Evaluate cost-effectiveness of different monitoring services/ approaches/team structures
Still, focus on/aim to statistical significance of key results

References: R. Schiavo, 2007, 2013; NCI/NHI, 2002

11/14/2007

Monitoring: An Essential Element of Program Implementation and Evaluation

Areas of Monitoring

News and trends on disease area/health issue, intended audiences, policies
Ongoing feedback by relevant communities, key groups and stakeholders on program content/vehicles
Process and progress indicators for all program activities/ materials
Collection of summative evaluation data (monitoring behavioral, social and organizational indicators)

11/14/2007

What is a Monitoring System?

An organized and interrelated set of activities, tools and communities of people that aims at monitoring program and outcome indicators on the basis of a consistent logical framework, work plan and timetable – Schiavo, R.
Relies on in-depth understanding of:
Key questions to be addressed by monitoring efforts
Adequate allocation of human and economic resources
Selection of adequate tools and activities
Advance planning

References: Schiavo, R. 2007, 2013; UNICEF, 2006

11/14/2007

Establishing Sound Monitoring Systems/Teams

Dedicated staff

members

Community/audience feedback process

Internet/media

analysis/other

special tools/ services

Formal data collection (process, progress, summative)

11/14/2007

A Quick Look to the Evaluation Look to the Evaluation of New Media-Based Interventions

How is Measurement Being Done? Evaluation of New Media-Based Interventions

Most evaluation efforts still focus on:
Audience exposure/tracking program reach (e.g. number of new or recurring visitors, number of hits, page views, downloads, etc.)
Increased message dissemination/outreach

Increased emphasis on evaluation
Still, most documented case studies on text messaging/mobile use only
Several organizations looking at evaluating awareness and behavioral results of existing/on-going programs
Avatars being used to create proof of concept on potential behavioral impact of new media (e.g. CDC and WhyVille)

“Verb [a CDC-campaign] was able to significantly prolong the life of its campaign after the campaign budget ended, as the ads posted on YouTube and its website continued to receive many downloads and views after the campaign officially ended” – Abroms, Schiavo and Lefebvre, 2008; Hunman, 2008

How is Measurement Being Done?
Evaluation of New Media-Based Interventions

Whether online of offline tracking surveys still the best way to assess changes in awareness, knowledge, skills and most importantly behavior

Baseline data should be collected online or offline prior to the introduction of new tools/initiatives

Online efficacy testing methodologies increasingly available to asses internet- and new media-based interventions; still, offline survey/panel studies provide valuable alternatives

11/14/2007

How is Measurement Being Done?
Evaluation of New Media-Based Interventions

Web-based panels increasingly used to evaluate effectiveness of new media-based interventions
Knowledge Networks – www.knowledgenetworks.com (offer a probability-based sample; geographical samples; laptops provided to users who have not access to technology)
Harris Interactive – www.harrisinteractive.com (offer large samples of individuals “who have opted in their panel” as part of Harris Poll Online Members)
E-Rewards (www.erewards.com) and MySurvey (www.mysurvey.com) – opinion panels used primarily for commercial marketing purposes
Option to create own sample by using Internet-based tools

(e.g., Survey Monkey – www.surveymonkey.com )

Avatars/virtual worlds may be useful tools to measure changes in message relevance/retention/awareness/online efficacy (e.g. CDC and WhyVille)

11/14/2007

Evaluation of New Media-Based Interventions

Assessing Intermediate Steps to Behavioral Change by Using Virtual World’s Tools

Figure 14.3 – Flu Vaccine Campaign 2009 in WhyVille

Evaluation of New Media-Based Interventions
Key Trends and Needs

Increased emphasis on evaluation
Most case studies on text messaging/mobile use
Several theories to model/summarize new media’s influence
Need to create:
Proof of concept on:
Potential behavioral impact of new media and/or impact on intermediate steps towards behavior change
Most evidence still as applied to chronic disease management and treatment/copying skills
Link online participation, awareness and behavioral intentions to actual offline behavior
Optimize use of new media; clarify expectations for each specific new media tool

Linking Outcomes to Strategic Communication Programs

11/14/2007

“Please remember that communications programs take place in a real-world setting, where there are many other influences on your target audiences. In many cases, it may be impossible to isolate the effect of your program on behavioral change from other concomitant activities and/or influences.”

NIH/NCI – Pink Book, 2002

11/14/2007

Linking Outcomes to Strategic Communication Programs

Key Factors

Consistent model for program development, implementation and evaluation1
Well-defined and measurable outcome and communication objectives1
Adequate mix of quantifiable parameters and qualitative analyses1
Intermediate steps to assess progress and process-related results/early changes1
Sound monitoring system1,2/dedicated monitoring team1
Collect data at different stages (baseline, during, end of program)1,2

References: 1) Schiavo,, 2007; 2) UNICEF, Essentials for Excellence, Nov. 2006

11/14/2007

Linking Outcomes to Strategic Communication Programs

Refrain from attempting to link specific activities/materials to long-term changes in knowledge, attitudes, skills or behavior1
“Change does not often occur as a result of just one specific activity3”
Contribution analysis2
Consider plausible alternatives/if existent, factor them in evaluation report2
Adequate funds/cost-effective solutions/early budget allocation1

Sources: 1) Schiavo, 2007; 2) UNICEF, Nov. 2006; 3) NCI/NIH, Making Health Communication, 2002.

11/14/2007

Linking Outcomes to Strategic Communication Programs

“Outcomes appeared at appropriate time after efforts began
Outcomes faded when efforts stopped
Only those outcomes appeared that you should have affected
Outcomes appeared only where/when program was active
Biggest outcomes appeared where/when you did the most
No plausible alternatives for outcomes, or if there are, you have recognized them/ objectively accounted for them in evaluation conclusions”

Source: UNICEF, Essentials for Excellence, Nov. 2006, p. 15

Try it now!

Get 20% Discount on This Paper

Type of paper needed:

Pages:

Academic level:

We'll send you the first draft for approval by at

Total price:

$0.00

How it works?

Follow these simple steps to get your paper done

Place your order

Fill in the order form and provide all details of your assignment.

Proceed with the payment

Choose the payment system that suits you most.

Receive the final file

Once your paper is ready, we will email it to you.

Our Services

Ace Writing Center has stood as the world’s leading custom essay writing services providers. Once you enter all the details in the order form under the place order button, the rest is up to us.

Essays

Essay Writing Services

At Ace Writing Center, Nowadays, students normally have extremely busy schedules. You will note that some of them have to take on some evening or weekend jobs in order to get some income that can help them to sustain in college or in the university. This can deny them a chance to write all the essays given. Others usually get bombarded with a lot of work by their lecturers. This can still delay such students from working on all their essays. However, some of them usually try to work on all these essays but end up delivering their work late. This can prevent them from graduating since most lecturers are strict on deadlines. If you want to write a business essay, the wise way is to hire an outstanding essay writing service like us, so that you can get the best results. If you are keen, you will note that many companies usually overcharge their customers. Some of them are there only to make money. And in reality, they really don’t care to build a long term commitment with students. You should not choose such companies. You should take your time and choose a reliable company to work with. Ace Writing Center is the ultimate solution for you. We have been offering our writing service for more than 7 years. This is a clear indication that you will get quality essay writing service. We have a wide range of writers who can work on any business essay that you might have. We believe in doing extensive research so that we can provide quality work to all our clients. .

Admissions

Admission and Business Papers

Have you ever had to write an admission essay for college? The majority of students face the same issues when applying to a university or college and many in such situations decide they need professional help to cope with this matter. They get in a situation when the deadline keeps coming closer but lack motivation to start because they are just not sure if their writing skills are strong enough. We have a solution for you! Ace Writing Center is the best admission essay writing service with a large professional team and years of experience in providing high-quality papers to students of all levels and faculties. The mission of our team is to help students make their dreams of entering a good college come true and that’s what we offer!.

Editing

Editing and Proofreading

Sometimes all the words for your paper just flow out of your mind and into your fingers. You type quickly at your keyboard and there they are, your beautiful words right there on the screen. But you have no idea how to polish it up. You may be wishing there was a paper writing service that offered this type of writing service. Look no more! Here at Ace Writing Center, we offer you an editing and proofreading option that you can't find anywhere else..

Coursework

College Essay Writing

In case you are familiar Ace Writing Center, you know the way to distinguish a better company from a cheap one exactly. First of all, poor service website does not have a sufficient support. We think support team is an essential part of success; it has to answer all clients’ questions and be a connecting link between clients and their writers. On our web-service you will get answers about anything you need and your writer will receive all your instructions, assignments and requirements exactly and swiftly. A writing service that we run has got a flexible pricing system that will save you from senseless wastes and many bonus systems that let you sparing money for something important for you.

See all services