Click here for Primer Menu

Why Communications Evaluation?

Communications and public education campaigns can't be called a success unless they're evaluated. Sounds obvious, right? But too often, even organizations with large budgets - like the government - don't design evaluations INTO their communications efforts.

Additional Evaluation Resources Good campaigns require good evaluations – and both require planning up front. When you build evaluation into your campaigns, you’ll be forced to be more strategic about your communications efforts. In short, you’ll craft a better plan. We can’t cover all the issues you’ll need to know to design an evaluation for your particular program, but we can recommend a number of resources to help you pick the tools to help you get started: Environmental Education Theory Models: Visit the CoEvolution Institute to learn more about environmental education theory and evaluation. They have an excellent resource book called Measuring Results that you may want to order. The Logic or TOP Model: Learn more about this popular program and planning model at the Kellogg Foundation. The site features a remarkable collection of evaluation tools and support, especially their Logic Model Development Guide. Communications Theory Overview: Communications for Social Good by Susan Bales and Franklin Gilliam was written to introduce foundations to the latest theories regarding communications and evaluation. Although intended for funders, this paper has some wonderful information for non-profit program planners as well.

Those of us with limited resources can ensure our members, donors, and granted funds are used most efficiently by making evaluation a part of every communications program we pursue.

How Do I Get Started?

Whether you know it or not, you may already be conducting some form of evaluation on your communications or education work. If your group solicits donations through an annual mailing campaign, each reply you receive helps you gauge the success of your efforts - this is a kind of evaluation. Likewise, your organization's Web site service may allow you to count how many visitors you receive - this too, is a simple kind of evaluation.

However, an effective evaluation won't just give you raw data suggesting "success" or "failure." Instead, an effective evaluation will help you pin point why certain strategies worked or didn't work. And, after receiving a tremendous response on a request for donations, who wouldn't want to know exactly why that campaign worked better than others? Even when things don't go as you'd hoped, evaluations can help you avoid repeating mistakes that lead to disappointment.

Overarching Evaluation Principles

The Communications Consortium Media Center has provided the following recommendations on communications evaluation. These "overarching principles" are based on their research and the real experiences of non-profit organizations engaged in public education and communications campaigns throughout the United States. These principles should guide all your program planning, implementation, and evaluation efforts:

1. There's no "right" or "wrong" way to evaluate communications campaigns. There should be both recognition and acceptance of the fact that different evaluation needs and capabilities require different evaluation designs (and that causation is not always the most important question). The evaluation's design, focus, and methods should fit the information needs and available resources of stakeholders in the communications effort.
2. Assessing whether a campaign caused its intended impact is often important, and that's the activity funders tend to focus on. But evaluation for purposes of learning and continuous improvement is also important. At the same time, all sides should recognize that leverage to convince sponsors to invest in campaigns will be enhanced by evaluations that assess causation (which often require higher evaluation budgets).
3. Evaluations, like communications campaigns, need to identify up front their purpose and intended audiences. For example, is the evaluation intended to measure the impact of a campaign? Or is it to provide feedback so the campaign can learn over time from experience? Also, is the target audience for the evaluation the sponsoring foundation, the non-profit(s) implementing the campaign, or both?
4. It's best to design the evaluation early and in conjunction with the campaign. This will maximize opportunities to use the evaluation for both learning and impact assessment.
5. Campaign staff members should participate whenever possible in the evaluation's design as well as its implementation. Campaigners and evaluators both need to understand the existing challenges and opportunities. For example, is the campaign seeking to change public opinion and then induce action by policy-makers? Or is the campaign building upon existing favorable opinion and then mobilizing people to a particular action? Obviously, campaigns to change public opinion are more difficult from the start.
6. Evaluation should push for methodological rigor and innovation whenever possible. It should also acknowledge that more than one evaluation approach (i.e., more than pre- and post-campaign polls) can capture useful information.

7. Evaluating Great Lakes Forever Evaluating Biodiversity Project's Great Lakes Forever Wisconsin pilot program was an arduous affair. With more than three years of background planning and research, it was important to understand the context that framed the public phase of Great Lakes Forever. A tremendous amount of data - literally several thousand pages - were distilled to create a final evaluation report. In brief, here are the milestones: The Formative Evaluation: This was the initial groundwork for creating our communications strategy. We took inventory of the status quo, did background research (the public opinion poll), and sought to find our particular "niche." Strategy Development: We defined our approach, our goals and outcomes, our target audiences, messages, messengers and pathways. During this process it was useful for us to begin using the Logic model to evaluate our progress on an outcome-oriented basis. As the campaign strategy developed, we sought measurable outcomes for the campaign timeline. Partner Recruitment: Although not relevant to every campaign, Great Lakes Forever was designed to coordinate communications in the region. The involvement of partners and their sense of ownership for the program was crucial to Biodiversity Project's strategy. Activities and Product Outcomes: With partners in place, the campaign could proceed on schedule to produce communications products for our target audiences. These included: Media Outreach, Advertising, Point Of Experience, Targeted Mailings, Bioblitz Events, Great Lakes Forever Web Site and Email Newsletter. Data collection and evaluation measures were included in every aspect of these program activities. Cumulative Impact; Big Lessons Learned: Throughout the latter half of the program some clear successes were apparent, some lessons were learned and there remained some unresolved issues. All of these were then wrapped into the final report using all the measures outlined in the original Logic model designed during the campaign planning phase. Adaptive Management: The real purpose of any evaluation is to improve the efficiency and effectiveness of your organization's communications work. Biodiversity Project is currently working to apply some of the lessons learned from their evaluation to Great Lakes Forever and all of their future program plans.

   Different evaluation designs have different interpretive boundaries. It's important to understand those boundaries and avoid the temptation to make broad claims of success based on limited data or designs that do not warrant such claims.
8. It's important to be realistic about impact. In commercial marketing campaigns, attitude improvements of one-tenth of one percent are deemed important because they can represent millions of dollars. But sometimes funders of communications campaigns want to see attitudinal shifts of 10 to 30 percent. In response, non-profits sometimes make promises to funders they can't possibly fulfill.
9. Sometimes simple things like having a good press list or establishing ongoing professional relationships with key reporters are the most significant measures of success, especially for locality-specific or small-budget efforts.
10. "Values" are important to both campaigns and their evaluation. Typically, non-profit communications efforts put forth information to achieve either behavioral or societal change. However, widely held and deeply entrenched values can often trump useful information (e.g., values about the meaning of "family," "community," "independence" or "self-sufficiency"). Successful communications campaigns must acknowledge the "values vs. information" dichotomy, and evaluation must take this into account when judging impact.
11. Evaluation should be based on sound (and where possible research-based) theory for predicting how the campaign will achieve social change.
12. Evaluation can respond to hard-to-answer questions about the value and effectiveness of communication campaigns (e.g., whether information alone can lead to behavior change or whether attention to the social and policy context is also a necessary ingredient; and whether media advocacy can contribute).

These principals, the "Communication-Related Theories and Concepts" described in Appendix A and the "Field Guide to Evaluation Types" come from the final Working Paper in a series of papers prepared for the Communications Consortium Media Center's (CCMC) Media Evaluation Project. It summarizes the main findings of the first four Working Papers in the series, and offers guidelines for the evaluation of non-profit communications efforts. To read the full paper and learn more about CCMC, visit their Web site.

Communication-Related Theories and Concepts

Social scientists have developed a number of theories about how social change happens. Being aware of and explicit about which theoretical model(s) you're using to construct your campaign can be invaluable for design and evaluation. See Appendix D in the print version of the primer for a brief list of social change theories.

Field Guide to Evaluation Types

According to the CCMC, there are four possible types or areas of focus that can be used for an evaluation. Depending on resources available and information needs, the evaluation may include one, two, three, or all four.

1. Formative evaluation usually takes place at a campaign's front end and collects information to help shape the campaign's activities. For a public will campaign, this might involve measuring issue awareness through public polling or testing of messages and materials in focus groups, either formally or informally. Sometimes a "meta-survey" or summary analysis of existing polling data can serve the same purpose.
2. Process evaluation examines the campaign's implementation, or the way activities roll out. Process evaluation might count the number of materials distributed, the development and dissemination of messages and materials, and the number of efforts to work with the media.
3. Outcome evaluation examines the campaign's outcomes, which usually means its effects on its target audience(s). Evaluators often use surveys, polling, or more qualitative means of gathering this type of information.
4. Impact evaluation examines effects at the community, state, national or international level, or a campaign's long-term outcomes (including the effects of behavior or policy change). Impact evaluation can also attempt to determine causation - whether the campaign caused the observed impact(s). This type of focus typically requires more rigorous evaluation design methodology, such as experimental or quasi-experimental techniques.

Evaluation Focus	Purpose	Example Questions
1) Formative	Assesses the strengths and weaknesses of campaign materials and strategies before or during the campaign’s implementation.	*How does the campaign’s target audience think about the issue? *What messages work with what audiences? *Who are the best messengers?
2) Process	Measures effort and the direct outputs of campaigns – what and how much was accomplished. Examines the campaign’s implementation and how the activities involved are working.	*How many materials have been put out? *What has been the campaign’s reach? *How many people have been reached?
3) Outcome	Measures effect and changes that result from the campaign. Assesses outcomes in the target populations or communities that come about as a result of grantee strategies and activities. Also measures policy changes.	*Has there been any affective change (beliefs, attitudes, social norms)? *Has there been any behavior change? *Have any policies changed?
4) Impact	Measures community-level change or longer-term results achieved as a result of the campaign’s aggregate effects on individuals’ behavior, and the behavior’s sustainability. Attempts to determine whether the campaign caused the effects.	*Has the behavior resulted in its intended outcomes (e.g. lower cancer rates, less violence in schools)? *Has there been any systems-level change?

Click here for Primer Menu