A Step-by-Step Approach for Evaluating Your Arts Program's Outcome

Follow these steps to conduct a well-planned evaluation of your arts-based program:

1. Prepare your planning model
2. Develop your evaluation questions
3. Assess whether an evaluation is feasible
4. Plan your evaluation
5. Identify data collection instruments
6. Collect data
7. Analyze data
8. Present findings and disseminate lessons learned
9. Use evaluation findings
10. Think about conducting a follow-up

For more information, you can also download the full Evaluation chapter as a PDF and read pages 129-158.

Step 1: Prepare your planning model

The first step in developing your program evaluation is to clarify your assumptions about the specific changes the program is intended to cause within the target population, how it will achieve these changes, and how you will know when these changes have occurred. The planning model will help you to accomplish this step.

For detailed information about planning models, see the Program Planning section.

The process of developing and/or reviewing the planning model provides an ideal opportunity for all program partners, including youth from the community, to share their assumptions about and perceptions of the program and to ensure that their expectations are realistic given the scope of the program and the nature of the targeted problem. Once you have completed, reviewed, revised, and achieved stakeholder consensus on your planning model, you can begin identifying potential measures, or indicators, of each planning model component and determine how the relevant data can be collected.

Step 2: Develop your evaluation questions

When you look at your completed planning model, you should be able to identify numerous questions that an evaluation could answer. For example, an evaluation:

• could determine whether the program actually addresses the identified problem or need,
whether it actually served members of the target population,
• whether it was implemented as planned,
• whether it achieved its expected outcomes.

To keep the scope of your evaluation manageable, you will need to rate your evaluation questions in order of priority based on your information needs, the needs and requirements of your funders and other audiences, and your time constraints. One way to do this is to imagine that you will present your evaluation results to an important audience.

Ask yourself the following questions:

• What are the three most important points you would like to be able to make about your program? That is, on which parts of your planning model would you like to focus? Think about the people and organizations to whom you plan to present your evaluation results.
  
  Different audiences will be interested in different parts of your model. For example, representatives from arts agencies will be more interested in whether you can show that your program increased participants' arts knowledge and creativity than will representatives from juvenile justice organizations. Figuring out who your audience includes will help you to prioritize your questions and, thus, keep the scope of your evaluation manageable.



• When do you need to make this presentation? You will likely want to use findings from your evaluation in proposals for new or continuation grants, progress reports to existing funders, and similar fundraising and marketing efforts. Keeping a calendar of key dates by which findings are needed will help you to develop realistic evaluation questions.
  
  For example, if findings are needed for a continuation grant at the end of the program's first year, you will need to focus some of your efforts on gathering information about program implementation and about the immediate outcomes of your program. You will not be able to measure its long-term or overall effectiveness within the program's first year. Specifying a time frame will help you to determine which evaluation questions are realistic to answer.

Once you have thought through these questions, you should be ready to develop your evaluation questions. Remember that your questions should test some aspect of your planning model and be clear, specific, and realistic.

Many factors must be considered in order to develop realistic expectations for an arts-based delinquency prevention or intervention program.

These factors include the characteristics of the target population, the types of risk factors addressed by the program, attendance and participation rates, and a range of outside factors — such as gang activity in the youths' community — that might influence the participants' attitudes and behaviors.

Step 3: Assess whether an evaluation is feasible

Assessing early on whether it is feasible for you to conduct an evaluation at all can save you a lot of time and energy and help ensure meaningful evaluation results. This process, sometimes called evaluability assessment, involves answering questions such as:

• Do sufficient resources exist to support the evaluation?
• How feasible is it for you to access existing data sources?
• How feasible is it to collect new data?
• Do similar evaluation efforts exist?

After finding answers to these questions, you should be able to decide whether it is possible for you to conduct an evaluation that will answer your evaluation questions and meet the information needs of your intended evaluation audience. You also should be able to estimate the level of effort required to gather your evaluation data and determine whether you will need outside assistance.

If you decide that it is not possible for you to conduct a well-planned outcome evaluation that will answer all of your evaluation questions, don't despair. See page 136 of the "full chapter PDF for the "Making Do With What You Have" section, which can help you to determine the types of evaluation activities you can successfully complete given the amount of money available for you to spend on evaluation.

See page 174 of the full chapter PDF for information about "The Program Manager's Guide to Evaluation."

Step 4: Plan your evaluation

This step involves preparing a detailed written document that can be circulated to and reviewed by the key players involved in the evaluation. Reviewing evaluation plans can lead program partners to provide additional information about the program and their expectations for the evaluation, which can help guide the evaluation in the right direction.

A comprehensive written evaluation plan includes the following:

• background and purpose of the evaluation
  Includes a brief program descriptions, program planning model, evaluation questions, and an explanation of how the evaluation results will be used.



• outcome evaluation design
  An evaluation design specifies when, from whom, and about whom you will collect outcome evaluation data. It determines how you will measure changes in program participants and how you will prove that these changes resulted from your program.



• process evaluation plan
  Well-planned outcome evaluations also include process evaluation activities that answer questions about how the program was intended to operate and how it actually operates on a daily basis. They provide valuable information about factors that facilitate and impede program implementation, promising program strategies and areas that need improvement, as well as the contextual information needed to interpret changes in participants' knowledge, attitudes, and behaviors.



• data collection strategy (data map and data collection instruments)
  This section describes how you will collect the data needed to answer the evaluation questions, using a data map and data collection instruments. As discussed in Step 2 and Step 3, a data map shows how you plan to answer your evaluation questions. Specifically, it is a table that links the planning model and evaluation questions to the indicators and data sources (see Table 2 on page 132 of the full chapter PDF). You should describe all of the data sources included in your data map-surveys, intake forms, school records, interviews, and so forth. If you are developing or adapting program-specific data collection forms or surveys, you should describe their contents and include copies of these instruments in an appendix.



• data collection plan
  This plan describes the "who, what, when, and where" of data collection. It tells who will be responsible for collecting data from the sources included in the data map and describes any training the data collectors will receive to prepare them for this task. It also describes how and when the data collection instruments will be administered to the appropriate subjects.



• data analysis plan
  These plans describe how the collected data will be analyzed and how these analyses will be used to answer the evaluation questions. You can describe the methods that you intend to use to analyze your data in text and/or include them in a column of your data map. You also should describe any anticipated constraints on your analyses.



• draft outline for the final evaluation report
  Draft outline for final evaluation report: It's a good idea to include a draft outline for your future evaluation product in your evaluation plan, whether it will be an evaluation report, briefing, article, or other type of written or oral presentation. Laying out what you plan to say about your program and evaluation effort will help you to "stick to the point." A sample evaluation report outline appears in Table 3 of the full chapter PDF.



• timeline
  Your timeline should include each of the evaluation steps discussed here, as well as the specific tasks that will occur within these steps, including conducting meetings, distributing draft items-for example, planning models or data maps-to program stakeholders for review, and developing data bases. A portion of a sample timeline format is provided in Table 4 of the full chapter PDF.

If it is not possible for you to produce a detailed written plan, you should at least try to address each of these components in outline format and discuss them with your program partners.

Step 5: Identify data collection instruments

When selecting or preparing your data collection instruments, you will need to consider which type of instrument best suits your needs. For example, written surveys or questionnaires are often used to gather large amounts of information from many people, while interviews (by phone or in person) are often used to gather qualitative information about program implementation and operations and program outcomes from program participants, staff, and other partners on an individual basis.

More Resources

• For more about all the data collection instruments, see pages 145-147 of the full chapter PDF.
• Read more about the data collection instruments used in the YouthARTS outcome evaluation here.
• Focus group questions

SAMPLE EVALUATION QUESTIONS FROM THE YouthARTS EVALUATION:

• To what extent are the YouthARTS programs providing arts-based activities to youth during non-school hours? (That is, what are the attendance and completion rates?)

• To what extent do the programs increase participants' art knowledge and improve their program-related skills such as communication and cooperation?

• To what extent do the programs have the desired effects on the attitudes and behaviors that affect delinquency and academic performance? (For instance, do the participants have healthier attitudes about drug use and improved self-esteem?)

• To what extent do the programs decrease juvenile delinquency (that is, reduce court referrals) and increase academic achievement (increase grade point averages)?

Step 6: Collect data

This section explains the steps you should take before collecting your program evaluation data, as well as the differences between qualitative and quantitative data.

More Resources

• Sample consent forms
• Download data collection instruments
• YouthARTS data collection implementation guide
• See pages 148-149 of the full chapter PDF for more information about collecting data.

Step 7: Analyze data

This section is designed to walk you through some of the basic methods that you will need to use to analyze your outcome data.

Table 5: Sample Evaluation Data

Two types of analyses can be conducted on these quantitative data to evaluate your programs outcomes: descriptive analyses and comparative analyses. For a more in-depth look at both, see pages 151-154 of the full chapter PDF.

Step 8: Present findings and disseminate lessons learned

You are now ready to share what you have learned from your program evaluation effort by presenting your findings in a written report, an executive summary, an oral briefing, or another type of presentation.

In general, your presentation should describe

• the program and its planning model,
• the purpose and methodology of the evaluation,
• and the process and outcome evaluation findings.

If you choose to write an evaluation report, you should include the data collection instruments and supporting documents in an appendix. The type of presentation, its format, and its level of specificity should reflect the needs and preferences of its intended audience.

NOTE: If you are preparing the report for a government agency that is funding your program, you may want to contact that agency to find out if it has any specific requirements or preferences for evaluation products. Many government agencies expect evaluation reports to contain executive summaries, which highlight key findings, conclusions, and recommendations and help facilitate decision-making. You also may want to disseminate your findings and any lessons that you have learned to wider audiences through press releases, newspaper articles, the Internet, or other media.

Step 9: Use evaluation findings

In general, evaluation findings can be used to:

• fine-tune, expand, or curtail a program
• make management and administrative changes
• influence policy decisions
• add to existing knowledge about juvenile delinquency prevention
• undertake a new evaluation effort

No matter how informative an evaluation is, its worth lies in the extent to which the program and/or the field are able to use the information to improve existing programs, create new programs, replicate promising approaches, and/or conduct new research that will guide future programming efforts.

Step 10: Think about conducting follow-up

In order to decide whether to conduct a follow-up, you should consider the following questions:

• How useful would positive follow-up results be to your program?
• What would you expect the long-term outcomes of your program to be, given what you now know about your program and the target population?
• How difficult will it be for you or your data collector to track—contact and collect data from—members of the treatment and comparison groups? Do you think you could find enough of the youth to make it a worthwhile effort?
• How committed are program staff and others involved in the evaluation process to the follow-up evaluation effort?

If your answers to these questions are encouraging enough for you to conduct a follow-up evaluation, then you will need to decide what data to collect. While it is a good idea to re-administer some of the same instruments that you used during your original evaluation (to assess trends over time), you may want to include only those questions that focus on topics of particular interest. You also may choose to administer new instruments, such as interview and focus group protocols, that gather more qualitative information about program outcomes or focus on potential program outcomes that were not addressed in the original evaluation.

TIPS FOR SURVEY OR TEST QUESTIONS

Here are general points to keep in mind when you are developing surveys or tests that will be administered to program stakeholders.

• Use clear, simple language that all respondents will be able to comprehend.

• Make sure the questions ask what you want to know.

• Each question should ask about only one thing.

• Avoid generalizations; each question should be specific.

• Do not use a leading question (that is, one that suggests that there is a preferred response).

• Make sure respondents understand what you are asking.

• Make sure respondents are familiar with the topic you are asking about.

• Identify whether the respondent should mark one choice or all choices that apply.

• Response choices should be comprehensive (include "not applicable" if necessary) and exclusive (choices should not overlap).