Approved 4/25/02
PROGRAM EVALUATION
Level IV
RATIONALE: Many community programs are funded by government and private
sources that have specific requirements for documenting the delivery of the
program, the effects the program has on the participants and the elements of
success and failure. Program evaluation is also important to the implementation
of programs: programs often use the information provided by evaluations to make
internal decisions about the effectiveness and quality of specific program activities.
Evaluation is an integral part of how programs improve their performance, document
their results and maintain accountability, all critical ingredients of public
and private support for human services.
Program evaluation is an example of applied social science research. Conducting good program evaluations requires knowing the basic principles of social science research and experimental design, and the use of data collection methods similar to those of social science research. It also requires an understanding of the relationship between theoretical concepts of human behavior or social change, on the one hand, and particular programmatic interventions, on the other. Furthermore, it involves the ability to assess whether a proposed evaluation is feasible, i.e., the practicality of actually assessing program performance and impacts. In addition, it is important to know and implement appropriate ethical guidelines for informed consent, confidentiality and cultural competence.
The purpose of this competency is to provide the student with the skills necessary to plan appropriate evaluations for community-based services. The competency requires basic understanding of the various types of evaluation approaches and the criteria for selection of the most appropriate type for the intended purpose. It will also requires demonstration of skills in the design of appropriate and ethical evaluations. The competency does not require the student to actually carry out the evaluation.
Prerequisite: Social Research: Consumption and Design
COMPETENCY: Can utilize understanding of the various approaches to community
program evaluation to design an appropriate evaluation plan for a specific program
and purpose that meets applicable professional and ethical standards.
CRITERIA:
1. Identify and describe a human services, education, or community program for which to plan an evaluation
2. Describe the program theory that underlies the program to be evaluated and the logic model of the program
3. Describe the purpose and the assess the feasibility of an evaluation of the program
4. Select the most appropriate evaluation approach for the purpose of the evaluation
5. Select and clearly describe the questions to be addressed in the evaluation
6. Develop an evaluation plan that addresses the questions posed. The plan should describe carefully the design or designs to be used as well as the data collection methods.
7. Describe how the ethical issues of risk, informed consent, confidentiality and cultural competence will be addressed in all aspects of implementation of the evaluation.
8. Assess the strengths and weaknesses of the evaluation design including those related to validity, reliability, and generalizability.
9. Describe how the information gathered through the evaluation could be used by the organization or other decision makers to improve program performance.
Portfolio Links: You are expected to use the Level III Communications
Portfolio writing standards as guidelines for the written work required
by this competency.
STANDARDS:
1. For Criterion 1, the program may be a proposed one or one already in operation. Its description must include:
a. name of the organization of which the program is part
b. mission of the organization;
c. approximate overall annual operating budget of the organization
d. approximate total number of full- and part-time staff working in the organization;
e. types of programs delivered and units within the organization;
f. description of the specific program or unit for which the evaluation will be planned including:
i. name and the mission of the program
ii. approximate annual operating budget of the program
iii. number of full or part time staff working in the program.
2. For Criterion 2, articulate clearly the program’s theory of change, that is the underlying links between the problem the program addressed and the specific activities undertaken by the program. In addition, present a logic model that includes goals, objectives, target groups and activities as well as the expected outcomes and processes of the program.
3a. For Criterion 3, explanation of the purpose of the evaluation must clearly indicate whether the evaluation to be undertaken is a process or an outcome evaluation, which stakeholders will use the results of the evaluation and whether it will involve participatory methods.
3b. Criterion 3 further requires that an assessment be conducted of the “evaluability" or the stage of readiness of the program to be evaluated. This assessment must take into account at least the following:
a. stage of development of the program
b. information needs of the organization/unit
c. purpose of the evaluation
d. the context of the evaluation (such as is the program under attack, or it is a self assessment to be used by the agency to improve services etc.)
e. the time and resources available to conduct the evaluation
4. In Criterion 4 the discussion of the selected evaluation approach should describe and consider each of the types of evaluation approaches listed below and provide a rationale for the selection of the particular approach:
a. experimental or quasi-experimental
b. goal oriented
c. user/decision focused
d. process oriented
e. participatory
5. For criterion 5, a clear statement of evaluation questions is required. At least four evaluation questions in total must be developed. The specific questions to be addressed in the evaluation must:
a. match the type of evaluation approach, that is:
i. experimental or goal oriented evaluations must identify quantifiable process and outcome objectives (at least two of each), while
ii. evaluations designed for other purposes must focus on clear questions relevant to the context of the evaluation.
b. match program needs as well as the political context of the evaluation
c. be consistent with the amount of time and of resources available to conduct the evaluation.
6. For Criterion 6, the data collection methods must be described in enough detail so that the evaluation could be carried out by someone else following the description of the methods. Details must include:
a. specific types of information to be collected
b. drafts of data collection instruments (whether commercial, already developed by others, or newly drafted) must be provided, along with a rationale for the use of each; references must be provided for all data collection instruments not developed by the student
c. from whom and how often data will be collected
d. whether ongoing program records will be used and for what purpose
e. the timeline for data collection.
7. For Criterion 7, provide a discussion of confidentiality procedures, informed consent and human subject protections which follow the legal and ethical guidelines of professional associations such as the National Institutes of Health, the American Sociological Association, or the American Psychology Association. Indicate the process required for Institutional Review Board approval. Show that you have considered class, gender, racial and cultural issues in the design of the evaluation.
8. For Criteria 8 and 9, the discussion of the usefulness of the evaluation information must consider for whom the information is useful, why, and what are possible limitations of the evaluation. Standard social science research concepts must be used.
EXAMPLES OF DEMONSTRATION:
1. Prior Learning: As part of your agency work you were asked to plan an evaluation of a new service component to provide after school child care and educational support services to 6-12 year olds, for purposes of applying for new funding. You developed a written evaluation plan as part of the grant application. You submit the evaluation plan along with information describing the agency, the context of the evaluation, and a description of the usefulness of the evaluation information. You provide a letter from a supervisor as to your role in developing the evaluation. You meet with the faculty evaluator to answer questions about the plan and provide additional information, based on your work, to meet criteria and standards.
2. Independent Learning: After obtaining readings about program evaluation from the faculty evaluator, you approach a program in your agency and request permission to use it as a learning site, stressing that your task is to produce an evaluation plan and not to conduct an evaluation. Once permission is obtained, you request basic information about the program and work with program staff in developing the evaluation questions and selecting an evaluation approach. You, with or without their participation, develop an evaluation plan. Finally, you write up the plan and present it to the agency with your recommendations for implementation.
3. Course: A student completes a CPCS course addressing the competency or an appropriate transfer.
4. Field Project: As part of a group of students working with a faculty member and a community agency addressing youth violence issues you plan an evaluation of the street worker program. You participate with the group in identifying the focus and approach, developing the evaluation questions, and planning the methods. You present the written plan to the agency with your recommendations for implementation and make an oral presentation to the agency, which addresses the usefulness of the information and the decisions the agency can make based on the information.