If
you want life to be easy, you must pay for it.
This is the
last step in the program planning process. Yes, evaluation should
be considered during the planning process. It should not wait
until the program has been completed.
In order
to understand evaluation's place in the program planning process
you will need to review module E-4. In order to plan an evaluation
correctly it should follow the other steps in program planning
(described in modules E-5 through E-9 and M-2).
LEARNING
OBJECTIVES
When you
finish this module you should be able to:
1. Define
"evaluation."
2. Plan for the evaluation of some aspect of your group's educational
program.
EVALUATION
DEFINED
Evaluation
is simply "a process to determine if objectives have been
met." Well written objectives are essential to planning an
evaluation. When planning an evaluation, start by reviewing the
objectives. If you lack a set of clear, appropriate objectives,
you will find it difficult to get agreement on any plan for evaluation.
LEVELS
OF EVALUATION
Dr. Claude
Bennett, Evaluation Specialist, Extension Service, U.S. Department
of Agriculture, has outlined seven levels of evaluation which
will help us understand the benefits and limitations of particular
evaluation techniques and instruments.
1. Resources
How many dollars, how many handouts and exhibits, what resource
people, what materials did we use in the educational program that
we are evaluating?
2. Activities
How many planning meetings, demonstrations, workshops, field trips,
contests, telephone calls, newspaper articles did we complete?
3. Participants
Who participated in the activities that we organized? What were
their ages, ethnic backgrounds, gender, socio-economic status,
area of residence, organizations represented (and how many of
each category)?
4. Reactions
Did the participants like the activities? What comments or suggestions
did they make regarding the activities?
5. Changes
in Knowledge, Attitudes, Skills and Aspirations (KASA) As
a result of your program's activities how did the participants
improve (or regress) in KASA? Was there really any difference
in the participants as a result of the program?
6. Adoption
of New Practices Did the participants retain the KASA changes
over time? Were these changes of short duration or did they become
part of the participants' lifestyles?
7. End
Results What were the long term effects of your program on
the local community or society in general? Did the number of teen
pregnancies decrease? Did the production of wheat (bushels per
acre) increase? Were new organizations created? Was the lifestyle
of community members improved? Can the improvements be attributed
to the educational program that you provided?
These levels
of evaluation represent an increasing time requirement and increasing
difficulty for the evaluator. They also represent an increasing
value of evaluation results. Determining resources, activities
and participants is relatively easy--it can be done by observation
and counting. To determine reactions you will need an organized
technique (a trained observer/recorder or a post-meeting questionnaire).
To determine changes in KASA and adoption of new practices you
will need to gather evidence before and after the program in a
reliable and valid procedure. End results take time and great
sophistication of measurement by evaluators.
On the other
hand, if you can gather evidence about changes in KASA or adoption
of new practices, you will have stronger evidence for continuing
your program or for getting funding for similar programs.
EVALUATION
ALTERNATIVES
To most of
us, evaluation means completing an evaluation form at the end
of a meeting or training program. Other, often more useful, forms
of evaluation are described below.
Round
Robin
The round
robin evaluation simply involves calling on each person in turn
to share reactions to a given activity. For example, "What
was your evaluation of last week's meeting (4-H horse show, training
session)? What did you particularly like, and where might improvements
be made?" Appoint a secretary to record responses. An important
ground rule is no discussion, except for questions of clarification.
This keeps the group from getting bogged down in defending, disputing
and discussing individual comments until everyone has been heard.
The round robin insures input from everyone, without debate. Once
everyone has been called upon, the composite listing can then
be discussed, if necessary, and conclusions drawn.
One of the
most commonly used forms of evaluation is a call for public input.
The chair (or other official) poses a general question to the
group as a whole, e.g., "Any comments on last week's horse
show?" Experience shows that only a few people respond to
such calls for input. The round robin, in contrast, will insure
greater input and involvement. If members find that their input
is respected and used, they will be more willing to speak openly
in the future, and the quality of evaluation will increase.
Buzz Groups
Buzz groups,
often called Process Groups, involve quickly breaking into small
groups of two to five people. Each buzz group is to assess the
meeting (workshop, etc.). Each person is asked to share his frustrations,
concerns, suggestions and satisfaction with how things are going.
Where there is consensus, the buzz group should initiate action
to improve the meeting or workshop, e.g., propose to the group
that..., or talk to the chair about..., or suggest....
Set a time
limit of 15 to 30 minutes for this exercise. One option is to
incorporate the process group assignment into an extended break.
The advantage of this exercise is that people tend to share more
openly in smaller more private groups than they do in larger public
groups. Also the focus of the exercise is on action rather than
complaints. Periodic process breaks are invaluable in developing
a cohesive team spirit along with an added sense of personal responsibility.
Observer
Arrange for
one or two individuals to serve as a process observer. They will
not enter into the discussion but instead sit back and observe
how the meeting is progressing, (i.e., what are the group dynamics,
who is and isn't talking, how are decisions made, what team building
roles were exhibited). The observer is to identify strengths as
well as weaknesses. The checklist provided earlier in this chapter
can serve to guide the observer in analyzing a group.
The observers
are then called upon at the end of the meeting to share their
observations and suggestions with the group, and to pinpoint team
building skills the group could work on. The advantage of designated
observers is that they are removed from the operation of the group
and can be more systematic and objective in their assessment.
It is difficult for individuals who are directly involved in a
meeting to step back and critique what is happening.
Review Goals
Every organization
should plan time at least quarterly to review its goals and objectives.
What has been accomplished? What has not? What adjustments (additions
or deletions) need to be made? How do individuals in the group
feel about the group's accomplishments? How well is the group
(or designated committees) working together to accomplish those
goals? Do these goals still reflect the priority interests of
group members? What are the group's short-comings?
Such evaluation
has the advantage of focusing the group's attention on accomplishments.
Changes in the group's goals, committee organization, or how meetings
are conducted can be made now rather than put off until next year.
One-on-One
Consultation
One of the
most useful forms of evaluation is face-to-face consultation.
Good managers are in frequent contact with workers (group members).
They seek feedback and advice. Most people are reluctant to give
advice unless it is asked for. To get useful feedback, ask
specific questions.
Quickly get
any areas of potential awkwardness out in the open, i.e., "I
heard you were upset with me about. . .Let's talk about that first."
Be an active listener. Don't respond defensively.
In giving
feedback that has not been requested, one-on-one communication
also works best. No one likes to be criticized in public. (Review
module V-6, for guidelines in giving constructive criticism.)
Written Evaluation
Survey
The traditional
evaluation form asks participants to rate the meeting (workshop
or event) according to certain listed criteria. Several open ended
questions should also be included for a more detailed and personal
response, e.g., "Where might improvements be made?"
Two examples of survey evaluations forms are attached ("End
of Meeting Suggestion Slip" and "Workshop Evaluation,"
pages 10-11). Such surveys have the advantage of being anonymous.
Also, the results lend themselves to statistical analysis. Take
care in preparing the evaluation form to insure that the questions
asked are easily understood and that they provide information
that can be used.
WRITTEN
EVALUATION INSTRUMENTS
A.
End of Meeting Suggestion Slip
1. Please
rate today's meeting on the basis of the following criteria.
