|
|
Evaluation Techniques and Instruments |
|
Evaluation Technique |
Type of Instrument |
How It Is Administered |
What It Evaluates |
Advantages |
Disadvan-
tages |
|
Interview (individual or group) |
Questionnaire |
Written or oral
Self-administered |
Attitudes
Opinions
Reported experience and practices |
Easy to administer
Can be given to large groups
Can generate qualitative or quantitative data
Answers can be clarified and expanded
Sensitive topics can be probed |
Can be costly and time-consuming
No assurance that questions are understood
Respondents must be literate to complete self-administered questionnaire
Respondents may say what they think evaluator wants to hear
Questions can reflect biases of evaluators
Requires thorough training of evaluators
Difficult to analyze data |
|
Test |
Form or questionnaire |
Written or oral |
Knowledge
Skill |
Low cost
Easy to administer
Can be given to large groups
Easy to analyze responses |
Respondents must be literate for written test
May not accurately reflect skills or actual behavior on the job
May be threatening to respondents |
|
Observation |
Guidelines
Checklist |
Watching and recording (actual performance, simulation, role play, videotape, etc.) |
Skill
Performance
Attitudes |
Can show actual behavior
Generates qualitative and quantitative data |
Costly and time-consuming
Requires thorough training of evaluators
Can induce behavior changes at time of observation |
|
Review of documents |
Checklist
Questionnaire |
Review of records (client and staff records, service statistics forms, critical incidents file) |
Performance
Compliance with legal or professional requirements |
Convenient
Avoids burdening staff with extra demands during visit
If accurate, will yield objective data |
If records are handwritten, may be hard to read
May be incomplete, inaccurate, or overly generalized
Files may be poorly organized and impede search for relevant documents
Medical records may need to be read by an expert. |
Pretesting and Revising Instruments
Even when instruments have been designed carefully, there will be surprises when they are used for the first time. Questions and procedures that seemed absolutely clear sometimes can confuse or bring anxiety to the person being interviewed. The best way to reduce unpleasant surprises is for members of the evaluation team to pretest the instruments with a small group of trainees, preferably at more than one site. The pretest can provide the basis for revising instruments—and, occasionally, indicators or standards. The pretest should duplicate as closely as possible the arrangements that will be made for the actual TIE: careful scheduling, letters to the sites, and introductory explanations that explain the TIE to the project directors, trainees, and others who may be involved.
If it can be arranged, you should work in pairs during the pretests, just as you will during the final TIE. This will allow one person to concentrate on answers and pick up on interesting comments or visual cues without having to slow down the process by writing. It will also help to confirm observations and impressions.
Because pretesting adds to the cost of impact evaluation and takes valuable time, it is often omitted. But the pretest is important, because if the instruments are not designed well, they will not yield credible data, the analysis and interpretation of the results will be flawed, and any conclusions and decisions about training or organizational support will be questionable. To reduce the cost in staff time and money without compromising the learning opportunity, you should seek ways to combine this pretest with other organizational activities.
|
