ID 553 - Module 7: Evaluation: Determining the Effect of the Intervention

Formative and Summative Evaluation

Estimated reading time: 30 minutes.

Video: Formative and Summative Evaluation. Lecture: the two evaluation questions, a formative evaluation of a real module through one-to-one, small group, and field trial, analyzing the data by objective, and Kirkpatrick's four levels. (Instructor: build with make_videos.py or record and embed your own.)

Two questions, two evaluations

Formative evaluation asks: how can this instruction be improved before it is finished? It happens during design and development and its audience is the design team.

Summative evaluation asks: did the finished instruction work, and is it worth continuing? It happens after implementation and its audience is the decision makers who fund and mandate the instruction.

Both are part of the design, not afterthoughts. Dick and Carey give each its own component and make revision a loop that formative evaluation feeds. Every plan you write in practice should say when and how both will happen.

Formative evaluation: three stages

Dick and Carey's three-stage sequence is the standard.

One-to-one evaluation

Work through the draft instruction with one learner at a time, three or so learners representing the range of the target population (one above average, one average, one below on entry skills). Sit beside them, ask them to think aloud, and note errors, confusions, pacing problems, and their reactions. Ask about clarity, impact, and feasibility. Give the posttest. This stage catches the gross errors: wrong content, unclear instructions, missing steps, examples that do not work.

Small-group evaluation

After revising, try the instruction with 8 to 20 learners under conditions close to real use, with the designer observing but not intervening. Collect the pretest and posttest, item-level results, time on task, and a short attitude survey. This stage tests whether the revisions worked and whether the instruction functions without the designer present. Analyze results by objective: which objectives were achieved, which were not, and what in the instruction accounts for it.

Field trial

Try the revised instruction with a full group (30 or more) in the actual setting, delivered by the intended facilitators or the intended self-paced system. The designer is absent or a distant observer. This stage tests implementation: facilitator guides, logistics, technology, and the effect of the real context. Collect the same data as the small group plus facilitator feedback.

Expert review

Alongside learner-based stages, have a subject-matter expert check accuracy, a second instructional designer check alignment and strategy, and an accessibility reviewer check conformance. Expert review is fast and finds things learners cannot name.

Analyzing and using formative data

Build an item-by-objective table. An objective with low posttest success and adequate instruction time points to a strategy problem; low success with little time points to a coverage problem.
Compare pretest and posttest by objective, not only overall. A high overall gain can hide one objective nobody learned.
Pair every problem with a location in the instruction and a proposed revision. Revise, then retest the revised parts if the stakes justify it.
Keep a revision log; it is evidence for the summative report and for your own learning.

Summative evaluation

Summative evaluation judges the instruction against its purpose. The classic framework is Kirkpatrick's four levels, updated as the New World Kirkpatrick Model:

Level	Question	Typical measures
1. Reaction	Did learners find it engaging, relevant, and satisfactory?	Post-instruction survey with specific items, not just "was it good"
2. Learning	Did they acquire the intended knowledge, skills, attitudes, and confidence?	Posttest, performance assessment, pre-post comparison
3. Behavior	Do they apply it on the job or in the next course?	Observation, supervisor reports, work samples, system data, follow-up at 30 to 90 days
4. Results	Did the targeted outcomes change?	Error rates, help desk calls, performance metrics, retention, cost, the accreditor's criteria

Two cautions. First, level 1 alone tells you almost nothing about effectiveness; satisfaction and learning are only weakly related. Second, level 4 outcomes are affected by many factors besides instruction, so claim contribution, not sole cause, and design the evaluation so plausible alternative explanations can be addressed (a comparison group, a baseline, time-series data).

Summative evaluation has two phases in Dick and Carey's framing: an expert judgment phase (is the instruction congruent with the organization's needs, complete, accurate, and well designed?) and a field trial phase (does it produce the outcomes with the intended learners in the intended setting?).

Evaluation and the models

ADDIE: Evaluation is the fifth phase and also runs across the others.
Dick and Carey: components 8 (formative), 9 (revise), and 10 (summative).
SAM: evaluation is built into every iteration; the "evaluate" step in each cycle is formative, and the final implementation is evaluated summatively.
Backward Design: stage 2 (acceptable evidence) is the assessment design; program-level evaluation is added on top.

Designing an evaluation plan

For your project, the plan should specify:

Formative: which stages you will run (at minimum one-to-one and small group, or a justified adaptation), with participants, instruments, data, and the decision rules for revision.
Summative: the levels you will measure, the instruments, the timing, the comparison or baseline, and who receives the report.
Logistics: who does what, when, at what cost, and what happens to the data (consent, storage, privacy).
Reporting: the form of the report and the decisions it will inform.

An evaluation plan that can actually be carried out with the resources available beats an ambitious one that will not happen.

References

Dick, W., Carey, L., & Carey, J. O. (2015). The systematic design of instruction (8th ed.). Pearson. Chapters 10 to 12.

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Tessmer, M. (1993). Planning and conducting formative evaluations. Kogan Page.