Designing
for Evaluation
by
Susan MacDonald
Corporate Trainer
Evaluation
by Design participant, November
10-12, 2005, Toronto, Ontario
My Background
In
the past I have worked in the corporate environment doing training
design and delivery. In my experience, evaluation of training
is not taken very seriously in this environment. Often the only
attempt at evaluation is getting participants to complete "happy
sheets" at the end of their training. I find participants
are often very compliant and accepting of very poor quality
training, usually because they have no experience of training
of any other kind. So it was with a great amount of interest
that I enrolled in Global Learning Partners "Evaluation
by Design" course.
My
Sample Design
I have been volunteering at a hospice where I worked on a volunteer
training module called "Care for Caregivers." I did
not have complete control over the module so it was not designed
using the principles and practices of Dialogue Education. There
was no LNRA, but I hoped it was okay to work on in the Evaluation
by Design course.
During the recap of the Seven
Steps I noted the following quote, "At the foundation of
this entire approach are the WHO and WHY steps, which help designers
avoid making assumptions about participants and purposes".
These did indeed prove to be key steps which would come back
to haunt me later in the course…
Transfer
Objectives
It was when I came to write transfer objectives for my module
that I really came unstuck - and when the power of evaluation
really hit me. I was trying to formulate transfer objectives
for volunteer caregivers in a hospice who are learning about
monitoring their own stress levels. But I don't really know
if this is even what they need to be taught! And by the time
we came to consider the variables in the impact of training
(individual characteristics, learning experience and work environment)
I realized I was seriously at sea.
Individual Characteristics
Because we had not done an LNRA I did not know if the attendees
would be comfortable talking about their stress levels in a
group, or even if they were used to working in a group at all.
For some, this may be their first experience of training since
school. And they may have very little experience of talking
about stress with relative strangers.
The Learning
Experience
Back to my question - is this a valid task ("HOWs"
are so much easier than "WHAT FORs", don't you find?).
Perhaps they already have this knowledge and skill. Perhaps
it is not a priority learning task given other things they need
to learn? I can only know by talking to experienced volunteers
to assess the worth of the module and what really needs to be
included. And I would need to assess the needs of the attendees
via an LNRA.
Hospice Environment
Will there be follow up discussions on stress of the volunteers
once they are working there? Will they be supported to continuously
monitor their stress levels? Without the follow up, I think
attendees are much less likely to develop the habit which I
first identified as a desired outcome.
My Conclusion
The process of building evaluation into the design of my module
made me confront some difficult but essential questions about
my training design. In hindsight I now think that it would be
very useful to complete the Accountability Planner for a few
key "WHAT FORs" before completing the detail of the
design. In this way, you can be confident that your aim is sure.
Why the Diagram?
I completed this evaluation
diagram a few hours after completing the course. The diagram
was my way of recalling a pathway for the next time I do a training
design. I want this to ensure that evaluation is built into
the training from the "Get-go".
I found the diagram very easy to create
- which I think underlines how well thought out the "Accountability
Planner" is. It builds well on the 7 steps of design and
with a reference back to the "What for?", really provides
the trainer with a solid basis for confidence in the design
of a course.
Why Evaluate Learning?
Because it is
Good for Learners
This section really expanded my understanding of the value of
training evaluation. Firstly, let's look at it from the learner's
perspective, "It informs learners of their progress during
the program and improvements in their capacities when they finish
the program…."
The thing I love about this is that it
firmly places the learner as an equal in the learning process.
The note I wrote in my manual is that "evaluation increases
trust in the participants." This is quite a turnaround
from the testing that accompanies many learning experiences
and which puts the learners under the strain of proving their
abilities. While I think there is a need for clear criteria
for evaluation in some instances, I think evaluation needs to
be a process that both the learner and the trainer enter into
as equals and with equal enthusiasm.
Because it is
Good for Trainers
"Evaluation improves decision-making because future decisions
can be based on experience and results" This rings true
for me as a training designer. It emphasizes a process of continuous
improvement instead of the pretense that there is "one
way" that is "correct" which we should then cling
to. As a trainer I love the idea of being a continuous learner;
someone who, through training and modification of course content
and method can grow and evolve my practice and understanding
as my courses develop.
Because it is
Good for Our Clients
Finally, the statement that "evaluation shows that we are
serious about our work and committed to best practice"
signifies a quantum shift from my previous state of fear around
evaluation; proving the effectiveness of my training design
and delivery. This is now built in to the design rather than
me having to hold my breath to find out if I "got it right"
at the end. I want to feel confident in my work and be able
to show that it does work. And I want my clients to take the
training seriously enough to want to know that it has worked.
Levels of Evaluation
We were shown that there are three levels of evaluation: learning,
transfer and impact. The beauty of this model is that it broadens
our focus from just the learning event. In so doing it also
welcomes new players into the learning process.
During transfer the trainee's manager
may have a role in reporting on the degree to which the learners
are using the skills taught. Excellent! Get them involved from
the get-go (as Canadians would say). This shows honesty in acknowledging
the training process as just that - a process. It is not a one
off event which "fills up" the attendee with all the
knowledge they need, which they can automatically transfer to
the work environment unaided.
By involving co-workers and management
in transfer activities we can improve their understanding of
the learner and the learning process, as well as, hopefully,
improving their commitment to that process. They can become
actively involved by supporting and monitoring new behaviors
and skills, in the ongoing process of skills acquisition and
practice.
<<back
