Wednesday, May 14, 2014

ADDIE: Evaluation

At last, the training is done and you can relax, right?  Wrong!  The last step of the ADDIE process is arguably one of the most important because, as a way to determine its effectiveness, the evaluation of instruction will aid the trainer in determining what succeeded and what failed.  It may seem as though the evaluation would be the conclusion of any training program but, instead, it is often just a midway point to perfecting training.  Evaluation can and should be conducted throughout the process of ADDIE through formative assessment and, at the conclusion of an instruction period, summative assessment.  The most well known summative assessment is Kirkpatrick's Four Level's.  I've written about Kirkpatrick before in relation to training strategies, but I want to delve a little deeper into the importance of each level and the different strategies you can use to get feedback from staff and help solidify the training they received.

Level 1: Reaction
It should come as no shock that immediate feedback from your learners is a great way to judge how well you did and determine what you can improve upon.  Gaining reaction can be done through formative assessment, via the instructor's observations during training, or through summative assessment, via a survey at the end of the training.  Both of these methods should be used to gauge how well the training went.  As far as evaluation goes, this is the bare minimum that one can do.  It is relatively easy to carry out a summative assessment immediately following training but the learner responses will only affect the trainer and will not play a role in information retention for the learner.

I have found it wise to use simple surveys that ask no more than several questions.  The more questions you ask, the less likely a participant is to respond with free text.  What you value, scores or written feedback, should determine the type of survey you give.

Level 2: Learning
The actual learning a participant does may not even happen until he/she is back in the workplace and is putting into practice the methods taught in the training.  In the second level of evaluation, trainers have the ability to help solidify what the learner has absorbed by following up with the employee a week or two after the training has occurred. You may want to give the learners an assessment test or an assignment that will bring their focus back to the information they have learned and how they have successfully applied it.  At the very least, ask the learner to fill out a self-assessment about how they feel they have applied the information.

Level 3: Behavior
Several months after the training, there should be a significant change in behavior.  At this level it becomes more difficult to gauge how the training has helped.  Qualitative feedback from employees' direct supervisors will be of great help in this level of evaluation.  Supervisors can give feedback based on the overall performance behaviors that have changed in their employees.  If little to no change has occurred, then the trainer will know that, ultimately, the training was unsuccessful.  Having direct communication and an open dialogue with supervisors is a great way to receive feedback on what is working and what is not.

Level 4: Results
The final level is often never achieved as it deals with return on investment.  If you recall from the Analysis phase of ADDIE, Mager's 5th step of analysis says:

If the estimated cost of the discrepancy is small, stop!  In other words, it is only a problem because you say it is but is not having any impact on the rest of the world.  So, STOP!

If you have completed training then the assumption is that the original cost analysis warranted it.  Now, in the fourth level of evaluation, a trainer should determine if the cost of training actually paid off.  This is determined through company statistics or other measurables.  In order to determine if the training "paid off," key stakeholders will be looking at the bottom line.  

In libraries, the measurables will be somewhat different from a corporation as the bottom line is not necessarily monetary growth.  But other statistics can be gleaned after training has taken place: increase in circulation, increase in attendance, decrease in complaints, decrease in staff time to accomplish a task, etc.  All of these items can be looked upon to determine the worth of a training.  Of course, finding the results is not without its own cost. It takes staff time and resources to inquire about statistics of this nature which is why Level 4 of Kirkpatrick's Levels of Evaluation rarely happens.

Whether or not you manage to accomplish all four levels, the bigger question is "are you completing the ADDIE phase of evaluation at all?"  Within Kirkpatrick's levels, the more levels you achieve the more likely your training will pay off in the end.  But if the evaluation phase is not your strongest, feel comfortable in taking baby steps.  Start with basic feedback immediately following a training.  Once you feel comfortable with this method, step up your evaluation game and proceed to level 2 and 3 by communicating with learners and supervisors weeks and months after the training.  Trust me, you won't regret it.

No comments: