Wednesday, May 14, 2014

ADDIE: Evaluation

At last, the training is done and you can relax, right?  Wrong!  The last step of the ADDIE process is arguably one of the most important because, as a way to determine its effectiveness, the evaluation of instruction will aid the trainer in determining what succeeded and what failed.  It may seem as though the evaluation would be the conclusion of any training program but, instead, it is often just a midway point to perfecting training.  Evaluation can and should be conducted throughout the process of ADDIE through formative assessment and, at the conclusion of an instruction period, summative assessment.  The most well known summative assessment is Kirkpatrick's Four Level's.  I've written about Kirkpatrick before in relation to training strategies, but I want to delve a little deeper into the importance of each level and the different strategies you can use to get feedback from staff and help solidify the training they received.

Level 1: Reaction
It should come as no shock that immediate feedback from your learners is a great way to judge how well you did and determine what you can improve upon.  Gaining reaction can be done through formative assessment, via the instructor's observations during training, or through summative assessment, via a survey at the end of the training.  Both of these methods should be used to gauge how well the training went.  As far as evaluation goes, this is the bare minimum that one can do.  It is relatively easy to carry out a summative assessment immediately following training but the learner responses will only affect the trainer and will not play a role in information retention for the learner.

I have found it wise to use simple surveys that ask no more than several questions.  The more questions you ask, the less likely a participant is to respond with free text.  What you value, scores or written feedback, should determine the type of survey you give.

Level 2: Learning
The actual learning a participant does may not even happen until he/she is back in the workplace and is putting into practice the methods taught in the training.  In the second level of evaluation, trainers have the ability to help solidify what the learner has absorbed by following up with the employee a week or two after the training has occurred. You may want to give the learners an assessment test or an assignment that will bring their focus back to the information they have learned and how they have successfully applied it.  At the very least, ask the learner to fill out a self-assessment about how they feel they have applied the information.

Level 3: Behavior
Several months after the training, there should be a significant change in behavior.  At this level it becomes more difficult to gauge how the training has helped.  Qualitative feedback from employees' direct supervisors will be of great help in this level of evaluation.  Supervisors can give feedback based on the overall performance behaviors that have changed in their employees.  If little to no change has occurred, then the trainer will know that, ultimately, the training was unsuccessful.  Having direct communication and an open dialogue with supervisors is a great way to receive feedback on what is working and what is not.

Level 4: Results
The final level is often never achieved as it deals with return on investment.  If you recall from the Analysis phase of ADDIE, Mager's 5th step of analysis says:

If the estimated cost of the discrepancy is small, stop!  In other words, it is only a problem because you say it is but is not having any impact on the rest of the world.  So, STOP!

If you have completed training then the assumption is that the original cost analysis warranted it.  Now, in the fourth level of evaluation, a trainer should determine if the cost of training actually paid off.  This is determined through company statistics or other measurables.  In order to determine if the training "paid off," key stakeholders will be looking at the bottom line.  

In libraries, the measurables will be somewhat different from a corporation as the bottom line is not necessarily monetary growth.  But other statistics can be gleaned after training has taken place: increase in circulation, increase in attendance, decrease in complaints, decrease in staff time to accomplish a task, etc.  All of these items can be looked upon to determine the worth of a training.  Of course, finding the results is not without its own cost. It takes staff time and resources to inquire about statistics of this nature which is why Level 4 of Kirkpatrick's Levels of Evaluation rarely happens.

Whether or not you manage to accomplish all four levels, the bigger question is "are you completing the ADDIE phase of evaluation at all?"  Within Kirkpatrick's levels, the more levels you achieve the more likely your training will pay off in the end.  But if the evaluation phase is not your strongest, feel comfortable in taking baby steps.  Start with basic feedback immediately following a training.  Once you feel comfortable with this method, step up your evaluation game and proceed to level 2 and 3 by communicating with learners and supervisors weeks and months after the training.  Trust me, you won't regret it.

Thursday, May 8, 2014

ADDIE: Implementation

If you've been following along with all the steps so far in the ADDIE process, then you should have a pretty good idea of who your learners are, the steps you need to take to achieve the desired outcome, and the style of learning you will use to implement the actual training.  If you follow these steps then you have already won half the battle.  It may take you days, weeks, or even months to make it to implementation - the part of ADDIE which usually takes the least amount of time.  But implementation is my favorite part because it is at this point where our ideas will come to fruition.

The best thing you can do to make instruction easy on yourself is to write everything down.  For this, creating an instructional plan or course handbook will be beneficial to you now and in the future.  For an idea of what your course handbook should look like, you can see the one I created for my catalog tutorial.  Having an instructor's manual will ensure that all of your course material is easily found in one place (links to videos, handouts, etc.)  Chances are, you may want to use the same instruction at a later date or there may come a time when you need to pass the course information over to another instructor to deliver the training.  Whatever the circumstances may be, it is always a good idea to write down your plan and save it.

Photo by Trey Ratcliff, via Flikr

The better the plan the more likely your training will have an impact.  If you took a glance at my sample course handbook, hopefully you were not surprised by the detailed nature of it.  All of those details meant that, when it came time to implement the training, I was prepared!  The details of the plan forced me to be familiar with the ins and outs of each enabling objective and, as a result, the training was more fluid to cater to the individual needs and questions of each learner.  Therefore, a more detailed plan does not necessarily mean rigidity.  Instead, it allows the instructor to be more knowledgeable in the subject area permitting him/her to deviate from the plan without losing any of the content.

Depending upon the type of training you are implementing and which instructional theory you decide on, your instructional plan may be smaller or larger than mine.  After completing the plan and implementing the training, don't think you are out of the woods yet.  There is one more step in the ADDIE process and it is a doozie.  It's now evaluation time!

Wednesday, May 7, 2014

ADDIE: Development

Once you have a clear roadmap of what tasks need to be accomplished in order to achieve the goal, you can move on to the development phase of ADDIE.  Chances are you have already given some thought to how you want to present your instruction, but now is the time to really focus on the presentation and the tools you will want to use.  In order to get started, you may want to consider which approach to learning is most conducive to your topic.  Within instructional design, there are typically three psychological approaches to teaching and learning: Behaviorist, Cognitivist, and Constructivist. These theories sound extremely academic and complex, but at the root of it all these instructional theories are relative simplistic and applicable to all training.

You've probably heard of the Pavlov and his work with dogs and bells.  Not to simplify his extraordinary
Image courtesy of
work, but his work on psychological conditioning is supposed to be just that - simple.  Behaviorists theorize that there is no way to understand the mind and how one learns.  As a result, conditioning is a prevalent factor in training methods.  You may find it a bit depressing to be likened to the brain of a dog, but the truth is that conditioning is something that happens naturally through associations.  For example, the smell of a particular scent reminds you of a person or the sound of an object makes you recall a specific event.  By associations, we are able to remember things that may seem unimportant to another.

Based on this idea, instructional designers who incorporate behaviorists theories typically use mnemonic devices to stimulate call and response of the brain.  The act of performing a behavior in training is also important.  Typically, training in technology is routed in behaviorist theory.  Often assuming that the learner has no prior knowledge of the subject, a trainer gives instruction while action is taking place.  Theoretically, the simultaneous events of action and instruction should ignite an association with the learner, leading to recall after the instruction has taken place.

Any computer-based instruction where a trainer leads the learner through a course of actions is a direct example of the behaviorist theory in action.

Training has gone through some pretty radical changes in terms of theories used and ethics applied, and the jump from behaviorist to cognitivist theories was a major one.  Cognitivism prescribes to the notion that a learner's environment has everything to do with how he/she is able to learn and retain that knowledge at a later date.  Training developed under the cognitivist approach usually gives a lot of thought to the learner(s) and the best tools to be utilized for teaching.  For example, the use of audiovisual tools, print, images, and even the timing of instruction are given careful consideration before the implementation.  The theory behind the cognitivist approach is rooted in the engagement of the learner and gives credence to the three main learning styles - tactile, auditory, and visual.  It is theorized the more engaged a learner is, the more likely they will be to retain the information.

You will know you are in a cognitivist classroom if the trainer uses many tools to peak your interest.  The use of a video to accommodate visual learners, the use of role play for tactile learners, and the use of music or lecture for auditory learners.

Typically, training that is meant to teach a new policy or thought process may base the instruction in cognitivist theory. 

In the last decade, there has been a shift in education that reflects the use of technology.  Social media and quick access to information has revolutionized how people view learning.  Constructivist theory is based on the fact that each learner comes to the classroom with his/her own viewpoints and experiences which, in turn, shades the way he/she processes the information.  Instead of a subject matter expert, an instructor will take on the role of facilitator.  The learners, as a result, are encouraged to create their own learning environment through discussion and hands-on activities.

Constructivism may also use a variety of tools to engage different types of learners, but the tools will be used differently.  For example, if a video is shown, the instructor is using it to incite discussion among the learners.  Open ended questions are more prominent and group activities where learners can share experiences are used to promote self-instruction.

Much like the cognitivist classroom, constructivism works well with introducing new policies and thought processes to the learners.

As a trainer, you may find that you use one theory more than another.  But more often than not, you may find it helpful to blend the theories together to obtain new outcomes.  Although behaviorism may lend itself to technology instruction, there is no reason why the other two theories cannot be applied.  Knowing your learners and the content will help you decide which classroom is best suited to achieve the objectives.