How to Evaluate your company's Training efforts

How to Evaluate your companys Training effortsPageIntroductionThe evaluation bridge a modelThe importance of evaluationSection I ReactionPitfalls to avoidGuidelines to followSummarySection II LearningHow to measure LearningClassroom performancePaper & pencil testsSummarySection III Behavior ChangeAcquisition of new behaviorMaintenance of new behaviorHow to evaluate behavior changeBehavior evaluation methodsSummarySection IV Organizational ResultsHow to assign value to a training effortHow to use your value calculationsSummarySection V Management Support GuidanceHow to measure current management commitmentHow to increase management commitmentHow to increase supervisory supportConclusionINTRODUCTIONThis guide is designed to help you answer the very simple question, “How do I know my training programs working?”Youve developed a pretty good employee development program. Youre trying to meet the training needs of all levels of employees and youve gotten enthusiastic response to the sessions youve held so far. But us your training program really working?How do you know your program is on target? How do you know its meeting the needs of all employees? And how do you begin to justify tge cost to upper management?Only an effective evaluation program can lead you to those answers. Only evaluation can bridge the gap between training and the results of training, and direct you toward areas which need improvement or change.But between those two shores of training and known results, lies the rough water of expensive evaluation surveys, statistical analyses and employee questionnaires. The waves of time and energy crash against your feet. How do you cross safely without drowning in inconclusive summaries and irrelevant data?The Evaluation Model A “Bridge over Trouble Water”Management SupportCost Time - EnergyReactionLearningBehavior ChangeOrganization ResultsThis guidebook is designed to provide the bridge across the rough water of employee training evaluation.As the Evaluation Model illustrates, evaluation is like a bridge built in four spans, each span following upon the one before it. It begins with employee reaction, then follows with evaluation of learning, behavior change and finally, organizational change.You can incorporate one, two, or even three spans into your evaluation process. But only the combination of all four spans together will bring you to the results you need.The Evaluation Bridge is built on a sure foundation management support and guidance. Without this foundation no evaluation and no training program can stay above water. This guide will show you how to measure the support you now have and how to increase management commitment to your training effort.Measuring the evaluating employee training can be a challenging task. But, by using the tolls in this guide, you too can bridge your trouble waters to a more targeted and effective training effort.THE IMPORTANCE OF EVALUATIONEveryone agrees that evaluation is necessary following a training session, yet how much evaluation effort is necessary? Why go beyond asking “Did you like the seminar?”Measuring trainee reaction, the first span on the Evaluation Bridge is the easiest evaluation to do, but it may be the least valuable to the organization. A brief questionnaire at the end of the training session can tell you the group liked the program. It cant tell you the dollar savings to the company which will result from increased employee productivity. Only a thorough evaluation process can give you that kind of information. And its the tangible dollar results of training, not just employee reactions, that management needs and wants from its training program.Thats why the Evaluation Bridge goes beyond measuring employee reaction. In the Learning stage, the evaluation becomes progressively more difficult to measure. This stage measures the learners achievement of the program objectives. Additionally it demonstrates what principles, facts, skills and attitudes have been obtained from the training effort.A change in an individuals performance on the job following the training program is the next stage in the Evaluation Model. This evaluation consists of an observable behavior change by the supervisor and by other employees.Finally, Organizational Results, while the most difficult to measure, are the most valuable evaluation level. This is the point we want to achieve in the evaluation effort. It measures bottom line results of training for the organization. It helps determine the return on investment for the training effort. But most important it measures the value of the training program. This establishes that the learned behavior has been used for on-the-job application and has achieved results. The following chart illustrate that the evaluation results achieved corresponds directly to the amount of time and dollar invested in evaluation.Quality of DataLevel of CredibilityHighLowHighReactionLearningBehavior ChangeOrganizational ResultsCost of Evaluation & Time ExpendedA thorough evaluation program is important for four reasons:1. To enlist management supportA recent study conducted by Mitchell Edward Kusy, Jr., at the Training and Development Research Center at the University of Minneapolis in St. Paul. Found that a results-oriented approach to evaluation received that most management support.Four training evaluation methods were studied: reaction, learning, behavior change and organizational change. The organizational method received overwhelming management support more than 80 percent of the managers preferred results-oriented accountability, with progressively less support for behavior, learning and reaction respectively.Increasingly, training managers must prove to management that training can offer a significant return on investment for the training dollars spent. Its essential to use the kinds of evaluation methods which will generate that support to build the kind of training program your company needs.2. To assure improved training effortsEvaluation results can guide you to changes, improvements or cancellations of an existing training program which may be inefficient or unproductive. This in turn, may be an indicator that further training in a specific area, which is not presently provided, would be beneficial.3. To provide improved trainer confidenceEvaluation can result in increased self-confidence and personal satisfaction for the trainer. Evaluation can give the trainer important feedback and concrete evidence that he or she is accomplishing tangible results for the organization.Such feedback helps the trainer grow, spot and correct weakness, and build in strengths. It is vital for keeping the trainer and the training effort on target.4. For the continued survival of the training functionFinally, evaluation is important for the ultimate survival of the training function in a company. Organizations need to know whether their investments in training are worth their time, money and effort. Your ability to give management specific results, in the language management speaks, will enhance your credibility and the credibility of your training program. Only a thorough evaluation program can measure the effectiveness and worth of the training function, and for you, thats the bottom line.Management SupportCost Time - EnergyReactionLearningBehavior ChangeOrganization ResultsTrainingResultsI ReactionThe first span in the Evaluation Bridge is the measurement of employee reaction to the training which has just taken place. This measurement is concerned with the emotional reaction of the employee, or in measuring the “heart-response.”Reaction can be defined as how well the trainee liked a particular training program. It does not include a measurement of any learning which may have taken place.Reaction is usually measured with participant end-of-course questionnaires, sometimes called “happiness sheet.” Since reaction is relatively easy to measure, nearly all training directors use them, despite several drawbacks. First, its difficult to obtain meaningful and unbiased assessment of the usefulness and effectiveness of the training program with a single, multiple-choice questionnaire. Second, the reaction form often invites generous ratings of the training sequence. And third, the forms are usually administered to participants at the end of an exhausting training experience, resulting in less-than-complete responses.However, reaction sheets can be an effective evaluation tool of properly constructed. Lets look more closely at the pitfalls to avoid and guidelines to follow when drafting a reaction questionnaire.Pitfalls to AvoidThere are five common pitfalls to avoid when designing your reaction sheet:1. Too complex rating scale2. Unclear wording3. Incorrect assumptions4. Questions which affect response5. Complex questions1. Too-complex rating scaleRefer to Slide #3The one-to-nine or one-to-seven rating system, as shown in Illustration 1, are common forms of evaluation, yet they have several disadvantages.The complex scale asks for fine distinctions. The difference between “strongly agree” and “very strongly agree” may not be clear in the participants mind. Be careful not to ask your trainees to make too fine a distinction, especially for very general questions such as “Rate the teaching techniques used.”A five-point scale, from “5” (high degree) to “1” (low degree) is often a better measurement.2. Unclear wordingThe wording of the questions is important, because it can often affect the participants response. For example, the second question in Illustration 1 refers to “high expectations.” However, not all participants may have begun with high expectation. They may be unsure as to how high a “high expectation is.The third question in Illustration 1 would also affect the participants response. There is no definition given for “teaching techniques.” Does it include clarity, humor, use of audio or visual aids? Define or explain any terms which may influence trainee response.3. incorrect AssumptionAlways double-check the assumptions that underlie your questions. As yourself, “Is this really what I want to ask?”For example, in Illustration 1, the first question has an assumption that could be stated: “Its the trainers job to make this course interesting, because the participants cant be counted on to be interested in the materials on their own.” That assumption will cloud how the question is answered.The second question assumes that participants arrived with high expectations. Expectations can vary greatly one persons high hopes could be moderate interest on the part of another. Its unreasonable to generalize about expectations.The third question assumes that participants have a background in education or traiing. Just because youre a student, doesnt make you an expert on how to teach. To rate teaching techniques, the trainee should be able to know and compare various teaching techniques. Dont assume participants have attended other workshops or been involved in similar programs.4. Questions which affect responseRefer to slide #4Please circle oneHigh degreeLow degree1. Did this workshop meet your needs and objective?543212. Was the instructor knowledgeable about the topic?543213. Will you be able to apply the content directly to your job?543214. Would you recommend this course to a friend, co-worker, or supervisor?543215. Were the meeting facilities satisfactory?54321Please circle oneHigh degreeLow degree1. Did this workshop meet your needs and objective?543212. Was the instructor knowledgeable about the topic?543213. Will you be able to apply the content directly to your job?543214. Would you recommend this course to a friend, co-worker, or supervisor?543215. Were the meeting facilities satisfactory?54321Please circle oneHigh degreeLow degree1. Did this workshop meet your needs and objective?543212. Was the instructor knowledgeable about the topic?543213. Will you be able to apply the content directly to your job?543214. Would you recommend this course to a friend, co-worker, or supervisor?543215. Were the meeting facilities satisfactory?54321Questions can be structured in such a way that the trainees response will be influenced. In the above illustrations, for example, the five commonly asked usually result in a tendency to choose answers from the same range of numbers.If you give question one a low rating, for example, you will probably respond with a low rating for the following questions as well. This response pattern is illustrated in above second example.What these results tell you is that you are not getting an object answer to all questions and that you have no way of telling whether or not the answers have been given much thought.5. Complex QuestionsA quick response to a complex question will tell you little. For example, in second illustration, questions one asks “ Did this workshop meet your needs and objectives?”What this questions is really asking is :”1.) Evaluate your personal needs and objectives and, 2.) rate how the workshop measured up to those needs.”How should the trainee respond of he doesnt know his own needs? How should she respond of she hasnt had time to properly assess the workshop?When learning new information, people need time to think through their new ideas. Do not expect participants to be able to make immediate transference of new information to a present job situation however much you would like to have that kind of information.Do not ask participants for evaluations they cannot be expected to make. It will only result in incorrect or inadequate feedback.Guidelines to FollowYou evaluation goal is to obtain the truest measure possible of the initial reaction to your training session. Skillfully designed questions can give you that measurement.There are four guidelines to consider when drafting your reaction evaluation:1. Cover both training process and impact;2. Include essay and scaled questions;3. Accumulate reactions;4. Have supervisor also complete questionnaire; and,5. Should be anonymous, if possible, to maximize candor.1. Cover both training process and impactYour questionnaire should contain questions which measure both process and impact. This includes such aspects of the training process as participant attitudes (both before and after training), the effectiveness of the teacher or leader, and the environment in which the session was held. The training impact includes and evaluation of the content of the program, its onjective and its relevance for the participants.2. Include essay and scaled questionsBy using a combination of types of questions, you solicit more complete answers than by using only true and false or only a five-point rating scale.There are basically five types of questions: the open-ended question; the two-way question; the multiple choice questions; the checklist; and, the ranking scale.Open-ended or essay questions ask the participant to express what they are feeling or reason behind a particular opinion. They cannot be answered by a yes or no, and generally need medium to short answer responses. Analyzing these answers may be more difficult because of their variety, but they are valuable for soliciting subject viewpoints. They typically begin with “what,” “who,” “when,” “how” or “where.” For example, you may ask: “What problems have you encountered in using the team building skills presented in the workshop?”The two-way ot two choice question directs the participant to a specific answer, either “Yes” or “No,” or “True” or ”False. These questions can be answered quickly and analyzed easily. The two-choice questions may sometimes be followed by and open ended questions for a more complete reply. For example: Did the workshop leader speak clearly and distinctly?Yes _No _If No, explain:_The multiple choice question asks a participant to respond by choosing one of several answers. The replies to multiple choice questions can be easily analyzed. However, unlike the open-ended questions, they give no indication of the thought process behind the answer chosen. For example:Training cost calculation should include:A. SalariesB. Fringe ExpensesC. Travel ExpensesD. Preparation TimeE. All of the aboveThe checklist, like the multiple choice questions is easily analyzed. It is particularly appropriate for determining a broad range of attitudes or opinions, or for soliciting factual information. To increase the effectiveness of the checklist, a blank space should be left at the end of the list to allow for participants to add pertinent information. This will give you a clearer picture of the thought process involved in answering the questions. For example:Place a check beside the segments of the course you found most relevant to your job:_Importanc