Submitted by BPI’s Chicago Associate, Mr. Alan Scott, who read about the technique in the book Elevators by John Noonan.
No matter where this idea came from it may be a means to meet a significant challenge for training professionals. Even at the more “enlightened” companies (who don’t regard their training department (aka: Human Resource Development; or Talent Development Department) as a cost center but rather as an investment center) quantifying the value of their training “investment” is a difficult task. We also recommend the interested reader to a more recent publication on this topic in the book: Let’s Get Real Or Let’s Not Play by Khalsa and Illig.
Donald Kirkpatrick’s Four Levels of Training Evaluation (K1-4) is given lip service as the industry standard for measuring training success (since 1979).
I. K-1: measures participants’ reaction to the training program (aka: smile sheets) and
II. K-2: measures the learning that has occurred, usually through pre- and post-testing.
III. K-3: measures behavior changes back on the job and
IV. K-4: The seldom measured Return On Investment.
Lip service aside, per the Learning Resources Network, 77 percent of organizations do K1-reaction measures; 36 percent add K2 learning evaluations; 15 percent add K3-behavior change; and only 08 percent do all four levels of evaluation to measure results.
That’s right – fewer than 15 percent of our colleagues’ measure behavior changes or return on investment from even one of their training programs. Yet, behavior changes and ROI are the very things almost every organization wants from spending time and money on trainings. Despite this justifiably high level of interest, some 85% of the HRD or Talent Development professionals are unable to provide them with any hard data in these areas.
Why not? I propose that:
- It’s hard to do and
- HRD and Talent Development have zero formal incentive for measuring K3 or K4 results.
“One of the key challenges is quantifying soft data. For instance, how do you put a numerical measure on a new process’s ease of use? How do you measure changes in attitude? Work habit? Effectiveness of communication?
“While no system is perfect, some new tools have been developed to help us get around these obstacles. To remove the guesswork, we wanted to find a method for evaluating K3 and K4 using known data. We recently discovered a tool that uses participant salary and benefits costs to show one type of training impact. This information, which most training organizations have or can easily obtain, is combined with a self-assessment by the training participant, the person most familiar with their work situation.
Self-Assessment Of Training Results –only 3 questions!
This self-assessment method for Kirkpatrick Levels is reported to have been validated per the Noonan book. This method may be used to estimate training effects on the work behavior of participants K3 and begin to quantify K4 organizational impact.
The following is an example of the type of questions that may be used for self-assessment. These questions were written for our one-day Systematic Project Management workshop. Modify these for your own workshops.
[T] What percent of your total working time will you spend on tasks that require the skills or knowledge of the Systematic Project Management workshop? (circle one)
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100%
[P1] Rate your productivity before the Systematic Project Management workshop on job tasks that require the skills and knowledge of this course. (circle one)
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100%
[P2] Rate your productivity after the Systematic Project Management workshop on job tasks that require the skills and knowledge of this course. (circle one)
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100%
Once these percentages have been determined, we can plug them into the following impact formula:
Here is a sample calculation for a hypothetical team of four members who attended a job-related workshop. (V results should be multiplied by .73.)
*Multiply salary by 1.5 to estimate salary and benefits
More good news! This formula method drastically underestimates the real impact of training. It assesses only labor costs or savings. But, labor is used to produce revenue. If someone is made more effective at producing revenue (throughput) the impact is an increase in revenue at the same salary and benefits.
If someone is being paid $50,000 and their productivity improves by 10% the method shows a $5,000 net benefit through labor cost savings or expansion of the workforce with no extra costs. However, if being 10% more productive earns the organization new revenue, that is not accounted for. The formula sums reduced costs or labor expansion independent of any revenue gains equal to an average of
$2,500 per participant per year, year after year, by just attending the training once.
Example A: Suppose your job involved solving sales problems and then teaching 100 salespeople to sell more effectively. Making you more effective in solving sales problems would allow you to improve sales leveraged 100 fold. Solving just one sales problem could dwarf your entire salary and benefits package year after year!
Example B: (Based on a real project but using hypothetical figures): A five-person team responsible for installing a new radiology unit at a hospital attended a BPI Critical Thinking workshop after the project was launched. At first, they estimated the project would be 45 days late. After using the BPI Planning process, they completed the project on time! The organization realized a $450,000 gain in revenues from using the unit 45 days sooner. This would be in addition to the recapture of the team’s labor during those 45 days. BENEFIT: The workers devoted 50% of their time to the project. So, the organization gained an estimated 50% of their time (45 days) to be devoted to other work plus $450,000 in lost revenue or an estimated $150,000 in net profit on that revenue.
Conclusion: This impact formula has been validated for labor cost effects and is an easy way to demonstrate Level K3 and K4 results. But, extending this formula to include revenue/profit effects would demonstrate results one or two orders of magnitude higher.
BPI Workshops:
You can document savings with BPI workshops! One method we’ve used to document training impact is to have participants bring issues to class with known values to the organization. Resolving these issues in class demonstrates cost avoidance and revenue gains. Our workshops have consistently demonstrated multiple returns on the training investment within the workshop itself. The benefit of resolving just one major issue can easily repay all training costs including the salaries and benefits covering the time away from work of everyone attending the workshop several times over.
We guarantee a ten-fold return on your Critical Thinking training investment from problems resolved and decisions made during the workshop itself.
After the workshop, further gains can be realized on other issues and assessed using the methods discussed here, endlessly! Competence is the gift that keeps giving!
Troubleshooter-I Certificate Program
Measuring Training Impact
Business Processes Inc. * R & D * P.O. Box 1456 * La Jolla, CA 92038
www.critical-thinking.com