This is the fourth in a series of posts on the Kirkpatrick Model of Evaluation. Previous posts have provided an overview of the Kirkpatrick model and a detailed look at both Kirkpatrick level 1 and Kirkpatrick level 2. This post will focus on the detail of level 3.
What is Kirkpatrick level 3 evaluation all about?
The keyword that Kirkpatrick uses to describe this level of evaluation is ‘Behaviour’ (2006).Kirkpatrick, D & Kirkpatrick, J. (2006). Evaluating Training Programs (Third Edition). San Francisco: Berrett-Koehler Publishers, Inc. Donald & Jim Kirkpatrick define level 3 evaluation as determining how much transfer of knowledge, skills and attitudes has occurred following a training programme – i.e. how has behaviour in the workplace changed as a consequence of training.
If you’ve read my Kirkpatrick level 1 and Kirkpatrick level 2 summaries you may have already concluded that evaluation at this level is more complicated and resource intensive than the earlier levels.
When should I be conducting evaluation at Kirkpatrick level 3?
One of the reasons that evaluation at Kirkpatrick level 3 is more complex is because the process is harder for trainers to control. In order to be able to determine if individuals have transferred the training they have received you have to wait until the trainees have spent some time back in the workplace. The amount of time you have to wait will depend on numerous factors, including:
- the nature of the training.
- the opportunity available to implement the knowledge/skills.
- the level of encouragement from line management.
How and what should I evaluate at Kirkpatrick level 3?
Once you have decided how long to leave post training you will need to decide what methodology you’re going to use to collect your evaluation data. In my opinion it is important to have a systematic methodology, although research by ESI suggests that many organisations take a more ad-hoc approach when measuring the impact of learning.
Questionnaires and interviews are common methods to use but the techniques that you decide to use will need to be appropriate to the amount of resources that you’re prepared to commit. Remember, it is not always necessary to conduct evaluation at all levels for all your training – I have developed my own system called EVAL-IT? to help me prioritise training for evaluation.
Kirkpatrick Level 3 – opportunities to exploit:
There are a number of aspects of evaluation at level 3 that you can take advantage of:
- Key data can be collected to inform how training is been implemented or not. Decisions can then be made on how to improve the transfer rate.
- Enables direct engagement with the operational side of the business, this assists in developing relationships and maintaining the links between training and business requirements.
- Identifies areas of training that may no longer be relevant (removing these elements of training can reduce the length of the training and therefore reduce costs).
- Identifies gaps in the training where skills or knowledge need to be incorporated in order to more fully prepare individuals for their role.
Kirkpatrick Level 3 – threats to mitigate against:
Whilst there are advantages, there are also some issues that you should be aware of:
- Lack of consistent methodology can create difficulties in comparing results across training programmes.
- There can be a tendency to rely on ‘anecdotal’ evidence – this may not be sufficient for your board and should be reinforced with more scientific data collection.
- Can uses excessive amounts of resource if not focussed efficiently, resulting in the potential for costs to outweigh the benefits – use a technique like EVAL-IT? to prioritise.
- The wider orgnaistaion may not ‘buy-in’ to your evaluation efforts, concluding that “completing your questionnaires are a waste of time”. To overcome this problem you’ll need to actively engage across the workforce – check out who should be involved in training evaluation for more info.
Linking Kirkpatrick level 3 data with level 1 and 2 results…
One of the themes of the Kirkpatrick model is the concept of ‘generating a chain of evidence’ between the levels. The idea behind this principle is that you should be able to compare the different results that you have collected across levels 1, 2 and 3 for a particular training programme to see if you can identify any themes.
Results that show that participants who were dissatisfied with the training (level 1) and failed to demonstrate that they had assimilated the subject matter (level 2) are unlikely to then demonstrate effective transfer (level 3).
However, you may generate high satisfaction results (level 1) and effective knowledge gain (level 2) but still not achieve effective transfer (level 3). At this point you can then look to analyse the reasons why. Perhaps your training the wrong stuff? Perhaps line managers don’t agree with the ‘new way of doing things’ and are denying the opportunity for trainees to implement their new skills? When you come up against negative results that give you cause for concern you’ll need to intensify your research to determine the true cause before you can implement an action plan to improve the situation.
It’s more complicated to achieve but that doesn’t mean we shouldn’t do it…
The Kirkpatrick’s have made claims in the past that level 3 is the “missing link” in evaluation. Trainers will attempt to do the stuff they can control at levels 1 and 2, and may even attempt to demonstrate some links at level 4, but they fail to consider how effectively training is been transferred.
Yes it takes time and energy. Yes it means engaging with the wider workplace. Despite these facts, it should be conducted for those programmes that you (and your major stakeholders) believe are key for organisational success. If your employees are spending time on expensive training programmes but are not behaving differently at work as a consequence then what is the point?
To effectively conduct level 3 evaluation I believe you need to decide what methodologies you’re going to use to conduct level 3 evaluations. Engage with the relevant people and explain what you’re going to do (and why). Get on and do it.
Do you have any experience of conducing level 3 evaluations? Do you think level 3 is a waste of time? Please share your thoughts in the comments below.