Vol. 34, Issue 1, Apr 2004
Training and development refers to a variety of educational and learning-based activities that are used to acquire knowledge or skills needed by an employee to perform effectively on the present job. Training activities can vary in complexity – a classroom seminar, computer-based learning, an in-house executive course, individual coaching and mentoring.
Training and development has become an essential component of most organizations for a number of reasons:
A new approach to measuring training and development
Measuring the effectiveness of training and development is often an informal and unsophisticated practice, despite the large amounts of the salary budget allocated to it. Often assessment consists of completing a one or two page evaluation form at the culmination of a training course. The quality of this information to make effective strategic decisions regarding training and development is of minimal value.
Without such information, budgetary decisions to do not ensure an adequate return on the investment of training and development funds. The full potential for training and development to support critical organizational change and to promote improved employee growth/job satisfaction is minimized.
The course management index (CMI) is a new and innovative instrument used to measure the effectiveness of training events from a lifecycle point of view. The lifecycle approach ensures that measurement tracks the progress of a training event from the initial perceptions of participants (baseline measures), through their reaction to its content and delivery (design/delivery measures). Further, it assesses the degree to which participants acquired the desired knowledge and/or skill (knowledge acquisition/skill development measures) and the degree to which the participant applied these abilities on the job (learning application measures). The CMI also tracks the cost of developing and delivering the training event (financial measures).
By using an index approach, the CMI allows the combination of qualitative and quantitative information yet provides the ability to quantify both so that an overall score can be obtained for each training event. Quantitative measures include financial measures. Qualitative measures include design and delivery measures, knowledge acquisition and learning application measures. The use of an index allows these measures to be easily adapted to the unique requirements of each training environment. These dimensions of the CMI can be expanded or decreased in response to changing conditions/experience or the complexity of information required about each training event or particular training environment.
Here is a graphical illustration of the course management index.
Course management index
The CMI lifecycle measures include baseline information, financial indicators, design and delivery measures, knowledge acquisition/skill development measures and learning application measures for each training event.
Each of these CMI measures will be described briefly.
Baseline information is used to provide an assessment of the learning environment, the requirements, and the motivation before an individual attends a training event. By determining the context of the training, it provides a baseline against which application of learning on the job can be assessed. The specific baseline questions relate to the work environment that required the acquisition of these new abilities and skills. Sample questions include the need that motivated participation in this course; the skills, knowledge, values and/or expertise expected from the training event; the frequency with which this new ability will be used; and the knowledge of whether the skill development or behavioural outcomes have occurred.
Financial measures provide a means to determining the most cost effective method of delivering a high-quality training event that will meet the needs of the staff and organization. This involves a consideration of the development, operational, fixed and overhead costs. From such an analysis, it can be decided whether it is more feasible to develop a tailored course offering or obtain it from a commercial vendor. These figures can also be used to track the cost of each training event and compare it with the costs of other course offerings. For those organizations providing training on a cost recovery or profit basis, additional measures provide the means to determine whether each training event is revenue neutral or providing a profit or loss.
Sample measures include actual versus planned development, operational, fixed and overhead costs. Cost recovery or profit/loss measures include course demand increasing, decreasing or steady; actual versus planned number of students per course; number of cancellations (declining or escalating); actual versus planned revenues from course fees; and the cost/revenue ratio per course.
Course design and delivery measures provide an outline of participant reaction to the training event in terms of its content, difficulty, amount of work required, and technical detail. It also provides a rating of the instructor and teaching assistant in terms of organization, preparation, knowledge of the subject area, ability to communicate, and ability to provide remedial assistance. Design and delivery measures also provide an assessment of the course overall in terms of whether or not the course should be recommended or whether another should be taken from the same source.
The learning evaluation measures determine the degree of acquisition of skills or knowledge by the student on a learning event-by-event basis. It provides a means of assessing how well the participant internalized the learning provided by the training event. It can be applied to a certification program such as those inherent in academic programs. It can also be applied to non-credit courses focusing on skill development. Sample measures include graduation/failure rate; dropout rate; average grade on course assignments of participants; average final grade level of participants; and the number of remedial training episodes.
The learning application measures determine the degree of use of knowledge or skill on the job. By comparing baseline information with current usage measures, a fuller picture can be obtained as to the degree of application of the behaviour or skills. Sample learning application measures include the frequency of usage; behaviours/skills most/least used; degree of satisfaction of the business need that initiated the request for training; and the degree of organizational support for the usage of new behaviour/skill.
In order to use the course management index effectively, an implementation strategy is needed with the following elements:
Each will be discussed briefly.
Roles and responsibilities
There are three major sets of roles and responsibilities with regard to the implementation and maintenance of the CMI.
The data collection strategies for the CMI will vary for each of the training event measures as follows:
Analysis of the training event results
An analysis of the CMI performance information provides the means to identify accomplishments and issues for each training event and across the entire training inventory as follows:
For each training event, the analysis of the CMI dimension of performance provides the information to improve each offering by pinpointing the specific areas of improvement necessary to better satisfy participant needs. The issues tend to deal with the design/delivery of the training, knowledge/skill acquisition, or the degree of its application.
For the entire inventory of training event offerings, such analysis permits the identification of training issues by participant groups, by region or over time. It helps determine whether there are gaps between participant expectations and the quality of training event offerings. It provides a means to determine the most cost effective method of delivery and any necessary changes in training philosophy or strategy.
The performance information presented in the CMI outlines accomplishments and issues related to the achievement of the full benefits of training and development across the entire inventory of training events from a financial, design/delivery, knowledge acquisition and application view points. Cumulatively, this information provides a causal link between achieving the continuous learning goals/mission of the organization and the strategic, tactical and operational issues interfering with this goal. Yet, given this information, the basic question remains of how to interpret it to take the necessary corrective actions. In other words “What is normal for my training and development environment?” The interpretation of CMI information can be accomplished using the following three methods.
Each of the CMI training and development measurement perspectives is captured in a training event status report. To ascertain the effectiveness of each training event on a lifecycle basis, the CMI uses a three option summary indicator:
Each training event is given a red, yellow or green rating for each of the CMI measures: financial, design/delivery, knowledge acquisition, and learning application measures. If one of these CMI measures is red, the training event is rated red. If most of the indicators are yellow or green without a red rating, then the training event is given either a yellow or green rating.
Then, an overall training score is given for each training event across the entire inventory of courses. These ratings, then, make it fairly easy to concentrate on those training events that are rated red while ignoring those in the inventory which are rated green. Management of the training becomes focused on attaining excellence in each training event while dealing with those few exceptions that require attention to bring them up to a higher standard. It is management by excellence and exception.
The availability of baseline information provides a historical perspective on the training performance dimensions permitting the analysis of trends over time. This is the most powerful method of interpretation and is required to understand any changes in the training and development organization. It is also the most difficult to obtain. Financial information is likely the only type of CMI measure available during the first year of operation. In subsequent years, year over year comparisons can be made across all performance measurement dimensions of the CMI.
Benchmarking allows comparisons with training and development functions in organizations of similar size and complexity in order to provide a comparative reference point. For example, spending three to four percent of a salary budget on training would seem very high in most organizations. Yet, this is the norm in most information technology organizations. Without this benchmark information, the interpretation of financial training performance indicator would have been faulty. However, useful benchmarking information is typically very difficult to obtain.
Communicating the training and development accomplishments and issues to both senior management and staff is imperative. The delivery of training events that are affordable, accessible and responsive to the needs of its participants can only be achieved if management is able to obtain effective feedback provided through all the measurement dimensions embodied in the CMI. Such feedback provides the information needed to steer the training and development in support of the mission of the organization in general, and towards its continuous learning objectives in particular.
A composite training and development report, summary training event status report (STESR), should be published quarterly. It is used to provide feedback in the form of accomplishments and issues for financial, design/delivery, knowledge acquisition/skill development and learning application measures across the entire training development program.
CMI can be applied to the management of training and development in large public sector branches or departments as a whole. It can be used by organizations specializing in providing training and development services on a certification basis such as colleges or universities or for personal development only. It can range in complexity from the full suite of CMI measures to only those needed in a particular environment. Whatever the level of complexity chosen, several best practices should be used to assist in applying CMI to the unique circumstances of each organization:
Unless the information generated from the CMI is used to take corrective actions to steer the training and development, the implementation effort is wasted. This CMI performance information must be used to move the training and development forward progressively. Where it is demonstrated that CMI performance information is used to improve the functioning of the training and development, then the CMI will become fully self-sustaining.