The Implementation of the Course Management Index (CMI) –
A Lifecycle Approach to Measure the Effectiveness of Training and Development

Bryan Shane

Introduction

Training and development refers to a variety of educational and learning-based activities that are used to acquire knowledge or skills needed by an employee to perform effectively on the present job. It ranges in complexity from a classroom seminar, to computer based learning, to an in-house executive course, or individual coaching and mentoring. It provides the means to maintain the organization’s competitive advantage by retaining current staff and enabling the attraction of new staff.. With training and development, employees can improve the quality of their work, make fewer mistakes, evolve towards more progressive responsibilities and serve their clients more effectively. Thus, training and development is a major enabler to achieving the mission and continuous learning objectives of the organization.

Yet, one of major problems that traditionally help to reduce its efficacy, is the effective assessment of training and development. Despite the millions of dollars of the salary budget (usually 2-3%) allocated to this function, evaluation of training and development is often an informal and unsophisticated practice Without high quality information to make effective decisions regarding training and development, the function tends to be reactive/ad hoc, and fails make a measurable contribution to developing skilled/productive staff or to the long term success of the organization.

Course Management Index (CMI)

The Course Management Index (CMI) is new and innovative instrument used to measure the effectiveness of training events from a life cycle point of view. The life cycle approach ensures that CMI will track the progress of a training event from the initial perceptions of participants (baseline measures), through their reaction to its content and delivery (design/delivery measures). Further, it assesses the degree to which participants acquired the desired knowledge and/or skill (knowledge acquisition/skill development measures) and the degree to which the participant applied these abilities on the job (learning application measures). The CMI also tracks the cost of developing and delivery of a training event along with demand for a training event (financial measures).

The CMI allows the collection of a combination of qualitative and quantitative information yet provides the ability to quantify both so that an overall score can be obtained for each training event. Quantitative measures include financial measures. Qualitative measures include design and delivery, knowledge acquisition/skill development and learning application. The use of an index allows these measures to be easily adapted to the unique requirements of each organization’s training environment. These dimensions of the CMI can be expanded or decreased in response to changing conditions/experience, and the complexity of information required about each training event or environment.

The CMI is used to ascertain the effectiveness of each training solution from its initial offering through its completion. It provides feedback on how well each training event contributes to participant learning, and application of learning in the work environment. More specifically the CMI:

• Serves as a guide to budgetary decisions justifying the expansion or revamping of the existing training and development program.
• Provides the basis to improve program elements as objectives, training methods/materials, and outcomes.
• Provides the basis for future strategies and plans by providing feedback on the success or failure of existing training and development initiatives and any new opportunities for future needs.

In short, the CMI provides the information necessary to improve decision making with regard to each training event, and to enable proactive problem correction and promote continuous improvement across the complete inventory of training courses. It provides the information necessary to harness the full potential of training as an essential strategy in managing ongoing organizational transitions related to personnel, business processes, information technology, structure and policy.

See Diagram 1 for a graphic illustration of the Course Management Index:

CMI Measures

The CMI life cycle measures include: baseline information, financial measures, design and delivery measures, knowledge acquisition/skill development measures; and learning application measures for each training event

Each of these CMI measures will be briefly described:

• Baseline Information is used to provide an assessment of the learning environment, requirements, and motivation before an individual attends a training event. By determining the context of the training, it provides a baseline against which the application of learning on the job can be assessed.
• Financial Measures provide the means to determine the most cost-effective method of delivering a high-quality training event that meets the needs of the participants and the organization. It is also used to gauge the demand for a training event. From such an analysis, it can be decided whether it is more feasible to develop tailored course offerings or obtain them from a commercial vendor.
• Design/Delivery Measures provide an outline of participant reaction to the training event in terms of its content, difficulty, amount of work required and technical detail. It also provides a rating of the instructor and an assessment of the course overall in terms of whether or not the course would be recommended or whether another should be taken from the same source.
• Knowledge Acquisition/Skill Development determines the degree of acquisition of skills or knowledge by the participant.  It provides a means of assessing how well the participant internalized the learning provided by the training event.
• Learning Application Measures: The learning application measures determine the degree of use of knowledge or skill on the job. By comparing baseline information with current usage measures, a fuller picture can be obtained as to the degree of application of the behavior or skills.

Implementation Strategy

In order to use the Course Management Index effectively, an Implementation Strategy is needed with the following elements:

• Roles and responsibilities of management, training administrators and training participants;
• Methods of data collection for performance measures previously described;
• Methods of analyzing the results to determine accomplishments and issues which influence the effective use of training;
• Interpretation of information to determine what is a normative level of training performance for the organization; and
• Methods for communicating the results of the CMI.

Each will be discussed briefly.

Roles and Responsibilities

There are three major sets of roles and responsibilities with regard to the implementation and maintenance of the CMI.

• The management of the organization is responsible for providing the leadership necessary to develop, implement and maintain the new approach to measuring the effectiveness of training and development. Management must allocate the resources necessary to carryout training and development within the organization and use the results from CMI to improve its management practices.
• The administrators of training and development are responsible for developing, implementing and maintaining the new training evaluation approach using the CMI.  This includes: customizing CMI measures; setting up or amending business processes to collect the CMI data; analyzing CMI information; interpreting and reporting the results. More information is provided upon these functions in upcoming sections.
• The training event participants provide the facts required for baseline information,; design/delivery; and knowledge application. Instructors provide information on knowledge acquisition/skill development.

Data Collection Strategies

The data collection strategies for the CMI will vary for each of the training event measures as follows:

• Baseline Information can be collected through the Course Registration Form or using a questionnaire at the beginning of each training event.
• Financial Measures can be collected from the financial or training information systems or course registration records.
• Design/Delivery Measures are collected from student Evaluation Forms completed at the end of each course. Participants’ opinions are given on a 5-point scale with 1 being the lowest and 5 the highest.
• Knowledge Acquisition/Skill Development Measures are collected from course completion records for such elements as graduation/failure rates, dropout rate and overall final mark.
• Learning Application Measures are collected through a questionnaire sent to a sample of training event participants three to six months after its completion. Participants’ opinions are given on a 5 -point scale with 1 being the lowest and 5 the highest.

Analysis of the Training Event Results

An analysis of the CMI performance information provides the means to identify accomplishments and issues for each training event and across the entire training inventory as follows:

• For each training event, the analysis of the CMI dimension of performance provides the information to improve each offering by pin pointing the specific areas of improvement necessary to better satisfy participant needs. The issues tend to deal with the design/delivery of the training, knowledge/skill acquisition, or the degree of its application.
• For the entire inventory of training event offerings, such analysis permits the identification of training issues by participant groups, by region or over time. It helps determine whether there are gaps between participant expectations and the quality of training event offerings .It provides a means to determine the most cost effective method of delivery, and any necessary changes in training philosophy or strategy.

Interpretation

The performance information presented in the CMI outlines accomplishments and issues related to the achievement of the full benefits of training and development across the entire inventory of training events from a financial, design/delivery, knowledge acquisition and application view points. Cumulatively, this information provides a causal link between achieving the continuous learning goals/ mission of the organization and the strategic, tactical and operational issues interfering with this goal. Yet, given this information, the basic question remains of how to interpret it to take the necessary corrective actions. In other words “What is normal for my training and development environment“. The interpretation of CMI information can be accomplished using the following three methods:

Rating Systems

Each of the CMI training and development measurement perspectives is captured in a Training Event Status Report. To ascertain the effectiveness of each training event on a life cycle basis, the CMI uses a three option summary indicator:

• Green Rating:  The green rating indicates everything is functioning according to plan. There are no major concerns with the training event.
• Yellow Rating: A yellow rating is assigned where there are issues that need to be addressed in order to deliver the training event according to participant / organizational expectations.
• Red Rating: A red rating indicates that there are major weaknesses or concerns with a training event that need to be addressed.

Each training event is given a Red, Yellow or Green rating for each of the CMI measures: financial, design/delivery, knowledge acquisition, and learning application measures. If one of  these CMI measures is Red, the training event is rated Red. If most of the indicators are Yellow or Green without a Red rating, then the training event is given either a Yellow or Green rating.

Then, an overall Training Score is given for each training event across the entire inventory of courses. These ratings, then, make it fairly easy to concentrate on those training events that are rated Red while ignoring those in the inventory which are rated Green. Management of the training becomes focused on attaining excellence in each training event while dealing with those few exceptions that require attention to bring them up to a higher standard. It is management by excellence and exception.

Use of Baseline Information

The availability of baseline information provides a historical perspective on the training performance dimensions permitting the analysis of trends over time. This is the most powerful method of interpretation and is required to understand any changes in the training and development organization. It is also the most difficult to obtain. Financial information is likely the only type of CMI measure available during the first year of operation. In subsequent years, year over year comparisons can be made across all performance measurement dimensions of the CMI.

Use of Benchmarks

Bench marking allows comparisons with training and development functions in organizations of similar size and complexity in order to provide a comparative reference point. For example, spending 3 to 4% of a salary budget on training would seem very high in most organizations. Yet, this is the norm in most information technology organizations. Without this benchmark information, the interpretation of financial training performance indicator would have been faulty. However, useful benchmarking information is typically very difficult to obtain.

Experience with the Organization

Organizational experience is also a key requirement to interpret any training performance information. Major changes in training and development must be interpreted within the context of the history of the organization as a whole. For example, spending on training and development during 1999/2000 fiscal year dropped significantly across many departments in the federal public service. Knowing, however, that the majority of operational and capital funding was spent to deal with Y2K issues during that year and that spending levels returned to more normal levels during subsequent years, provides a powerful interpretation on this information.

The most effective interpretation of CMI performance measurement information can be obtained from using as many of these techniques as possible, in combination.

Communicating

Communicating the training and development accomplishments and issues to both senior management and staff is imperative. The delivery of training events that are affordable, accessible and responsive to the needs of its participants can only be achieved if management is able to obtain effective feedback provided through all the measurement dimensions embodied in the CMI. Such feedback provides the information needed to steer the training and development in support of the mission of the organization in general, and towards its continuous learning objectives in particular.

A composite training and development report – Summary Training Event Status Report (STESR) should be published quarterly. It is used to provide feedback in the form of accomplishments and issues for financial, design/delivery, knowledge acquisition/skill development and learning application measures across the entire training development program.

Application of the CMI   

• CMI can be applied to the management of training and development in large public sector branches or departments as a whole. It can be used by organizations specializing in providing training and development services on a certification basis such as colleges or universities or for personal development only. It can range in complexity from the full suite of CMI measures to only those needed in a particular environment. Whatever the level of complexity chosen, several best practices should be used to assist in the applying CMI to the unique circumstances of each organization including:

• The development and implementation of CMI is both an Art and a Science.  The Science of developing and implementing the CMI involves the use of a basic set of steps previously described to provide similar philosophy, structure and process so that comprehensive information can be provided across all the complete inventory of training events. The Art of developing and implementing the CMI involves tailoring it to the unique requirements of each organization in terms of specific measures, timing, sequence of activities, and knowledge transfer.
• The Use of a Pilot provides evidence of the utility of the CMI performance measurement approach. It gradually builds acceptance/support for this approach to assessing training and development. It also provides the experience needed to tailor the CMI to the unique requirements of the organization while developing the in-house expertise necessary to maintain it.
• The development and implementation of the CMI must begin from the top of the organization and be implemented progressively downwards. It requires the full commitment and support of management in terms of providing a supportive environment, communicating the necessity and importance of this approach. It also requires a financial investment to develop, implement and maintain this operation.
• Adopt a “Just Do It” approach to the development and implementation of the CMI. Don’t be tied to expensive data gathering and implementation approaches. Use “Work Around” strategies to find simple but effective solutions. Where progress is delayed on development and implementing the measurement of one CMI perspective, focus more attention on others. Since the whole process is self- improving and self- correcting, the key is to go with whatever information, processes, interpretation methods you have developed.
• Where there are large numbers of participants in training events, sampling techniques can be used to reduce the amount time and effort associated with the gathering, analysis and interpretation of CMI training information.

Unless the information generated from the CMI is used to take corrective actions to steer the training and development, the implementation effort is wasted. This CMI performance information must be used to move the Training and Development forward progressively. Where it is demonstrated that CMI performance information is used to improve the functioning of the training and development, then the CMI will become fully self-sustaining.

Four of the measurement perspectives were adapted from the work of DL  Kirkpatrick (participant reaction, learning, behavior, results)

For a list of sample performance measures for each of the measure dimensions of the Course Management Index refer to the first article published in this series.

Scoring Guidelines for each of the measurement perspectives of the CMI must be developed based upon the unique requirements of the training environment in each organization.