Training and development refers to a variety of educational and learning-based activities that are used to acquire knowledge or skills needed by an employee to perform effectively on the present job. It ranges in complexity from a classroom seminar, to computer based learning, to an in-house executive course, or individual coaching and mentoring. It provides the means to maintain the organizationâ€™s competitive advantage by retaining current staff and enabling the attraction of new staff.. With training and development, employees can improve the quality of their work, make fewer mistakes, evolve towards more progressive responsibilities and serve their clients more effectively. Thus, training and development is a major enabler to achieving the mission and continuous learning objectives of the organization.
Yet, one of major problems that traditionally help to reduce its efficacy, is the effective assessment of training and development. Despite the millions of dollars of the salary budget (usually 2-3%) allocated to this function, evaluation of training and development is often an informal and unsophisticated practice Without high quality information to make effective decisions regarding training and development, the function tends to be reactive/ad hoc, and fails make a measurable contribution to developing skilled/productive staff or to the long term success of the organization.
Course Management Index (CMI)
The Course Management Index (CMI) is new and innovative instrument used to measure the effectiveness of training events from a life cycle point of view. The life cycle approach ensures that CMI will track the progress of a training event from the initial perceptions of participants (baseline measures), through their reaction to its content and delivery (design/delivery measures). Further, it assesses the degree to which participants acquired the desired knowledge and/or skill (knowledge acquisition/skill development measures) and the degree to which the participant applied these abilities on the job (learning application measures). The CMI also tracks the cost of developing and delivery of a training event along with demand for a training event (financial measures).
The CMI allows the collection of a combination of qualitative and quantitative information yet provides the ability to quantify both so that an overall score can be obtained for each training event. Quantitative measures include financial measures. Qualitative measures include design and delivery, knowledge acquisition/skill development and learning application. The use of an index allows these measures to be easily adapted to the unique requirements of each organizationâ€™s training environment. These dimensions of the CMI can be expanded or decreased in response to changing conditions/experience, and the complexity of information required about each training event or environment.
The CMI is used to ascertain the effectiveness of each training solution from its initial offering through its completion. It provides feedback on how well each training event contributes to participant learning, and application of learning in the work environment. More specifically the CMI:
In short, the CMI provides the information necessary to improve decision making with regard to each training event, and to enable proactive problem correction and promote continuous improvement across the complete inventory of training courses. It provides the information necessary to harness the full potential of training as an essential strategy in managing ongoing organizational transitions related to personnel, business processes, information technology, structure and policy.
See Diagram 1 for a graphic illustration of the Course Management Index:
The CMI life cycle measures include: baseline information, financial measures, design and delivery measures, knowledge acquisition/skill development measures; and learning application measures for each training event
Each of these CMI measures will be briefly described:
In order to use the Course Management Index effectively, an Implementation Strategy is needed with the following elements:
Each will be discussed briefly.
Roles and Responsibilities
There are three major sets of roles and responsibilities with regard to the implementation and maintenance of the CMI.
Data Collection Strategies
The data collection strategies for the CMI will vary for each of the training event measures as follows:
Analysis of the Training Event Results
An analysis of the CMI performance information provides the means to identify accomplishments and issues for each training event and across the entire training inventory as follows:
The performance information presented in the CMI outlines accomplishments and issues related to the achievement of the full benefits of training and development across the entire inventory of training events from a financial, design/delivery, knowledge acquisition and application view points. Cumulatively, this information provides a causal link between achieving the continuous learning goals/ mission of the organization and the strategic, tactical and operational issues interfering with this goal. Yet, given this information, the basic question remains of how to interpret it to take the necessary corrective actions. In other words â€œWhat is normal for my training and development environmentâ€œ. The interpretation of CMI information can be accomplished using the following three methods:
Each of the CMI training and development measurement perspectives is captured in a Training Event Status Report. To ascertain the effectiveness of each training event on a life cycle basis, the CMI uses a three option summary indicator:
Each training event is given a Red, Yellow or Green rating for each of the CMI measures: financial, design/delivery, knowledge acquisition, and learning application measures. If one of Â these CMI measures is Red, the training event is rated Red. If most of the indicators are Yellow or Green without a Red rating, then the training event is given either a Yellow or Green rating.
Then, an overall Training Score is given for each training event across the entire inventory of courses. These ratings, then, make it fairly easy to concentrate on those training events that are rated Red while ignoring those in the inventory which are rated Green. Management of the training becomes focused on attaining excellence in each training event while dealing with those few exceptions that require attention to bring them up to a higher standard. It is management by excellence and exception.
Use of Baseline Information
The availability of baseline information provides a historical perspective on the training performance dimensions permitting the analysis of trends over time. This is the most powerful method of interpretation and is required to understand any changes in the training and development organization. It is also the most difficult to obtain. Financial information is likely the only type of CMI measure available during the first year of operation. In subsequent years, year over year comparisons can be made across all performance measurement dimensions of the CMI.
Use of Benchmarks
Bench marking allows comparisons with training and development functions in organizations of similar size and complexity in order to provide a comparative reference point. For example, spending 3 to 4% of a salary budget on training would seem very high in most organizations. Yet, this is the norm in most information technology organizations. Without this benchmark information, the interpretation of financial training performance indicator would have been faulty. However, useful benchmarking information is typically very difficult to obtain.
Experience with the Organization
Organizational experience is also a key requirement to interpret any training performance information. Major changes in training and development must be interpreted within the context of the history of the organization as a whole. For example, spending on training and development during 1999/2000 fiscal year dropped significantly across many departments in the federal public service. Knowing, however, that the majority of operational and capital funding was spent to deal with Y2K issues during that year and that spending levels returned to more normal levels during subsequent years, provides a powerful interpretation on this information.
The most effective interpretation of CMI performance measurement information can be obtained from using as many of these techniques as possible, in combination.
Communicating the training and development accomplishments and issues to both senior management and staff is imperative. The delivery of training events that are affordable, accessible and responsive to the needs of its participants can only be achieved if management is able to obtain effective feedback provided through all the measurement dimensions embodied in the CMI. Such feedback provides the information needed to steer the training and development in support of the mission of the organization in general, and towards its continuous learning objectives in particular.
A composite training and development report â€“ Summary Training Event Status Report (STESR) should be published quarterly. It is used to provide feedback in the form of accomplishments and issues for financial, design/delivery, knowledge acquisition/skill development and learning application measures across the entire training development program.
Application of the CMI Â Â
Unless the information generated from the CMI is used to take corrective actions to steer the training and development, the implementation effort is wasted. This CMI performance information must be used to move the Training and Development forward progressively. Where it is demonstrated that CMI performance information is used to improve the functioning of the training and development, then the CMI will become fully self-sustaining.
Four of the measurement perspectives were adapted from the work of DL Â Kirkpatrick (participant reaction, learning, behavior, results)
For a list of sample performance measures for each of the measure dimensions of the Course Management Index refer to the first article published in this series.
Scoring Guidelines for each of the measurement perspectives of the CMI must be developed based upon the unique requirements of the training environment in each organization.