Implementing a Performance Measurement System in a Public Service Informatics Function
Bryan Shane
The development and implementation of a balanced PMS must begin at the top.
Federal government departments are attempting to provide programs and services to their clients that are affordable, accessible and responsive to their needs in a time of fiscal restraint and ongoing change. A major strategy to accomplish this has been the use of information management and information technology (IM/IT) to streamline operations, improve levels of service and pro- vide better information for decision making. This has been, however, fraught with problems. Projects have seldom been delivered on time, to budget and to specification. To maximize the contribution of IM/IT in developing and supporting outstanding government programs and services, there is a need to improve the decision-making process. This can be brought about through an effective performance measurement system (PMS). This article focuses on the stages involved in the development and implementation of a balanced approach to performance measurement in an informatics function of a public sector organization.
Performance measurement defined
A balanced approach to performance measurement can be defined from three points of view:
- It is a philosophy of continuous learning in which feedback is used to make ongoing adjustments to the course of an organization in the pursuit of its vision.
- It is a continuous and ongoing process that begins with the setting of objectives and the development of strategies/plans to achieve those objectives in support of the vision.
- It is a structure in which strategic, tactical and operational actions are linked via a feedback process to provide the information required to intervene and/or improve the program or service on a systematic basis.
Balanced performance measurement is a management system – an ongoing best practice that provides the means to assess the effectiveness of an organization’s operations from four perspectives - financial, client satisfaction, quality of service and innovation/learning, It is used to provide feedback at all levels – strategic, tactical or operational - on how well strategies and plans are being met. This performance feedback provides the information necessary to improve decision making within the organization, to enable proactive problem correction and to promote continuous improvement.
Principles
Implementation of a balanced PMS in an informatics organization requires that a number of principles be followed to define the scope and provide a philosophy of operations. These principles include
- understanding that the implementation and integration of a PMS into the organization’s culture requires time, effort, skills/expertise and perhaps, most importantly, the active support of senior management
- recognizing that performance measurement indicators must be organized around the department’s planning and budgeting cycles
- creating and/or adapting the performance measurement processes of financial planning and reporting, quality of service, status reporting, client satisfaction, employee satisfaction and continuous improvement to the department’s operational processes
- establishing a central office of performance measurement (OPM) that is responsible for the planning, implementation and ongoing operation of the PMS. (The OPM must report directly to and be supported by senior management.)
- using a phased implementation process to develop and implement the PMS * adopting a just-do-it mentality
- designing the PMS so that multiple lines of evidence are generated for each perspective at different organizational levels at different times. (The information generated from one perspective must be confirmed by another perspective so that there is a convergence of information to support the diagnosis of and action on a particular issue.)
Acceptance of a PMS involves a gradual process of change in organizational culture. The focus is on identifying and dealing with the issues necessary for achieving the organizational mission and on linking strategic plans with operational decision making. Over time, this approach allows for the development of an organizational culture that values and supports balanced and comprehensive feedback as an essential element in both examining the issues and providing the information necessary for effective decision making.
Roles and responsibilities
There are a number of roles and responsibilities associated with the development and implementation of a PMS.
Office of performance measurement
The OPM holds overall responsibility for the development and implementation of a PMS and must report directly to the Chief Information Officer.
Specific responsibilities of the OPM include
- building the performance measurement architectures (PMAs)
- designing the strategies required to implement performance measurement for each of the four measurement perspectives included in the balanced approach to performance measurement
- implementing the PMS on a pilot basis and if it is successful, extending it across the organization
- establishing and integrating performance measurement processes into the operations of the organization for finance, quality of programs and services, client satisfaction, employee satisfaction and continuous improvement
- providing a program to transfer knowledge, skills and abilities from the subject area experts to the informatics programs and service areas.
- marketing the PMS to promote and improve understanding, acceptance and support for it across the informatics organization
- providing performance measurement information to informatics programs/services senior management
- identifying the issues and suggesting actions to senior management to resolve problems.
IT/M program and services area
Implementation cannot rest solely on the efforts of the OPM. Cooperation and action are required from the IM/IT organization itself. Specific responsibilities include:
- participating in the development process of the PMS by providing feedback, attending meetings, modifying processes, being open to change, etc.
- providing performance measurement information related to the quality of informatics programs and services
- providing the resources necessary to participate effectively in the development, implementation and ongoing maintenance of the PMS and to facilitate knowledge transfer
- using the information provided to improve decision making and t ‘o develop strategies and plans for dealing with issues revealed by the PMS.
Steps in developing and implementing a PMS
This section briefly discusses the steps involved in the development and implementation of a PMS for a public sector informatics organization.
Step 1: Project orientation
Provide an orientation to management and staff on the balanced approach to performance measurement. This orientation is designed to alleviate fears associated with performance measurement by describing what it is and is not; to explain the roles and responsibilities required of participants, the stages involved in development and implementation and the time frames and principles involved; and to address anticipated problems. This step is necessary for improving understanding and building support for the project among the uninitiated members of the informatics organization.
Step 2: Readiness assessment
Assess the readiness of the informatics organization to accept the balanced approach. Several criteria may be used to complete this assessment:
- a business plan, mandate, vision and organizational philosophy
- a strategic IM/IT plan that reinforces and supports the business plan
- an organizational culture, relatively open to change, that values feedback as essential to providing the information necessary for decision making
- collection, examination and reporting of corporate sources of information, especially finance, quality of programs and services, client satisfaction, employee satisfaction and innovation
- the skills and abilities of staff to implement a PMS and/or willingness to accept knowledge transfer
- a commitment to resource the project appropriately for one to two years – the time it will take to inculcate a PMS into the organization.
This assessment must provide a clear picture of an informatics organization with a mandate/vision of how the functions support and reinforce the business plans of the department. Further, there must be evidence to demonstrate the commitment of management and staff and the resources to carry out the project.
Based on a positive analysis of organizational readiness to develop and implement a PMS, it is necessary to select a core functional area to act as a pilot.
Step 3: Performance measurement architecture
Develop the PMA, which is the design, content and structure of the PMS prior to its implementation, based on the departmental and informatics organization’s mission, vision and philosophy. This can only be done from the top down – business objectives must be developed first, then the informatics objectives, plans and strategies can be designed to support the business directions. Performance measures/indicators (PM/PIs) can then be derived for the informatics function. The PM/Pls are objective, quantifiable and output oriented. They are also qualitative in nature.
The development and validation of interpretive models, known as the “straw dog” approach, should be used. This method is time effective and ensures that the involvement and participation of senior management and staff are focused and time efficient.
- The development of a straw dog business model for the informatics organization specifies the organizational mission/vision, functions, related activities and critical success factors. Once this vehicle is constructed, it must be validated with staff members and revised according to their comments. This business model provides the information and insight from which the PMA is developed.
- The construction of a straw dog PMA based on the information gained from the business model has the following elements:
- organizational mission/vision
- objectives for each of the following perspectives: financial, quality of programs and services, and client satisfaction and innovation / learning, which are divided into employee satisfaction/continuous improvement
- Based on the objectives for each PMA perspective, PM/PIs are developed.
The PMA provides information on past performance using traditional financial measures. It also provides information on current operations by focusing on programs and services and the level of client satisfaction. Further, it provides input on future requirements that may arise from changing technology, client needs or staff needs. The PMA provides feedback across all the dimensions of an informatics organization needed for effective decision making. It is a causal link between the organizational vision/objectives and the information needed to deal with the multifaceted strategic and operational issues that hinder progress toward its attainment.
Once the PMA is developed and approved, it becomes the sanctioned PMS architecture for the informatics organization. It is a living document that evolves and changes as measures are implemented and feedback on their effectiveness is received. It must be placed under change control.
Table 1 provides a generic PMA of an actual ITAMÂ organization.
Step 4:Â Implementation
Implementation of a PMS requires a phased approach. It should begin on a pilot basis within one or two core informatics business lines to demonstrate the value of performance measurement and to build the competencies of the performance measurement team. After a successful pilot of 3 to 6 months, the second phase can be initiated. This phase concentrates on only high-priority informatics areas and/or projects. The third and final phase addresses the remaining parts of the informatics organization.
The activities required to implement a PMS are described below.
Development of a performance measurement profile
The measurement profile for each PWPI provides an assessment of the degree to which existing operational processes must be developed or adapted in order to implement the PMS. It indicates areas where information and processes related to the PWPI exist and where they must be adapted and/or created. It provides the basis for scheduling the implementation of different measures, together with an estimated degree of effort.
Development of an implementation strategy for each measurement perspective
The development of an implementation strategy for each of the four measurement perspectives provides a guide for defining the roles and responsibilities of participants; the frequency of reporting; the data collection methods; the methods of analyzing, reporting and interpreting the information; and the issues that may impede progress for each measurement perspective. The strategy also provides the guidelines for developing and operating the performance measurement processes once the implementation effort is complete. These strategies also assist in creating a common understanding and acceptance of all the required performance measurement elements prior to their development and use.
Integration of performance measurement processes into operational activities
The next activity in the implementation of a PMS is the development or adaptation of existing business processes to support data gathering for each of the four performance measurement perspectives.
In some cases, existing sources can provide the required information without much adaptation. This is normally the situation for the financial perspective, where information on revenues, expenditures, capital expenditures and funding ratios is usually readily available. Changes that may be required in the existing financial processes relate to establishing reference levels; year-end reconciliation procedures; financial coding for projects; reporting of revenues/expenditures down to the project level; and reconciliation of financial/time utilization information at the project level.
Measurement of performance can involve the creation of a new business process. This is often the case in measuring client satisfaction. With regard to quality of service, employee satisfaction and continuous improvement, the measurement of performance may involve either the amendment of existing processes or the creation of new ones.
Data gathering
Once business processes for each measurement perspective have been adapted or created, the information generated must be captured according to predetermined reporting formats.
- Financial Report – provides (at a minimum) data on revenues, expenditures, capital and funding ratios. This information is usually reported on a monthly basis.
- Quality of Program/Service Composite Status Report – summarizes the performance of all ITAM projects related to each performance measure – project management index, functional/technical quality, etc. All these performance measures are averaged to determine an overall grading for each project according to the strategy developed for this perspective. This is also a monthly report.
- Client Satisfaction Report – summarizes all the information in the client satisfaction index related to projects/services, staff/consultants, communication, improvements to existing programs/services/processes or requirements for new ones. Client retention information is also published. This is usually a semi-annual report that is refreshed on an ongoing basis by client satisfaction information gathered at the project level.
- Employee Satisfaction Report – outlines all the information in the employee satisfaction index related to plans/goals, role clarity, decision making/communication, team building, staff utilization, rewards/recognition and productivity. This may be either a semi-annual or annual report, depending on the initial degree of satisfaction assessed.
- Continuous Improvement Report – defines all the information in the continuous improvement index related to staffing, suggestions for improvement, professional development and rewards/recognition. This is a quarterly report. These reports are collated and published quarterly in a composite report called the Dashboard.
Interpretation of results
The performance information presented in the Dashboard outlines issues related to the achievement of organizational vision from the financial, quality of service, client satisfaction and innovation/learning perspectives. Cumulatively, this information provides a causal link between the achievement of the vision and the strategic, tactical and operational issues interfering with this goal. Yet the basic question remains of how to interpret the Dashboard information so that the necessary corrective actions can be taken. This can be accomplished in several ways:
- Rating system
Within each of the four measurement perspectives, an implementation strategy has provided guidelines. For example, within the quality-of-service perspective, all the performance information related to programs and services is captured in a project status report. When it comes to interpreting this information, the system developed provides a green rating if all dimensions of the project cost, schedule, functional/technical quality, etc., are within 10 percent of plan. A yellow rating is provided if these dimensions are between 10 percent to 20 percent. A red rating is provided for dimensions greater than 20 percent. For non-quantitative quality-of -service dimensions such as effective use of staff, a judgment of these dimensions is provided by the project manager. This judgment and the other information on the status report are verified by providing a copy to the client.
Another rating system used is based upon the interpretation of questionnaire information. On a five-point scale, an answer provided by clients (client satisfaction perspective) or by employees (employee satisfaction questionnaire) is rated red, yellow or green, based upon average scores between 1 to 2.5; 2.6 to 3.5, and 3.6 to 5.0 respectively. - Use of baseline information
Baseline information provides a historical perspective on the performance dimensions that permit the analysis of trends over time. This is powerful and necessary to understanding any changes in the organization. It is also difficult to obtain. Financial and employee information are probably the only types available and are not likely to reflect all performance measurement dimensions. However, it is worth the effort to gather as much of this information as possible in a reasonable length of time. - Use of benchmarks
Benchmarking allows comparisons with the business processes of leading organizations in order to provide the information needed to improve existing operations. As an example, a staff turnover rate of 11 percent annually is highly stressful for most organizations and indicates a major problem, but in the informatics world, that is a normal level of attrition. Without this benchmark information, the interpretation of this performance indicator would be faulty. - Experience with the organization
This experience is also a key requirement for interpreting any performance information. Major changes must be interpreted within the context of the history of the organization. For example, an overall employee satisfaction score of 3.5 out of 5.0 may indicate major problems in various dimensions. Knowing, however, that Program Review has just resulted in major cuts and that this rating is an improvement over that of the previous year provides a powerful interpretation of this score. - Multiple lines of evidence
The design of the PMA is completed in such a way that multiple lines of evidence are provided for each perspective at different levels in the organization at different times. For example, financial information is obtained from financial performance indicators and is also provided by the project status reports. Client satisfaction information is provided through client satisfaction questionnaires but is also included as a dimension of the project status report. Multiple lines of evidence from the PMS are designed to show convergence of information within each piece reinforcing the other. Any issue outlined in any perspective of the Dashboard should have corroborative information from another perspective if possible.
The most effective interpretation of performance measurement information can be obtained from using as many of these techniques as possible in combination. Their use is especially important in writing the executive summaries in the Dashboard.
Communicating the results
Communicating the results to both senior management and staff is imperative. It is only through communication and decision making that changes can be made either to correct the course of the organization toward its vision, or to reward the efforts made and encourage continued results.
The Dashboard provides feedback to the management committee that contains the actual results on how well strategies and plans are being met and identifies the issues at all levels – strategic, tactical and operational. This performance information provides the information necessary for improving decision making within the organization.
To ensure that the Dashboard is used effectively,
- the quarterly publishing date needs to be standardized at six weeks after the quarter so that all members of the management committee expect it as a regular feature of their management meetings.
-  it should be delivered to members of the management committee one week before the next management meeting to allow them sufficient time to read and understand its contents.
- at least a one-hour time slot should be allotted for the discussion of
- the progress achieved since the last meeting
- the issues/impacts related to each perspective
- the recommendations to deal with the important issues
- the decisions required.
Step 5:Â Institutionalization of a PMS into the organization
Two strategies are essential to ensure that performance measurement becomes integrated with the organization’s standard operating procedures.
Knowledge transfer
Knowledge from the subject matter experts to the organization must be transferred to make the informatics organization self-sufficient and self-sustaining in its ability to operate and update the PMS. Specific areas of knowledge transfer include
- the methodology for developing performance measurement architectures
- the means to establish baseline information in order to compare future performance
- methods for developing and testing questionnaires
- methods for collecting, analyzing and interpreting data
- development of an evaluation strategy to assess the effectiveness of the performance measurement pilot
- a business process renewal
- project management
- facilitation and structured interviewing techniques
- training techniques
- presentation techniques.
As the OPM is responsible for the development and implementation of the PMS, it should become the repository of these performance measurement skills/abilities. Sufficient dedicated resources are necessary to ensure this expertise is retained within the organization.
Organizational culture
The development of an organizational culture supportive of the PMS is essential to breaking down the organizational and individual resistance to change. There are techniques that will gradually do this and increase ownership of the system:
- Assessment of the readiness of the organization to adopt this approach to performance measurement is an indicator to staff that change is underway and may well raise their expectations for the organization.
- Use of straw dog models to present new concepts or tools allows staff members to participate, examine, review and revise these documents, thereby increasing their ownership and commitment and their expectation of continued involvement in the change process.
- Employment of facilitation and structured interviewing techniques helps in gaining effective feedback and in presenting new information and concepts. This assists in opening the lines of communication and encourages free exchange of ideas, issues and solutions.
- Holding regular weekly meetings helps deal with ongoing implementation issues and increases the involvement of staff in supporting the course for the organization.
- Use of flexible methods to integrate the PMS into the operational processes of the organization improves the morale of staff as they recognize their skills and talents and that they are being appreciated.
- Holding training/orientation sessions to transfer knowledge and skills to the program office and the program and service areas within the informatics organization encourages and supports staff growth and development.
- Using an issue log tracks problems hampering the implementation effort. Actions on the part of managers indicate to staff that management is serious about the changes effected by the PMS.
- Using an issue briefing note to describe a problem and outline its implications, solution options and recommended actions provides management with the information required to make sound practical decisions to drive the organization toward its vision.
- Ongoing leadership and communication by senior management is needed to support the cultural change created by the balanced PMS, such as the need for staff to participate and cooperate fully and the actions taken by managers to address the identified performance measurement issues.
Best practices
Experience has shown that several best practices can be used to assist in the smooth development and implementation of a balanced PMS:
- The development and implementation of a balanced PMS is both an art and a science. The science of developing and implementing a PMS involves the use of the steps previously described to provide similar philosophy, structure and process so that comprehensive information can be provided across all dimensions of the organization. The art of developing and implementing a PMS involves tailoring the design and implementation process to the unique requirements of each organization in terms of specific measures, timing, sequence of activities and knowledge transfer.
- The use of pilots provides evidence of the utility of this performance measurement approach. It gradually builds acceptance/support for a threatening project and provides the experience needed for tailoring the PMS to the unique requirements of the organization while developing the expertise within the organization.
- The development and establishment of a balanced PMS must begin at the top of the organization and be implemented progressively downward. This requires the full commitment and support of management in terms of providing a supportive environment and communicating the necessity and importance of this approach. It also requires a financial investment to develop, implement and maintain the operation.
- Adopt a just-do-it approach to the development and implementation process of the PMS. Don’t be tied to expensive data gathering and implementation methods. Use work-around strategies to find simple but effective solutions. Where progress is delayed in developing and implementing the measurement of one perspective, focus more attention on the others. Because the whole process is self-improving and self-correcting, the key is to go with the information, processes and interpretation already developed.
- Unless the information generated from the balanced PMS is used to take corrective actions in the form of strategies/plans to steer the organization toward its mission and vision, the development and implementation effort is wasted. This performance information must be used to move the organization forward progressively in terms of finances, quality of programs/services, client/employee satisfaction and continuous adaptation to changing circumstances. Where it is demonstrated to employees that performance information is used to improve the functioning of the organization, then the PMS will become self-sustaining.
Conclusion
The delivery of informatics programs and services that are affordable, accessible and responsive to the needs of Canadians can be achieved only if public sector informatics managers are able to obtain effective feedback provided through a balanced approach to performance measurement. Such feedback provides the information needed for the informatics organization to establish and sustain excellence in program and service delivery to the public.
*1 This is based on the Balanced Scorecard Approach developed by R.S. Kaplan and D.P. Norton.
Bryan Shane is senior partner of BPC Management Consultants in Ottawa. Since 1981 he has provided change management consulting services to both public and private sector organizations. Mr. Shane has a BA in Political Science from Carleton University and a BEd and Med from the University of Ottawa. He also completed postgraduate studies in statistics and evaluation.