Implementing a Performance Measurement System in a Public Service Informatics Function

Bryan Shane

The development and implementation of a balanced PMS must begin at the top.

Federal government departments are attempting to  provide programs and services to their clients that are affordable, accessible  and responsive to their needs in a time of fiscal restraint and ongoing change.  A major strategy to accomplish this has been the use of information management  and information technology (IM/IT) to streamline operations, improve levels of  service and pro- vide better information for decision making. This has been,  however, fraught with problems. Projects have seldom been delivered on time, to  budget and to specification. To maximize the contribution of IM/IT in developing  and supporting outstanding government programs and services, there is a need to  improve the decision-making process. This can be brought about through an  effective performance measurement system (PMS). This article focuses on the  stages involved in the development and implementation of a balanced approach to  performance measurement in an informatics function of a public sector  organization.

Performance measurement defined

A balanced approach to performance measurement can be  defined from three points of view:

• It is a philosophy of continuous learning in  which feedback is used to make ongoing adjustments to the course of an  organization in the pursuit of its vision.
• It is a continuous and ongoing process that  begins with the setting of objectives and the development of strategies/plans  to achieve those objectives in support of the vision.
• It is a structure in which strategic, tactical  and operational actions are linked via a feedback process to provide the  information required to intervene and/or improve the program or service on a  systematic basis.

Balanced performance measurement is a management  system – an ongoing best practice that provides the means to assess the  effectiveness of an organization’s operations from four perspectives -  financial, client satisfaction, quality of service and innovation/learning, It  is used to provide feedback at all levels – strategic, tactical or operational -  on how well strategies and plans are being met. This performance feedback  provides the information necessary to improve decision making within the  organization, to enable proactive problem correction and to promote continuous  improvement.

Principles

Implementation of a balanced PMS in an informatics  organization requires that a number of principles be followed to define the  scope and provide a philosophy of operations. These principles include

• understanding that the implementation and  integration of a PMS into the organization’s culture requires time, effort, skills/expertise and perhaps, most importantly, the active support of senior  management
• recognizing that performance measurement  indicators must be organized around the department’s planning and budgeting cycles
• creating and/or adapting the performance  measurement processes of financial planning and reporting, quality of service,  status reporting, client satisfaction, employee satisfaction and continuous  improvement to the department’s operational processes
• establishing a central office of performance  measurement (OPM) that is responsible for the planning, implementation and  ongoing operation of the PMS. (The OPM must report directly to and be supported by senior management.)
• using a phased implementation process to develop and implement the PMS * adopting a just-do-it mentality
• designing the PMS so that multiple lines of  evidence are generated for each perspective at different organizational levels  at different times. (The information generated from one perspective must be  confirmed by another perspective so that there is a convergence of information  to support the diagnosis of and action on a particular issue.)

Acceptance of a PMS involves a gradual process of  change in organizational culture. The focus is on identifying and dealing with  the issues necessary for achieving the organizational mission and on linking  strategic plans with operational decision making. Over time, this approach  allows for the development of an organizational culture that values and supports  balanced and comprehensive feedback as an essential element in both examining  the issues and providing the information necessary for effective decision  making.

Roles and  responsibilities

There are a number of roles and responsibilities  associated with the development and implementation of a PMS.

Office of performance  measurement

The OPM holds overall responsibility for the  development and implementation of a PMS and must report directly to the Chief  Information Officer.

Specific responsibilities of the OPM include

• building the performance measurement  architectures (PMAs)
• designing the strategies required to implement  performance measurement for each of the four measurement perspectives included  in the balanced approach to performance measurement
• implementing the PMS on a pilot basis and if it  is successful, extending it across the organization
• establishing and integrating performance  measurement processes into the operations of the organization for finance, quality of programs and services, client satisfaction, employee satisfaction  and continuous improvement
• providing a program to transfer knowledge, skills  and abilities from the subject area experts to the informatics programs and  service areas.
• marketing the PMS to promote and improve  understanding, acceptance and support for it across the informatics organization
• providing performance measurement information to  informatics programs/services senior management
• identifying the issues and suggesting actions to senior management to resolve problems.

IT/M program and services area

Implementation cannot rest solely on the efforts of  the OPM. Cooperation and action are required from the IM/IT organization itself.  Specific responsibilities include:

• participating in the development process of the  PMS by providing feedback, attending meetings, modifying processes, being open  to change, etc.
• providing performance measurement information  related to the quality of informatics programs and services
• providing the resources necessary to participate  effectively in the development, implementation and ongoing maintenance of the  PMS and to facilitate knowledge transfer
• using the information provided to improve decision making and t ‘o develop strategies and plans for dealing with issues revealed by the PMS.

Steps in developing and  implementing a PMS

This section briefly discusses the steps involved in  the development and implementation of a PMS for a public sector informatics organization.

Step 1: Project  orientation

Provide an orientation to management and staff on the  balanced approach to performance measurement. This orientation is designed to  alleviate fears associated with performance measurement by describing what it is  and is not; to explain the roles and responsibilities required of participants,  the stages involved in development and implementation and the time frames and  principles involved; and to address anticipated problems. This step is necessary  for improving understanding and building support for the project among the  uninitiated members of the informatics organization.

Step 2: Readiness assessment

Assess the readiness of the informatics organization  to accept the balanced approach. Several criteria may be used to complete this  assessment:

• a business plan, mandate, vision and organizational philosophy
• a strategic IM/IT plan that reinforces and  supports the business plan
• an organizational culture, relatively open to  change, that values feedback as essential to providing the information  necessary for decision making
• collection, examination and reporting of  corporate sources of information, especially finance, quality of programs and  services, client satisfaction, employee satisfaction and innovation
• the skills and abilities of staff to implement a  PMS and/or willingness to accept knowledge transfer
• a commitment to resource the project  appropriately for one to two years – the time it will take to inculcate a PMS  into the organization.

This assessment must provide a clear picture of an  informatics organization with a mandate/vision of how the functions support and  reinforce the business plans of the department. Further, there must be evidence  to demonstrate the commitment of management and staff and the resources to carry  out the project.

Based on a positive analysis of organizational  readiness to develop and implement a PMS, it is necessary to select a core functional area to act as a pilot.

Step 3: Performance measurement architecture

Develop the PMA, which is the design, content and  structure of the PMS prior to its implementation, based on the departmental and  informatics organization’s mission, vision and philosophy. This can only be done  from the top down – business objectives must be developed first, then the  informatics objectives, plans and strategies can be designed to support the  business directions. Performance measures/indicators (PM/PIs) can then be  derived for the informatics function. The PM/Pls are objective, quantifiable and  output oriented. They are also qualitative in nature.

The development and validation of interpretive  models, known as the “straw dog” approach, should be used. This method is time  effective and ensures that the involvement and participation of senior  management and staff are focused and time efficient.

• The development of a straw dog business model for  the informatics organization specifies the organizational mission/vision,  functions, related activities and critical success factors. Once this vehicle  is constructed, it must be validated with staff members and revised according  to their comments. This business model provides the information and insight  from which the PMA is developed.
• The construction of a straw dog PMA based on the  information gained from the business model has the following elements:
  • organizational mission/vision
  • objectives for each of the following  perspectives: financial, quality of programs and services, and client  satisfaction and innovation / learning, which are divided into employee  satisfaction/continuous improvement
• Based on the objectives for each PMA perspective,  PM/PIs are developed.

The PMA provides information on past performance  using traditional financial measures. It also provides information on current  operations by focusing on programs and services and the level of client  satisfaction. Further, it provides input on future requirements that may arise  from changing technology, client needs or staff needs. The PMA provides feedback  across all the dimensions of an informatics organization needed for effective  decision making. It is a causal link between the organizational  vision/objectives and the information needed to deal with the multifaceted  strategic and operational issues that hinder progress toward its attainment.

Once the PMA is developed and approved, it becomes  the sanctioned PMS architecture for the informatics organization. It is a living  document that evolves and changes as measures are implemented and feedback on  their effectiveness is received. It must be placed under change control.

Table 1 provides a generic PMA of an actual ITAM  organization.

Step 4:  Implementation

Implementation of a PMS requires a phased approach.  It should begin on a pilot basis within one or two core informatics business  lines to demonstrate the value of performance measurement and to build the  competencies of the performance measurement team. After a successful pilot of 3  to 6 months, the second phase can be initiated. This phase concentrates on only  high-priority informatics areas and/or projects. The third and final phase  addresses the remaining parts of the informatics organization.

The activities required to implement a PMS are  described below.

Development of a performance measurement profile

The measurement profile for each PWPI provides an  assessment of the degree to which existing operational processes must be  developed or adapted in order to implement the PMS. It indicates areas where  information and processes related to the PWPI exist and where they must be  adapted and/or created. It provides the basis for scheduling the implementation  of different measures, together with an estimated degree of effort.

Development of an implementation strategy for each measurement perspective

The development of an implementation strategy for  each of the four measurement perspectives provides a guide for defining the  roles and responsibilities of participants; the frequency of reporting; the data  collection methods; the methods of analyzing, reporting and interpreting the  information; and the issues that may impede progress for each measurement  perspective. The strategy also provides the guidelines for developing and  operating the performance measurement processes once the implementation effort  is complete. These strategies also assist in creating a common understanding and  acceptance of all the required performance measurement elements prior to their  development and use.

Integration of performance measurement processes into operational activities

The next activity in the implementation of a PMS is  the development or adaptation of existing business processes to support data  gathering for each of the four performance measurement perspectives.

In some cases, existing sources can provide the  required information without much adaptation. This is normally the situation for  the financial perspective, where information on revenues, expenditures, capital  expenditures and funding ratios is usually readily available. Changes that may  be required in the existing financial processes relate to establishing reference  levels; year-end reconciliation procedures; financial coding for projects;  reporting of revenues/expenditures down to the project level; and reconciliation  of financial/time utilization information at the project level.

Measurement of performance can involve the creation  of a new business process. This is often the case in measuring client  satisfaction. With regard to quality of service, employee satisfaction and  continuous improvement, the measurement of performance may involve either the  amendment of existing processes or the creation of new ones.

Data  gathering

Once business processes for each measurement  perspective have been adapted or created, the information generated must be  captured according to predetermined reporting formats.

• Financial Report – provides (at a minimum) data on  revenues, expenditures, capital and funding ratios. This information is  usually reported on a monthly basis.
• Quality of Program/Service Composite Status Report  – summarizes the performance of all ITAM projects related to each performance  measure – project management index, functional/technical quality, etc. All  these performance measures are averaged to determine an overall grading for  each project according to the strategy developed for this perspective. This is  also a monthly report.
• Client Satisfaction Report – summarizes all the  information in the client satisfaction index related to projects/services,  staff/consultants, communication, improvements to existing  programs/services/processes or requirements for new ones. Client retention  information is also published. This is usually a semi-annual report that is  refreshed on an ongoing basis by client satisfaction information gathered at  the project level.
• Employee Satisfaction Report – outlines all the  information in the employee satisfaction index related to plans/goals, role  clarity, decision making/communication, team building, staff utilization,  rewards/recognition and productivity. This may be either a semi-annual or  annual report, depending on the initial degree of satisfaction assessed.
• Continuous Improvement Report – defines all the  information in the continuous improvement index related to staffing,  suggestions for improvement, professional development and rewards/recognition.  This is a quarterly report. These reports are collated and published quarterly  in a composite report called the Dashboard.

Interpretation of  results

The performance information presented in the  Dashboard outlines issues related to the achievement of organizational vision  from the financial, quality of service, client satisfaction and  innovation/learning perspectives. Cumulatively, this information provides a  causal link between the achievement of the vision and the strategic, tactical  and operational issues interfering with this goal. Yet the basic question  remains of how to interpret the Dashboard information so that the necessary  corrective actions can be taken. This can be accomplished in several ways:

• Rating system
  Within each of the four measurement  perspectives, an implementation strategy has provided guidelines. For example,  within the quality-of-service perspective, all the performance information  related to programs and services is captured in a project status report. When  it comes to interpreting this information, the system developed provides a  green rating if all dimensions of the project cost, schedule,  functional/technical quality, etc., are within 10 percent of plan. A yellow  rating is provided if these dimensions are between 10 percent to 20 percent. A  red rating is provided for dimensions greater than 20 percent. For  non-quantitative quality-of -service dimensions such as effective use of staff,  a judgment of these dimensions is provided by the project manager. This  judgment and the other information on the status report are verified by  providing a copy to the client.
  Another rating system used is based upon  the interpretation of questionnaire information. On a five-point scale, an  answer provided by clients (client satisfaction perspective) or by employees  (employee satisfaction questionnaire) is rated red, yellow or green, based  upon average scores between 1 to 2.5; 2.6 to 3.5, and 3.6 to 5.0 respectively.
• Use of baseline information
  Baseline information provides a historical  perspective on the performance dimensions that permit the analysis of trends  over time. This is powerful and necessary to understanding any changes in the  organization. It is also difficult to obtain. Financial and employee  information are probably the only types available and are not likely to  reflect all performance measurement dimensions. However, it is worth the  effort to gather as much of this information as possible in a reasonable  length of time.
• Use of benchmarks
  Benchmarking allows comparisons with the business processes of leading  organizations in order to provide the information needed to improve existing  operations. As an example, a staff turnover rate of 11 percent annually is  highly stressful for most organizations and indicates a major problem, but in  the informatics world, that is a normal level of attrition. Without this  benchmark information, the interpretation of this performance indicator would  be faulty.
• Experience with the organization
  This experience is also a key requirement for  interpreting any performance information. Major changes must be interpreted  within the context of the history of the organization. For example, an overall  employee satisfaction score of 3.5 out of 5.0 may indicate major problems in  various dimensions. Knowing, however, that Program Review has just resulted in  major cuts and that this rating is an improvement over that of the previous year provides a powerful interpretation of this score.
• Multiple lines of evidence
  The design of the PMA is completed in such a way  that multiple lines of evidence are provided for each perspective at different  levels in the organization at different times. For example, financial  information is obtained from financial performance indicators and is also  provided by the project status reports. Client satisfaction information is  provided through client satisfaction questionnaires but is also included as a dimension of the project status report. Multiple lines of evidence from the  PMS are designed to show convergence of information within each piece reinforcing the other. Any issue outlined in any perspective of the Dashboard  should have corroborative information from another perspective if possible.

The most effective interpretation of  performance measurement information can be obtained from using as many of these  techniques as possible in combination. Their use is especially important in  writing the executive summaries in the Dashboard.

Communicating the  results

Communicating the results to both senior management  and staff is imperative. It is only through communication and decision making  that changes can be made either to correct the course of the organization toward  its vision, or to reward the efforts made and encourage continued results.

The Dashboard provides feedback to the management  committee that contains the actual results on how well strategies and plans are  being met and identifies the issues at all levels – strategic, tactical and  operational. This performance information provides the information necessary for  improving decision making within the organization.

To ensure that the Dashboard is used effectively,

• the quarterly publishing date needs to be  standardized at six weeks after the quarter so that all members of the  management committee expect it as a regular feature of their management  meetings.
•  it should be delivered to members of the  management committee one week before the next management meeting to allow them  sufficient time to read and understand its contents.
• at least a one-hour time slot should be allotted  for the discussion of
  • the progress achieved  since the last meeting
  • the issues/impacts related to each perspective
  • the recommendations to deal with the important  issues
  • the decisions required.

Step 5:  Institutionalization of a PMS into the organization

Two strategies are essential to ensure that  performance measurement becomes integrated with the organization’s standard  operating procedures.

Knowledge transfer

Knowledge from the subject matter experts to the  organization must be transferred to make the informatics organization self-sufficient and self-sustaining in its ability to operate and update the  PMS. Specific areas of knowledge transfer include

• the methodology for developing performance measurement architectures
• the means to establish baseline information in order to compare future performance
• methods for developing and testing questionnaires
• methods for collecting, analyzing and interpreting data
• development of an evaluation strategy to assess  the effectiveness of the performance measurement pilot
• a business process renewal
• project management
• facilitation and structured interviewing  techniques
• training techniques
• presentation techniques.

As the OPM is responsible for the development and  implementation of the PMS, it should become the repository of these performance  measurement skills/abilities. Sufficient dedicated resources are necessary to  ensure this expertise is retained within the organization.

Organizational  culture

The development of an organizational culture supportive of the PMS is essential to breaking down the organizational and  individual resistance to change. There are techniques that will gradually do  this and increase ownership of the system:

• Assessment of the readiness of the organization to  adopt this approach to performance measurement is an indicator to staff that  change is underway and may well raise their expectations for the organization.
• Use of straw dog models to present new concepts or  tools allows staff members to participate, examine, review and revise these  documents, thereby increasing their ownership and commitment and their  expectation of continued involvement in the change process.
• Employment of facilitation and structured  interviewing techniques helps in gaining effective feedback and in presenting  new information and concepts. This assists in opening the lines of  communication and encourages free exchange of ideas, issues and solutions.
• Holding regular weekly meetings helps deal with  ongoing implementation issues and increases the involvement of staff in  supporting the course for the organization.
• Use of flexible methods to integrate the PMS into  the operational processes of the organization improves the morale of staff as  they recognize their skills and talents and that they are being appreciated.
• Holding training/orientation sessions to transfer  knowledge and skills to the program office and the program and service areas  within the informatics organization encourages and supports staff growth and  development.
• Using an issue log tracks problems hampering the  implementation effort. Actions on the part of managers indicate to staff that  management is serious about the changes effected by the PMS.
• Using an issue briefing note to describe a problem  and outline its implications, solution options and recommended actions  provides management with the information required to make sound practical  decisions to drive the organization toward its vision.
• Ongoing leadership and communication by senior  management is needed to support the cultural change created by the balanced  PMS, such as the need for staff to participate and cooperate fully and the  actions taken by managers to address the identified performance measurement  issues.

Best practices

Experience has shown that several best practices can  be used to assist in the smooth development and implementation of a balanced  PMS:

• The development and implementation of a balanced  PMS is both an art and a science. The science of developing and implementing a  PMS involves the use of the steps previously described to provide similar  philosophy, structure and process so that comprehensive information can be  provided across all dimensions of the organization. The art of developing and  implementing a PMS involves tailoring the design and implementation process to  the unique requirements of each organization in terms of specific measures,  timing, sequence of activities and knowledge transfer.
• The use of pilots provides evidence of the utility  of this performance measurement approach. It gradually builds  acceptance/support for a threatening project and provides the experience  needed for tailoring the PMS to the unique requirements of the organization  while developing the expertise within the organization.
• The development and establishment of a balanced  PMS must begin at the top of the organization and be implemented progressively  downward. This requires the full commitment and support of management in terms  of providing a supportive environment and communicating the necessity and  importance of this approach. It also requires a financial investment to  develop, implement and maintain the operation.
• Adopt a just-do-it approach to the development and  implementation process of the PMS. Don’t be tied to expensive data gathering  and implementation methods. Use work-around strategies to find simple but  effective solutions. Where progress is delayed in developing and implementing  the measurement of one perspective, focus more attention on the others.  Because the whole process is self-improving and self-correcting, the key is to  go with the information, processes and interpretation already developed.
• Unless the information generated from the balanced  PMS is used to take corrective actions in the form of strategies/plans to  steer the organization toward its mission and vision, the development and  implementation effort is wasted. This performance information must be used to  move the organization forward progressively in terms of finances, quality of  programs/services, client/employee satisfaction and continuous adaptation to  changing circumstances. Where it is demonstrated to employees that performance  information is used to improve the functioning of the organization, then the  PMS will become self-sustaining.

Conclusion

The delivery of informatics programs and  services that are affordable, accessible and responsive to the needs of  Canadians can be achieved only if public sector informatics managers are able to  obtain effective feedback provided through a balanced approach to performance measurement. Such feedback provides the information needed for the informatics organization to establish and sustain excellence in program and service delivery to the public.

*1 This is based on the Balanced Scorecard Approach  developed by R.S. Kaplan and D.P. Norton.

Bryan Shane is senior partner of BPC  Management Consultants in Ottawa. Since 1981 he has provided change management  consulting services to both public and private sector organizations. Mr. Shane  has a BA in Political Science from Carleton University and a BEd and Med from  the University of Ottawa. He also completed postgraduate studies in statistics  and evaluation.