Performance Measurement in Canadian Government Informatics

Bryan Shane and Gary Callaghan

A balanced performance measurement  system requires that certain principles be followed to define the scope and  provide a philosophy of operations.

This is the third and final article in a series on developing  and implementing a performance measurement system (PMS) for an informatics function in a public service environment.’ The first article presented a  conceptual approach to performance measurement based upon the Balanced  Scorecard 2 approach, while the second article described the steps to developing  and implementing a PMS. This third article describes the realization of a PMS in  a Canadian government informatics organization, the Application Management  Service (AMS) of Public Works and Government Services Canada (PWGSC).  Specifically, this article describes the mission, services and clients of AMS;  the rationale for embarking on the performance measurement project; and the  development and implementation process. Finally, it provides an assessment of  the benefits and risks associated with implementing such an  endeavour.

AMS description

AMS is a sector within  the Government Telecommunications and Informatics Services (GTIS) branch of  PWGSC. Its vision is to develop information management and information  technology IM/IT partnerships with the department’s other business lines and  with Special Operating Agencies (SOAs) to maximize PWGSC’s value as a Common  Services Organization. As one of the department’s eight major business lines,  GTIS is recognized by the federal government community as the principal agent for  the management of common IM/IT infrastructure and services. GTIS provides a wide  range of IM/IT services to PWGSC and other departments of the Canadian  government.

AMS has 418 employees and is augmented by a similar  number of consultants. it has an annual budget allocation of $42 million and an  in-house supported technology inventory valued at approximately $300 million.  Its mission is to provide development and support services in respect of  administrative processes and education to government IM/IT professionals. it  delivers value by working closely with clients to reduce business risk, while  increasing flexibility and performance.

To achieve  its mission, AMS provides a number of services to its clients, including:

• Application  support – includes operational-support and functional-enhancement services  primarily for PWGSC applications.
• Application  development – includes major functional  enhancements, architectural re-engineering services and IT project management.
• Application  environment – includes architecture and  environment services, such as standard development tools and techniques and  quality management, primarily for AMS.

Other AMS  services include business process reengineering, Internet support, data  warehousing and software exchange.

Figure I  illustrates the client base of AMS. The majority of AMS clients (87 percent)  work within PWGSC, with the remainder coming from outside the department.  Most clients are using more than one AMS service, with 71 percent using at least  three services. The most widely used services include:

• application support (84 percent),
• application development (75 percent),
• application environment (41 percent),
• Internet  (39 percent),
• data  warehousing (26 percent), and
• software  exchange (24 percent). 3

Rationale

PWGSC and GTIS are committed to the review and renewal of federal programs  and services to ensure that they are  affordable, accessible and responsive to the needs of Canadians. This new  environment created a number of challenges and opportunities in the delivery of  these programs and services.

• There is a  need to deliver high-quality programs and services across the country in an  environment of ongoing fiscal restraint. Staff reductions and cost saving  initiatives have created an environment in which employees often have difficulty  managing their workloads, with a resulting negative effect on morale.
• There are  opportunities for more effective leadership and simplified management structures  that permit long-term, viable alternative approaches to delivering programs and  services at reduced costs, based upon new forms of collaboration across branch  and department lines.
• There is a  need for a means to facilitate the sharing of best practices in the delivery of  programs and services.

These challenges and opportunities, which are directly related to the effective  delivery of programs and services to the Canadian public, were further  compounded by the need for effective performance measurement practices in AMS.  Until the Balanced Scorecard approach was implemented, the performance of  programs and services was not measured in a holistic way.

At present, the established  standard in the federal government for reporting performance is solely based on  a financial perspective. This one-dimensional approach fails to provide  performance information related to client and employee satisfaction, quality of  service or continuous improvement. Many performance measurement systems do not  provide the information needed to determine whether objectives and strategies  are being effectively implemented. Much performance information is reactive  rather than proactive. Moreover, it fails to recognize outstanding group or  individual performance, which tends to undermine motivation and morale within  the organization. This situation is further compounded by the fact that while  approximately 10 percent of all informatics organizations have formal  performance measurement programs, only one organization in six that starts such  a program achieves a successful implementation.

Development and implementation  approach

To address  these problems, AMS undertook a holistic approach to the development and  implementation of a PMS. The objective was to develop and implement a balanced  and systematic PMS to assess the effectiveness of AMS’s operations from various  points of view:

• financial
• quality of programs and services
• client  satisfaction
• employee  satisfaction
• continuous  improvement.

The PMS  was designed to provide feedback at all three levels: strategic, tactical and  operational. It was also designed to evaluate the effectiveness of strategies  and plans, to improve decision making, enable proactive problem correction and  promote continuous improvement within the organization.

Principles

Developing and implementing  a balanced PMS requires that a number  of principles be followed to define the scope of the project and provide a  philosophy of operations. AMS was no different. These principles indicate:

• Performance measurement requires time, effort, skills, expertise and,  perhaps most importantly, the active support of senior management.
• Performance measurement must be organized around the department’s  planning and budgeting cycles.
• Performance measurement business processes for finances, quality of  service, employee satisfaction, client satisfaction and continuous improvement  must be created or adapted to collect and analyze the data.
• An Office  of Performance Measurement (OPM), in this case the AMS Program Office, must be  established and held responsible for the planning, implementation and ongoing  operation of the PMS. The OPM must report directly to, and be supported by,  senior management.
• The PMS  should be developed and implemented in phases.
• Simple and  existing information sources should be used. There is no need to pursue new,  expensive and/or time consuming approaches to data generation.
• The PMS  must be designed so that multiple lines of evidence are generated for each  perspective at different organizational levels and at different times. The  information generated from one perspective must be confirmed by another so that  there is a convergence of information to support the diagnosis and action on any  issue.

Steps in developing and implementing a  performance measurement system

As noted  above, the PMS for AMS was implemented in phases. Phase 1 was a pilot project  used in three core divisions, two offering development services and one  providing support services. In Phase 2, the project was extended to include all  high profile projects, those directly involving clients and some internal to AMS.  This expansion covered most directorates within the sector. Phase 3, now  underway, will extend the project to the two remaining directorates. The steps  used in developing and implementing each phase of the PMS are briefly described  below.

Step 1: Project orientation

The first step was to provide an orientation to management and staff on  the balanced approach to performance measurement. This orientation covered the roles and responsibilities of participants, the stages of development  and implementation, the time frames or milestones, the key principles of  performance measurement, and anticipated and unforeseen issues.

Step 2: Readiness assessment

The second  step was to assess the readiness of the AMS directorates and/or divisions to  accept the balanced approach to performance measurement. This assessment  would demonstrate how the organization’s mandate and functions supported or  reinforced the business plans of AMS, GTIS and the department. Those  responsible for developing the PMS were looking for evidence of management and  staff commitment, as well as an assurance that the necessary resources to  carry out the project would be made available. Following a positive analysis of  the organization’s readiness, it was necessary to select core functional  areas in which to begin a pilot.

Step 3:  Performance Measurement Architecture (PMA)

The third  step was the development of the Performance Measurement Architecture (PMA) to provide the design, content and  structure of the PMS. In developing the PMA,  business objectives were developed first, then informatics objectives were  designed to support the business directions. Performance measures and  performance indicators were then derived for the informatics function. These measures and associated indicators are objective, qualitative,  quantitative and output oriented, but are also quantifiable in nature  (see “Information quality” below for details).

The PMA was an evolving document, changing as  measures were implemented and feedback was received. Once the PMA was  approved, it became the sanctioned PMS architecture for AMS (see Table 1) and it  was important to place it under change control, i.e., proposed changes were  scrutinized and approvals were required.

Step 4:  Implementation

As noted above, Phase 1 began  on a pilot basis in three core areas. This approach was used to demonstrate the  value of performance measurement and to build the competencies of the  performance measurement team. After a successful three-month pilot, Phase 2 was  started. The second phase lasted six months and concentrated on high-priority  informatics projects involving AMS. Phase 3, currently underway, is addressing  the remaining parts of the AMS organization.

The activities required to implement the AMS Performance Measurement  System included:

• Performance Measurement Profile (PMP) – A PMP was developed from the PMA to assess the degree to which existing operational  processes needed to be developed or adapted in order to implement the PMS. The profile also identified areas where information and processes concerning the performance measures and  related performance indicators existed, and where information and processes  needed to be adapted or created (see Appendix). Further, it identified the  availability of baseline and benchmark information.
• Implementation Strategy – An implementation strategy was developed  for each of the performance measurement perspectives:
  • financial
  • quality of  service
  • client  satisfaction
  • employee  satisfaction
  • continuous  improvement.
• These  strategies provide the direction to guide the implementation effort by defining  the roles and responsibilities of participants, the frequency of reporting and  the data collection methods. They also outline the methods for analyzing,  reporting and interpreting the information, as well as for identifying any  issues that could impede progress for each measurement perspective.
• Business Processes – Various day-to-day, month-to month business processes were  developed or adapted to support data gathering for each of the  performance measurement perspectives.
• Information  Capture – Once business processes were adapted or created for each measurement  perspective, the information generated from these processes was captured  according to established reporting formats for each of the performance  measurement perspectives (e.g., monthly status reports, procurement activity  reports and action logs).
• Interpretation – This performance measurement information was interpreted  to ensure internal validity and to provide an comparative external view. The  interpretation of the information was accomplished in several ways: rating  systems, baseline information, benchmarks, experience within the organization  and the use of multiple lines of evidence.
• Reporting  – The monthly performance information relates to financial performance and the  quality of services. A full performance measurement report, the Dashboard, is  published quarterly. Both monthly and quarterly reports discuss the  organizational mission from various points of view. Cumulatively, this  information provides a causal link between the goals of the mission and any  strategic, tactical and operational issues interfering with their attainment.  When issues are raised, a briefing note is prepared and presented to management  in advance for discussion at the monthly review (see Appendix).
• Communication – Communicating results to senior management and staff is  imperative so that. changes can be made to keep the organization on course  towards its mission, to recognize and reward the efforts of individuals and  teams, and to encourage continued positive results.

Benefits

The  benefits of the AMS performance measurement project must be assessed from  several points of view. A discussion of the salient perspectives appears  below.

Knowledge  transfer

There was a considerable transfer of  knowledge from the subject matter experts to AMS staff. Some key areas of  knowledge transfer included:

• the methodology for developing performance measurement  architectures;
• the means to establish baseline information to compare  future performance;
• methods for collecting, analyzing and interpreting  information;
• ways of developing or adapting business processes to  generate performance information related to all measurement perspectives;  and
• the use of facilitation techniques and “Straw Dog Models”  to increase individual and organizational support for the  PMS.

As a  result, AMS is relatively self-sufficient in knowledge and is self-sustaining in  its ability to operate and update the PMS.

Organizational culture

AMS was successful in developing an organizational culture  that values and supports balanced and comprehensive feedback as an essential  element in examining issues and providing the information necessary for  effective decision making. By improving communication, a common language and  understanding developed in AMS concerning priorities, constraints and  opportunities, as well as problems related to finances, quality of programs and  services, client and employee satisfaction and continuous improvement. The PMS  fostered a higher level of motivation and morale among staff through a greater  appreciation of the issues and a greater involvement in formulating strategies  and plans to resolve them. it is now viewed as a necessary and valued component  of the management regime in AMS and has brought discipline to many of the  sector’s business processes.

Decision making

By providing AMS managers with multi-dimensional sources of  information, the PMS provides a framework for decision making on cost  effectiveness; improving, modifying or continuing programs or services; and  improving client and employee satisfaction. It also provides information on  program and service enhancements arising from changing client, employee or technology needs or innovations. Moreover, it provides a means of translating the AMS mission into  concrete objectives, strategies and plans, which can be monitored and adjusted  in response to ongoing environmental changes. Lastly, it provides a frame of reference for management and staff, working together, to sustain and enhance  excellence in program and service delivery to their clients.

Information  quality

The use of indices magnifies the complexity of information generated about any of the  measurement perspectives in the PMS. The PMA for AMS includes four indices:

• Project Management
• Client  Satisfaction
• Employee  Satisfaction
• Continuous  Improvement.

The use of  these indices allows qualitative and quantitative information to be combined,  yet provides the ability to quantify both so that all measures become output  oriented. For example, the Project Management index provides an assessment of  qualitative and quantitative measures of high-priority projects used to  deliver services to clients. Quantitative measures used include whether the  project is on time, within budget, within scope, and whether it meets  all functional and technical quality requirements. Qualitative measures include  the effective use of estimation, risk management, methodologies and tools,  the quality of user involvement, and the effective use of staff and consultants. These dimensions of project management were expanded or decreased in  response to changing conditions or experience. By using an index, these diverse  measures were reduced to a common overall score representing all dimensions of  project management. The use of indices in AMS provides a powerful and  flexible tool to provide comprehensive information on the areas mentioned.

Interpretation of the  information

Several  techniques are used to increase the rigor with which the information  provided by the AMS Performance  Measurement System is interpreted. Rating systems, baseline  information, benchmarks, and organizational knowledge and information from  multiple lines of evidence are used to ensure that the issues revealed by  the PMS are interpreted to ensure internal validity with a comparative  external perspective.

Identification of best  practices

The use  of the PMS has also resulted in the identification of a number of best practices  at both the project and sector level. By sharing these best practices,  AMS has given employees the knowledge and skills necessary to deliver programs  and services in a constantly changing IMAT environment.

Risks/Threats

There were a number of risks inherent in the development and  implementation of a PMS in AMS.

Leadership

Performance measurement requires the active support of  management in communicating the rationale and benefits of the system, to help  breakdown individual and organizational resistance to what may be perceived as a  threatening project. Without the active and ongoing support of management in  promoting the PMS and using the information to improve the functioning of the  organization, the exercise would have been viewed as another form of unnecessary  overhead. Even though there is strong support by the AMS executive for ongoing  performance measurement, there is always the risk that insufficient time will be  devoted to discussing and reviewing information at monthly and quarterly  presentations.

Resources

Developing and implementing a successful PMS requires time,  effort and money. It took just over a year to build the processes and framework  for AMS. For most of that time, the assistance of consulting expertise was  needed until the process became  self-sustaining. In particular, the process  required the support of an already overburdened staff.

Organizational context

The PMS  was built at the sector level but fits within the planning and reporting  framework (Performance Reporting and Accountability Structure) for GTIS and  PWGSC. Many of the financial, human resource and other information systems did  not produce the type of information required at the proper  time or to the required level of quality. There  was also a  reliance on staff members outside the sector, operating in a “virtual organization,” to produce certain types of performance  information. Together, these elements tended to restrict he ability of those implementing the PMS to produce reports with the  frequency and quality desired to  meet departmental business and financial cycles.

Exposure

By publishing monthly and  quarterly performance measurement reports, AMS provides a complete picture of  the financial, quality-of-service, client/employee-satisfaction and  continuous-improvement issues that it has dealt with during that period. By  providing greater exposure to the organization’s strengths and weaknesses, AMS  is identifying the issues and actions required to resolve them. While such  exposure is beneficial to a proactive organization such as AMS, it could be  detrimental when no action is taken, or when issues can only be resolved at the  branch or departmental level.

Conclusion

The AMS performance measurement system is now well launched and is  developing at an acceptable pace. it continues to  provide insights into issues needed to steer AMS towards its mission. It also  provides insights that can be applied to other organizations to assist in the  smooth development and implementation of a balanced PMS, which is portable and  highly applicable to other informatics organizations. To date, the insights and  lessons learned indicate:

• The development of a balanced PMS must be tailored to fit the  unique requirements of each organization in terms of specific measures,  timing, sequence of activities and knowledge  transfer.
• The use of pilots is necessary  to provide evidence of the utility of performance measurement, to build  acceptance and support, and to provide the experience needed to mold the PMS  to the needs of the organization and its  sub-organizations.
• The implementation of a balanced PMS requires the full support of  management in terms of providing the leadership to communicate the  necessity and importance of the approach and to supply adequate  resources.
• The development of a PMA must begin at the top of the organization and be implemented  downwards.
• The development and implementation of a PMS must avoid expensive  data gathering and implementation approaches. Use a “Just Do It” mentality  with existing sources of information, “work around” strategies, and available  methods for interpreting and maintaining progress. Once established, the  performance measurement process tends to be self-correcting and  self-sustaining.
• Most importantly, the performance information, along with the associated analysis, must be used to take  corrective actions in the form of strategies and plans to deal with issues  interfering with the achievement of an organization’s mission.

Endnotes

1. The  first article in this series, “Improved performance measurement: a prerequisite for better service delivery,” appeared in Optimum, Vol. 27, No. 4, pp. 1-5. The  second article, “Implementing a performance measurement system in a public  service informatics function,” appeared in Optimum, Vol. 28, No. 3, pp. 36-44.

2. R.S.  Kaplan and D.P. Norton. The Balanced Scorecard: Translating Strategy into Action (Boston: Harvard Business School Press, 1996).

3. These  numbers represent the total where multiple selections were available and are not  intended to be a cumulative total.

[divider style=’centered’]

Bryan Shane is  a senior partner with BPC Management Consultants. For more than 20 years, be has  provided consulting services in the areas of strategic management, information  technology and performance consulting, to a wide variety of public and private  sector organizations. Mr. Shane has a BA in Political Science from Carleton  University and a BEd and MEd from the University of Ottawa. He has also  completed graduate studies in statistics and evaluation.

Gary Callaghan is the Manager of the Application Management Service (AMS) Program Office for the Government Telecommunications and Informatics Services Branch of Public Works and Government Services Canada. He has over 20 years of experience in the information management and information technology field, all with the Canadian federal government. Mr. Callaghan has been a project manager for the past seven years and has spent the last two years developing and implementing the AMS performance measurement framework.