Public Sector Performance Measurement – Problems and Best Practices

FMI JOURNAL

Public Sector Performance Measurement –
Problems and Best Practices

Bryan Shane and Patricia Lafferty

Introduction

Despite the numerous efforts of public sector departments and agencies, there are few effective performance measurement systems (PM Systems) in place. In fact, while there are many effective PM Systems in organizations world wide, the vast majority of such efforts fail. In short, most PM Systems do not:

• Provide public sector managers with multidimensional sources of information to make decisions on improving, modifying or continuing programs or services;
• Facilitate program savings, efficiencies and elimination of duplication of effort;
• Facilitate proactive identification of issues to be addressed in order to provide effective service to clients
• Provide information on future program and service enhancements arising from changing client, employee or technology needs
• Help provide a business case that can support the creation, continued existence or enhancement of a specific program or service
• Facilitate improved communication and collaboration between the client and the department by addressing issues related to client satisfaction and changing technology, innovations, etc.
• Help to identify the problems of employees in the performance of their daily functions, such as a lack of knowledge, motivation, training or environmental concerns, with the intent of remedying them
• Provide a means of translating the  mission and vision of the organization into concrete strategies and plans that can be monitored and adjusted continually in response to ongoing changes to client, employee, technology, and fiscal requirements and  constraints
• Provide a frame of reference in which management and staff can work together, using a common language, to adjust their operations (through ongoing and balanced feedback) to sustain and enhance excellence in program and service delivery to the public.

• In this article, we will address the systemic issues that hinder the effectiveness of performance measurement and highlight best practices designed to ensure a higher degree of efficacy.

What is Performance Measurement (PM)?

There is still much confusion about performance measurement. Some individuals consider it to be Performance Appraisal designed to assess individual effectiveness. Others consider it to be an automated Management Information System embodied by such tools as Cognos Business Intelligence and other Commercial-Off-the-Shelf -Tools (COTS). Still others see it as a report containing lots of data but no relevant information. The confusing and inaccurate understanding of performance measurement render it among the most threatening and most fiercely resisted of all management disciplines.

In many cases, lip service is paid to its development and use when in fact decisions continue to be made by the “Science of Muddling Through” or by personal and/or impressionistic view of everyday situations. In any case, this lack of understanding severely prejudices the effective use of PM.

An effective understanding of PM has 4 dimensions:

• It is a philosophy of continuous learning in which feedback is used to make ongoing adjustments to the course of the organization toward its mission and vision. The information derived from financial, client satisfaction, pro- grams/services and innovation/learning, for example, provides the feedback needed to assess the effectiveness of an organization from multiple viewpoints. This facilitates ongoing adjustment to ensure continued excellence of programs and services in response to changes arising from both the internal and external environment.
• It is an ongoing process that begins with the setting of a strategic direction (as embodied in an organizations vision, mission and objectives) followed by the development of strategies/plans to achieve these objectives. Finally, performance measures are developed to assess progress.
• It is a structure, whereby the roles and responsibilities to be played by members of the organization -management, the planning/performance measurement staff and the program units- are defined.

Overall, performance measurement is a management system – an ongoing process that provides balanced, methodical feedback to assess the effectiveness of an organization’s operations from multiple vantage points – financial, client satisfaction, service delivery and the employee dimension. The feedback is used at all levels – strategic, tactical or operational – on to assess how well strategies and initiatives are succeeding. This performance information is necessary to improve decision making within the organization, to enable proactive problem correction and to promote continuous improvement.

Understanding and promoting performance measurement as a multi-dimensional discipline increases greatly the likelihood that departments will have a greater appreciation of its essentiality as a management tool. As such, it becomes easier to address the individual and organizational resistances. Without this foundation, building the business logic to develop and implement a PM System is vastly more difficult, if not impossible.

Methodology

Performance Measurement is a relatively new science and as a result there are few methodologies that provide a step by step guide to develop and implement PM Systems.  There is an expectation fostered by many staff members at all levels of the organization that a PM System is a ‘Plug and Play’ initiative.  Nothing could be further from the truth.

Without a methodology in place to define the structure, outline roles and responsibilities and determine the business processes needed to generate, analyze, interpret and report information, and the means to facilitate the establishment of leadership to create an organizational culture supportive of such a highly threatening project, failure is inevitable.

What is required is a step-by-step methodology that builds understanding and support in the organization, while transferring knowledge and skills needed to make it the PM System self-sustaining and self improving.

The balance of this article introduces the reader to a tried and true methodology for the implementation of a vigorous and successful performance measurement system.  It is based upon the Leadership Driven Method (LDM) 1 of developing a Performance Measurement System.

Readiness Assessment

There are certain prerequisites necessary to successfully engage the organization and staff in Performance Measurement. While it is prudent to build upon the existing and valid performance measurement elements, it is often difficult to determine an organization’s level of readiness to undertake such a more robust endeavour in terms of:

• a supportive organizational culture,
• a management team committed to leading and supporting the project,
• an appropriate planning and performance measurement framework,
• a staff skilled in performance measurement etc.

The absence of any of these elements severely limits the success of such an enterprise.

Conducting a Readiness Assessment is a Risk Management tool designed to assess the gap between current and needed performance measurement practices and processes. This assessment is used to:

• Identify both the opportunities and barriers to the development, implementation and acceptance of a PM System within the organization.
• Provide recommendations that when implemented, will allow the organization to successfully develop and implement a PM System.
• Provide an orientation to performance measurement that is designed to promote a consistent understanding in terms of its definition and benefits and to alleviate fears associated with it, what it is and what it is not.

This Readiness Assessment is mandatory to improve understanding and to begin to build support for the project among the uninitiated members of the organization.

PM Strategy

In most organizations there is no strategy that provides a roadmap outlining all the elements of the PM System and the order in which they should be developed and implemented. These elements include:

• The principles and values under which the development of a fully supported PM system will proceed;
• An organizational structure for planning and performance measurement that permits full cooperation and coordination towards common goals;
• A definition of the role and responsibilities for all staff involved in planning and performance measurement in all parts of the organization;
• A methodology outlining all of the necessary phases of development including a project plan, necessary to develop and implement a PM system; and
• Integration strategies to ensure that performance measurement becomes a stable, ongoing and essential element within the management regime.

Effective PM Systems are based upon the foundation of a PM strategy that provides:

• A cooperative, process-oriented approach across the organization to develop and implement the PM system by breaking down individual and organizational resistances to this type of project.
• A blueprint to mange the expectations of all stake holders and to measure progress in the development and implementation of what may be perceived to be a highly threatening project.

Strategy Map

The foundation of a PM System is built upon a results chain that cascades from the vision of an organization right through to its strategies and initiatives (Diagram 1).  The logic and power of the Strategy Map can best be explained as a Business Planning and Performance Measurement Framework which breaks the organization into a logical and interrelated chain of organizational functioning. At its most basic level, the logic of the Strategy Map can be understood as follows:

• Achieving the vision is a function of achieving the mission
• Achieving the mission is a function of achieving the organizations objectives
• Achieving your objectives is a function of defining your strategies
• Achieving your strategies is a function of achieving your initiatives
• Performance measures provide the information needed for decision making in order to navigate through the financial, service delivery, client, employee and external issues hindering you from achieving your strategic direction as embodied in the organization’s vision, mission and objectives

Wherever there is a breakdown in the results chain of the Strategy Map, the possibility of achieving organizations’ strategic direction is nil. Instead, organization functioning is reduced to managing issues on a day-to-day basis and/or in crisis management. In other words, most organizations are engaged in work that is unplanned, uncoordinated, uncooperative or unscheduled.  The organization becomes fixated on the process of doing the work rather than on producing tangible results.

There is a causal linkage between the achievement of an organization’s strategic direction and its strategies, initiatives and performance measures. Each element performs a vital role in cascading chain of causality.  This Strategy Map is a missing or misunderstood component in many public sector organizations.

• Many public sector organizations construct vague, contradictory and shifting strategic direction statements. The absence of an effective strategic direction tends to reinforce the stovepipe concept of competing branches instead of creating a shared and accepted future.
• The organization does not know, or understand the need for a Strategy Map. There is a perception that some elements of the results chain are unnecessary. In extreme cases, the organization functions with a mission or mandate statement and a set of ongoing activities. This is akin to building a house without a blueprint and with only a concept and bunch of busy laborers.
• Unbalanced measurement focused as it primarily is on finances and perhaps service delivery provides a limited and distorted view of the organization. It marginalizes or ignores the contributions of its employees. It ignores the clients to whom service is to be provided. It neglects the effect of external environment factors such as public opinion, shifting economic realities and changing political conditions.
• In some cases, public sector organizations use a program evaluation concept called a Logic Model (Diagram 2) instead of a Strategy Map. The Logic Model is a Program Evaluation tool and is designed to provide program effectiveness information in a longitudinal sense over a period of years rather than on a biweekly or monthly basis.  This tool is inappropriate because its causality chain is designed to infer and/or act as a proxy for results.  For example, activity-based performance measures such as projects on time are used to assess major projects, or a proxy measure such as the development of a risk management strategy is used as a measure of risk management. The Logic Model begins with activities producing outputs and then immediate, intermediate and final outcomes. It is almost impossible to develop any meaningful measures given such an inverted structure.

In reality the understanding and use of the Strategy Map concept is very limited in the public sector as a whole. Because of the resources and leadership requirements needed to introduce it a whole organization, the success ratio is much higher when used at the branch level. Creating a Strategy Map for discrete function corporate functions such as Acquisitions or Information Technology or with a program-specific function holds a much greater promise of success of instituting a successful PM System.

Performance Measures

Performance Measures specify the design, content and structure of the PM System. They are designed to provide feedback on the organizational accomplishments and issues interfering with the attainment of the organizations strategic direction. This feedback is used to provide the information needed to improve financial, service delivery, client/employee satisfaction decision making within the organization. It also enables proactive problem correction and helps to institutionalize continuous improvement. It is the basic ingredient necessary to move public sector organizations from ad hoc, crisis management style to an operational and proactive approach.

Effective performance measures are objective, simple, and quantifiable; outputs/results oriented and include qualitative and quantitative information.  Within the public sector today, performance measures are ineffective for a number of reasons:

• Unless there is a direct causal relationship between an organization’s strategic direction (as embodied in its vision, mission and objectives), the strategies and initiatives used to achieve the strategic direction and the performance measures used to provide the feedback to guide decision making, measures are relegated to the status of data.  All public service organizations have much data but very little useful information of how well their programs are performing or on the status of the employee climate or of client concerns.  Useful information can only be supplied by performance measures that assess the status of strategies and initiatives carried out to achieve organizational objectives and ultimately, its mission and vision.
• Useful performance information must be balanced from four aspects – providing a truer picture of financial, service delivery, client and employee strategies and initiatives. The present practice is to provide a high level view of many dimensions but none in an in-depth manner.
• Besides having many performance measures that are ineffective in supplying needed information to senior management, the quality and quantity of performance measures leave much to be desired. There is a tendency to measure too many dimensions of organizational performance – policy and programs, people, citizen focused service, risk management, stewardship, accountability, learning, innovation and change management.  Measures tend to be developed at the activity level or they serve as proxies for performance measures as previously discussed. As a result, the quality of the performance measures is very limited.
• The timing of performance information is also a major concern.  Information is often late and/or inaccurate in providing a solid picture of the organizational situation. Major decisions then must be delayed or made based upon personal or gut feeling.
• Leadership in supplying effective measurement approaches and tools is also lacking.

Improving the relevance and quality of performance measures is key to providing management with the ability to make effective decisions. This can be done by:

• Constructing a Strategy Map to provide a Planning and Performance Measurement Framework for the organization. The Strategy Map provides a framework for decision making, which illustrates a direct relationship between the achievements of the organization’s strategic direction, the strategies/initiatives required to carry it out and the measures needed to demonstrate progress and make necessary corrections.  Without this framework the quality of performance information can be neither appreciated nor be effectively developed.
• Improving dramatically the quality of the performance measures through the use of index-based performance measures. Such an approach allows the combination of qualitative and quantitative information yet provides the ability to quantify both so that all measures become outputs/results oriented. It is also matches the complexity of the organizational dimension being measured. The diagram below presents a representational view of indexed performance measures

Indexes = Performance Measures = KPIs

An index is a ‘super measure’ or the highest level of measure

Each index is composed of Component Measures

Within each component measure, there are Indictors

Indexes are relatively inexpensive to construct, easily summarized for management yet provide the ability to drill deeper if needed.  They can also can be tailored to the specific level of complexity and can be reported in an easily interpreted manner using a ‘management dashboard’.  A sample of the dashboard concept is provided below using a Project Measurement Index. 2 It consists of four component measures: financial, functional, technical and issue management. Each of these component measures consists of a number of indicators. For example, the financial component consists of the following indicators: on time, within budget and on scope.

Interpreting the dashboard is relatively straightforward.  Green indicates that all is proceeding as planned and there are few, if any issues to be dealt with.  Yellow indicates that there are unresolved issues that are being addressed but that may have an negative impact in the future.  Red indicates that management intervention is required immediately.

Information Requirements

A most overlooked aspect of performance measurement is the information required to support the implementation of a PM system. When it comes to assessing existing sources of information needed to support the approved performance measures, there is almost always a gap. This situation is especially true with the performance measurement information required to assess service delivery, client/stakeholder satisfaction and employee morale.

A Measurement Profile for each performance measure is needed in order to assess the degree to which existing business processes must be developed or adapted in order to generate the performance measurement information for each performance measure. The measurement profile describes,

• where information and processes related to the performance measures exist; and
• where information and processes must be adapted and/or created (gap analysis).

This analysis also defines who is responsible for providing the information, the audience for whom it is intended, and the frequency of reporting.

The completion of the measurement profile provides a complete inventory of the information required to support the PM System. It provides the basis to complete a detailed project plan to develop or adapt business processes required to generate information for each performance measure.

Performance Information Analysis and Interpretation

Statistics gathered, is simply data.  The data need to be analyzed to be transformed into information and the information must be assessed in order to provide knowledge. There are two skill areas that are required: data analysis and interpretation.

Performance information must be analyzed to determine trends or patterns in the information. Two general themes need to be captured: organizational achievements and the issues (related to finances, service delivery, client /stakeholder or employee satisfaction or others). Both dimensions of performance measurement are equally important. However, it is “organizational achievements” that are almost universally given short shrift.  When describing issues usually the most obvious are stated without a further examination and analysis into the root cause, i.e., what are the patterns related to organizational performance among client groups, over time or in different regions. Are client expectations realistic given organization needs?

Given that organizational issues are identified, the next step is to properly interpret the issues. That is to determine “What is a normal level of performance within the organization?”  For example, imagine that the PM System reveals that there is a 10% turnover rate among IT staff. Is that a normal level of attrition for the industry, or the department?  Should the manager focus on trying to retain and attract new IT resources. The ability to properly interpret performance information is the most important and least understood aspect of performance measurement.

It is crucial to develop a broad set of performance interpretation methods to determine a normative level of organizational functioning from financial, service delivery, client/stakeholder and employee satisfaction viewpoints. This normative level is determined using baseline information, benchmarks, service standards, confidence intervals and others. These performance interpretation skills must be developed among key staff members.

Integration Strategies

Unless the PM System can be made self sustaining as an integral element within the organization’s operations, it will ultimately fail. Three integration strategies are required during the entire development and implementation of the PM System:

• Communications The communication challenge within the organization is to change the system of shared beliefs, values, attitudes and behaviors to support the PM system. The PM System represents change to the organizational culture, in some cases an incremental change, in others a major adjustment. Such change generally does not happen easily. It requires leadership of senior management to provide a supportive environment for the change.
• A Supportive Organization Culture The development of a supportive organization culture within the organization is absolutely essential to break down both the organizational and individual resistances to change. While there are numerous techniques that will gradually reduce/eliminate these resistances and increase ownership of the PM system, the most important is “ongoing leadership and communication by senior management” to promote understanding, acceptance and provide financial support in the development and implementation of a PM system. These interventions create a climate of acceptance within the organization by stressing the importance of performance measurement and the need for staff to participate and cooperate fully in this endeavour.
• Knowledge Transfer The transfer of knowledge regarding the development and implementation of the PM system from subject matter experts to staff is required to make the organization self -sufficient and self-sustaining in its ability to operate and update the PM system. The employment of pilots as a means to develop, implement and test the PM system over a period of six months to a year will greatly aid in developing the performance measurement competencies of staff members.

The development and use of these integration strategies throughout the creation and implementation of the PM System is a prerequisite to ensure that the PM system becomes a self-sustaining.

Implications

Why bother with all the effort and expense required to measure performance? This question is ably answered by David Osborne and Ted Gaebler in their book “Reinventing Government. 3

• If you don’t measure results, you can’t tell success from failure.
• If you can’t see success, you can’t reward it.
• If you can’t reward success, you’re probably rewarding failure.
• If you can’t see success, you can’t learn from it.
• If you can’t recognize failure, you can’t correct it.
• If you can demonstrate results, you can win public support

Good performance measurement would provide departments and those who work within them the ability to:

• Use multidimensional sources of information to make decisions on improving, modifying or continuing programs or services;
• Facilitate program savings, efficiencies and elimination of duplication of effort;
• Proactively identify issues and propose solutions to ensure effective service to clients
• Provide information for the business case that can support the creation, continued existence or enhancement of a specific program or service
• Improve communication and collaboration between the client and the department
• Increase levels of employee effectiveness and job satisfaction

Finally, and by no means least, sound performance measurement provides the means of translating the  mission and vision of the organization into concrete strategies and plans that can be monitored and adjusted continually in response to ongoing changes to client, employee, technology, and fiscal requirements and  constraints.

1 Bryan Shane, “Performance Measurement System-A Leadership Driven Methodology”, Optimum Online, The Journal of Public Sector Management, Volume 33,  Issue 3, September 2003.

2 Bryan Shane, Patricia Lafferty,  “Major Project Measurement-An Excellence Driven Approach” Optimum, Online, The Journal of Public Sector Management, Volume 35, Issue 1 , March 2005.

3 David Osborne and Ted Gaebler, “Reinventing Government”, Plume, 1993.
____________________

Contact Bryan Shane or Patricia Lafferty 

FMI Journal Spring/Summer 2010