Major Project Measurement:  An Excellence Driven Approach

Bryan Shane, Patricia Lafferty
Vol. 35, Issue 1, Mar 2005

Similarly, the KPMG Canada IT Project Management Survey (1997) of 1450 public and private sector organizations revealed:

• over 61 percent of projects were deemed to have failed by respondents;
• over 75 percent of projects were late in completion by 30 percent or more; and
• over 50 percent of projects exceeded budgets substantially.

Why is there such a high failure rate among major projects?

There are many reasons, including: inadequate project definition, lack of senior management understanding and support, lack of client involvement, inflexible procurement approaches, lack of communication, etc. But the most important reason for project failure is the lack of effective project progress measurement. Project progress measurement is a prerequisite for identifying issues and providing the information needed to effectively deal with any of the above mentioned reasons for project failure. It acts as a self-correcting mechanism to guide and correct the course of major projects from initiation to completion. Major projects are usually complex, multi-year endeavours worth millions of dollars and range in type from those related to information technology to construction.

Definition

It is critical that an excellence driven approach (EDA) be understood as a project measurement system (PM) that is

• A philosophy of continuous learning in which feedback is used to identify achievements and to make adjustments in response to ongoing changes arising from the environment to ensure continued excellent project progress.
• A process that begins with the development of a project plan, that includes objectives, budget, schedule and deliverables. This is followed by the development of a PM system and an implementation strategy.It is a process in which all project plans are linked through a feedback process. The performance measures provide the feedback necessary to improve decision making in order for the project to progress towards the attainment of its objectives stage by stage.
• A structure in which there are distinct roles and responsibilities that must be played by project management, clients and the project management office to gather, analyze and report on project performance.

Overall, the EDA to major project measurement is:

• A measurement system that provides a balanced and systematic attempt to assess the effectiveness of a project’s progress from multiple points of view: financial, functional, technical, risk management and client satisfaction.

• A management system that provides the feedback at all levels – (strategic, tactical or operational) on how well project deliverables are being carried out. It provides the essential feedback to improve decision making within the project by enabling proactive problem solving and by institutionalizing continuous improvement.

Principles

There are a number of principles that guide the development and implementation of an EDA PM system:

• It must support the organization’s strategic priorities.
• A formal champion for major project measurement must be established to lead the development and implementation of a PM system.
• The development, implementation and integration of the PM system require time, effort, skills/expertise and perhaps, most importantly, the active support of senior management and staff.
• Performance measurement is designed to highlight achievements and reveal exceptions that hinder progress toward the attainment of major project success.
• Roles and responsibilities must be established for the planning, implementation and operation of the major project PM system.
• The performance measures must be tailored to the specific needs of the project.
• There must be a cooperative and shared approach to the development and implementation of the project PM system that specifies agreed to objectives, deliverables and performance measures. This approach must also specify the methods to collect, analyse, interpret and report the project performance measurement information.
• Performance measurement business processes must be adapted or created for collecting, analysing, interpreting and reporting the performance measurement information.
• The project performance measures must be objective, simple, quantifiable and outputs/results oriented. They must also include qualitative information.

Benefits

The benefits of using this EDA PM system include the following:

• It provides a consistent understanding of project planning and the attainment of project goals. The focus is on identifying and dealing with exceptions that interfere with attaining project objectives, linking the project plan with decision making, and on identifying and rewarding project achievement.
• It provides a process-oriented approach to developing and implementing a major project PM system that ensures its acceptance through a gradual process of change in organizational culture. Over time, this approach encourages the development of a culture which values and supports balanced and comprehensive feedback as an essential element in both rewarding achievement and providing the information necessary for effective project decision making.
• It provides a framework for decision making in which managers have a basis for making decisions that conform and support the project’s strategic directions.
• It facilitates proactive identification of exceptions that need to be considered and addressed by managers in order to deliver the project effectively.
• It facilitates improved communication and collaboration between project management, the contractor and its clients by identifying issues related to its effectiveness.
• It is a relatively “inexpensive investment” when implemented as part of an overall project management office, in comparison with the cost of major project failure.

Performance measures

Under the excellence driven approach (EDA), major projects are measured using a project measurement index (PMI). The PMI specifies the design, content and structure of the PM system.

The power of the PMI lies in its ability to aggregate and quantify a series of related quantitative and qualitative project performance measures to derive an overall score, representing all the measures. It is able to match and measure the complexity of the major project in a simple and accurate way.

The project measurement index includes the following performance measures:

Financial performance measures

• Expenditure performance measures refer to actual versus planned expenditures as defined in the Project Plan. As the project progresses, decisions regarding procurement, design, development, deployment, etc. will be assessed with respect to their impact on expenditures. Actual expenditures will be compared to a baseline, and any variances will be reported to management for corrective action.
• Schedule performance measures refer to the timely completion of project deliverables as compared to a baseline schedule defined in the project plan.
• Scope performance measures are primarily concerned with product scope (the set of functions and features that characterize the product or service) and project scope (work that must be accomplished to deliver the product/service with the specified functions and features). Scope is measured based upon the degree of compliance of baseline product/service features and functions with proposed project deliverables (the means used for their delivery).
• Functional quality refers to the quality or correctness of the products and/or services features/functions delivered as a result of the project. Functional quality can be measured by comparing the quality or correctness of the baseline product and services features/functions to the proposed project deliverables. It can also be measured using the number of product and service change requests made, approved and effectively implemented or the number of critical, serious and non-critical defects outstanding and resolved on a weekly basis.
• Technical quality performance refers to the technical infrastructure that provides the foundation for product and service delivery. In the case of an information technology (IT) project , such indicators as system availability, downtime, problem resolution, response time and network utilization would measure technical quality performance.
• Issue management performance refers to the identification and resolution of issues or exceptions that are impacting the successful delivery of the project. Issues can be related to communications, human resources, contracts, product/service features and functions, etc. The purpose of issue management is to ensure that all matters requiring resolution, decisions or direction are addressed as soon as possible to avoid negative consequences on project objectives and deliverables (cost, schedule, scope or products/services).
• Client satisfaction performance measures. Often certain project services or products become operational during the life of the project. It is essential that these ongoing products or services be measured as well as ongoing project progress. Failure to deliver product or service offerings and/or effectively deal with problems during the course of a project will have disastrous results for ongoing project progress. Client satisfaction measures include client perceptions on various aspects of achieving a high degree of client satisfaction with implementation support or with operational products/ services.

The PMI provides the feedback to highlight achievements and to identify issues interfering with the achievement of major project objectives from financial, functional, technical, issue management and client satisfaction viewpoints. This feedback is used to provide the information needed to improve decision making within the project and to enable proactive problem correction.

Implementation strategy

In order to use the EDA PM System effectively to assess major project progress, an implementation strategy is needed to make it operational. It contains the following elements:

• roles and responsibilities of project management, project management office and the contractors;
• methods of data collection for PMI measures previously described;
• methods of analyzing the performance to determine accomplishments and issues that affect the delivery of the major project;
• interpretation of the project performance information to determine what is an acceptable level of project performance from financial, functional, technical performance, issue resolution and client satisfaction perspectives; and
• methods for reporting the results of project performance.

Each of these elements will be discussed.

Roles and responsibilities

There are three major sets of roles and responsibilities with regard to the implementation of a major project EDA PM system.

• Project management is responsible for providing the leadership necessary to develop, implement and maintain the new approach to measuring project effectiveness. Management must allocate the resources necessary to carryout project performance measurement within the project. In the case where some products or services will become operational during the life of the project, their performance must be assessed until the project is completed. It must also use the results from PMI to improve its project management practices.
• Project management office is responsible for implementing and maintaining the new project PM system using the financial, functional/technical performance, issue management and client satisfaction measures. This includes: further customizing the project measures to best suit the environment; setting up or amending business processes to collect the data; analyzing project information; interpreting and reporting the results.
• The contractor must deliver the solution, and provide the facts required to assess financial, functional/technical and issue management performance.

Methods of data collection

The data collection strategies for the EDA project PM system include reviews conducted at various levels of the organization and at various intervals. The following are examples of methods used to collect performance information:

• a project management system to provide information on project costs and schedule;
• a change management database to track scope change requests and their resolution;
• an issue management system to track the identification and resolution of risks and exceptions;
• a bug reporting system to provide information on defects and their resolution;
• a requirements traceability matrix to ensure all product and service features and functions are complaint and functioning effectively; and
• contractor status reports to provide more complete information on deliverables and their cost, schedule and scope, and functional/technical performance.

Analysis of the performance information

Analysis of the major project EDA PM system performance information provides the means to identify project accomplishments, issues and trends. Some or all of the following criteria may be used to identify project issues or trends:

• Are there variations in project budget, schedule or scope?
• Is there a high degree of compliance between project deliverables and baseline product/service features and functions?
• Is there a high degree of quality in project deliverables as compared to their baseline product/service features and functions?
• Are there any critical defects that have not been repaired and threaten the functionality or technical infrastructure of the project?
• Are there a large numbers of critical and material risks or issues outstanding?
• Are there variations among clients, within regions, over time?
• Are expectations consistent across client groups?
• Is there a gap between client expectations and the quality of the deliverables? If so, what is it?
• Are standards established on a project basis?
• Are the staff/clients involved in their development?
• Are clients aware of these standards?
• Is management comfortable with them?

As a result of analyzing major project performance using some or all of these criteria, two types of accomplishments, issues or trends can be found:

• Micro-level analysis of the project performance measures provides information on financial, functional/technical performance or risks by pinpointing accomplishments and specific areas of improvement necessary to better develop and implement the project.
• Macro-level analysis permits the identification of accomplishments and issues/trends for the project as a whole. It helps determine whether there are gaps between client expectations and the quality of project deliverables. It provides a means to deal with macro issues that threaten the health of the entire project plan or strategy.

Using these criteria to analyze project performance will result in the emergence of accomplishments and several operational/strategic issues or exceptions that must be interpreted before they are reported to management.

Methods of data interpretation

Project accomplishments and issues, as revealed by the data analysis, require interpretation to determine their relative and comparative importance. In other words, to determine “What is normal for this particular project environment?”.. Using the EDA, the following criteria must be used to correctly interpret project performanceinformation from financial, functional/technical, issue management and client satisfaction perspectives:

• At what phase of the project is the exception occurring?
• What is the cost impact?
• What is its schedule impact?
• What is its impact on the quality of products and services?
• Is resolution of the issue critical to project success? e.g., does it affect the critical path?
• Can the issue be resolved within the project or must it be escalated for resolution?
• Are there repeated occurrences of the same type of issue?
• Does the issue affect other deliverables?
• Is the client satisfied with the products and services?
• Is the present level of uptake and usage acceptable?

Once these questions (and others as determined by the project office) are answered, two methods can be used in combination or separately to properly interpret overall project performance at any one point in time:

• Management interpretation. Using the interpretation criteria listed above, project management assesses the current state of project progress.
• Confidence interval approach. Using the symbolic representation of red, yellow or green, a confidence interval is used to assess project progress. An example of such a confidence interval approach is presented below for financial performance measures.
• Green – 0 to 10 percent actual deviation from planned budget, schedule or scope results in a green, meaning everything is fine.
• Yellow –10 to 20 percent actual deviation from planned budget, schedule or scope results in a yellow meaning that it needs to be monitored and that there is an issue the must be addressed.
• Red – > than 20 percent actual deviation from planned budget, schedule or scope results in a red meaning that something is very wrong and must be highlighted for immediate and urgent attention

If any of these dimensions of major project performance are rated red, then the whole project is rated red at that point in time. Otherwise, mostly green or yellow ratings in each of the 5 dimensions of project measurement lead to an overall green or yellow rating for that point in time.

These methods of interpretation ensure that elements of project performance are treated as an integrated whole. Where one element of project performance is at serious risk, so is the whole project.

Reporting

Communicating the project accomplishments and issues to both senior management and project staff is imperative. The delivery of a project that is on time, within budget, on scope with required products/services, and to the satisfaction of its clients can only be achieved if management is able to obtain effective feedback, provided through all the measurement dimensions embodied in the PMI. Such feedback provides the information needed to manage the project in support of its product and/or service objectives.

Using the EDA to major project measurement ensures project management by excellence and exception . The expectation is that all staff, regardless of reporting relationships, will work effectively to ensure that project deliverables and/or issues will receive an “excellent” rating. Only those few exceptional deliverables and/or issues that threaten the overall success of the project will be addressed by management.

The information generated from these PMI performance measures must be presented according to a predetermined project status report. The EDA project status report presented bi-weekly at a minimum includes the following:

• Financial status including the following elements:
  • Cost: provides information on project planned versus actual expenditures. It also outlines the cost of change requests initiated through, schedule, scope or work results change requests;
  • Schedule: provides information on project planned versus actual schedule related to each deliverable and overall. It also outlines the status of any schedule change requests;
  • Scope: provides information on the degree of compliance of project deliverables as compared to required product or service features and functions. It also provides information on change requests, approved and implemented to project scope.
• Functional quality: provides information on the degree of quality of project deliverables as compared to required product or service features and functions. It also describes change requests and critical/serious defects that hinder the achievement of full project functionality and their degree of resolution.
• Technical quality: provides information on the technical infrastructure. For example, an information technology project may include such items as system availability, downtime, problem resolution, response time and network utilization.
• Issue management: provides information on serious issues that can affect project performance and their degree of resolution.
• Client satisfaction: provides information on client perceptions on implementation support and specific products and services.

See Diagram 1 for a graphic illustration of the EDA to major project measurement.

Excellence Driven Approach (EDA) to Major Project Measurement

Application of the EDA PM system

This EDA to project performance measurement applies to any major project or to an inventory of projects undertaken by a public or private sector organization. In order to effectively adapt and implement this approach to the measurement of a major project(s), a number of immediate steps are required:

• Refine performance measures. It is necessary to revise the PMI measures based upon feedback from the management and stakeholders until there is consensus on their acceptability. While some PMI measures may apply to all projects, others will use all of them. It is especially important that client satisfaction measures be developed and accepted by the project management for products or services that become operational during the life of the project.
• Refine interpretive methods. Criteria used to determine the cumulative and relative importance of project issues until management and stakeholders find them acceptable. Whether “confidence interval approach”, “management interpretation” or a combination of both is used, each must be specified in advance of the project and tailored to its specific circumstances.
• Roles/responsibilities. The high-level roles and responsibilities associated with the development and implementation of a project PM system have been outlined. These must be further developed for the unique circumstances of the project and then presented, accepted and used. These roles and responsibilities are needed to ensure a high level of coordination, cooperation, and communication in terms of planning, development, implementation and operation of the project PM system.
• Integration strategies are essential to ensure that the project PM system becomes a self-correcting and self-sustaining element within project operations. These strategies include:
  • knowledge transfer. Appropriate staff member(s) must be selected and trained to ensure they have the skills to properly analyze, interpret and report project performance information through coaching and mentoring.
  • supportive organizational culture. The development and implementation of a project PM system must begin from the top and be implemented progressively downwards. It requires the full commitment and support of management in terms of providing a supportive environment, communicating the necessity, and importance of this approach. It also requires a financial investment to develop, implement and maintain this performance measurement operation.
  • demonstrate the effectiveness of the project PM system on a pilot basis before full adoption across the project. The use of a pilot approach is necessary to tailor the PM system to the unique requirements of the project. It is also necessary to develop and/or adapt many of the performance measurement business processes for gathering, analyzing, interpreting and reporting on performance. These pilots also provide evidence of the effectiveness of the performance information generated, and identify problems/constraints that impede the effective implementation of these performance measures more widely across the organization. Finally, pilots build acceptance/support for this endeavour and develop the performance measurement competencies of staff.

Conclusion

The performance information generated by EDA to major project PM system outlines the accomplishments and issues from a financial, functional, technical, issue management and client satisfaction viewpoints. Cumulatively, this information provides a causal link between achieving project objectives and the strategic and operational issues interfering with project success. Unless the information generated from the EDA PM system is used to take corrective actions to effectively manage and steer the major project, the effort is wasted. The performance information provided by the EDA PM system must be used to move the project forward progressively until its successful implementation. Where it is demonstrated that project performance information is used to improve the functioning of the project, then it will become fully self-sustaining.