Major Project Management and Measurement: A New Approach

Brian Shane, Patricia Lafferty

Vol. 36, Issue 4, Dec 2006

Major projects are usually complex, multi-year endeavours worth millions of dollars and range in type from those related to information technology to construction. They are also temporary endeavors common to all organizational environments. Projects have specific objective(s), defined start and end dates, consume monetary, human and other resources, and create client -specific products or services. Even though projects are common and ubiquitous in most organizations and project management is a fairly well understood discipline, most major projects routinely fail either partially or completely. More specifically, projects succeed or fail based upon whether they are delivered on time, within budget, based upon original scope, and whether there are functional or technical defects or major issues that impede their successful completion.

Project failure costs billions of private and public sector resources each year around the world. The benchmark failure rate for major projects is 80 to 90 percent.[1]

This article addresses the development and implementation of a new approach to major project management and measurement that is designed to radically improve the probability of success. The management approach is based on the dynamic baseline model (DBM) [2] while the approach to measurement is based upon the excellence driven approach (EDA).[3] Together, these two methods provide an integrated and comprehensive way to both manage and measure major projects.

The dynamic baseline model

Under the current paradigm, there is a tendency to turn to standard project management practices as the “one-size-fits-all” solution regardless of whether a product is being assembled, a building constructed, an IT system developed or a business transformed. Notwithstanding the diligent application of classical project management rules and methods, today’s environment of fast-paced change often precludes the use of this stable and sequential approach.

Through the DBM concept, it is suggested that not all projects are created equal and that tailoring the project management response to the complexity of a given project scenario is key. Projects can be logically and simply grouped into one of five discrete levels of complexity using the dynamic baseline model. This categorization can help determine whether to proceed with a project, how best to proceed, and where to focus management attention for optimal performance.

The initial project assessment places the project in the appropriate project management level. A diagnostic is conducted through interviews with project representatives to characterize the impact of the project concept on its surrounding organizational environment and the state of technological maturity. This establishes the basis for complexity classification, solution tailoring and the appropriate measurement approach. Initial assessment categorizes a project in one of five levels:

• Level 1 is a project having a standard product and a stable proven design. Operation would be highly routine and systematized with standard processes and operate within the bounds of relatively fixed rules. Here is process management that establishes an implementation model applicable to initiatives where product design and functionality is mature and readily understood. The standard rules and regulations are the correct reference for conduct of the work and the nature of the challenge is one of logistically bringing the factors of production together to maximize productivity. Level 1 rule-based solutions feature maximum predictability and outcomes generally surpass initial project expectations.
• Level 2 is a project having a tangible product, an evolving design, stable technology and low integration. Project results are foreseen to operate within broader bounds of relatively fixed methods. At level 2, project management establishes an implementation model applicable to initiatives that are custom in nature. Here the standard rules and regulations don’t sufficiently address the circumstances of the initiative and the standard and routine practice must be dynamically altered accordingly. This is the level appropriate for the classical project management response addressed earlier. The nature of the challenge at level 2 is in planning and implementing careful organization of people and materials to ensure the planned outcome. Level 2 methods-based solutions feature reasonably high predictability and outcomes generally approximate the initial project expectations.
• Level 3 is a project with evolving requirements and a containable total system responsibility. It entails closed system engineering with significant internal integration risk. It generally features a semi-tangible product and leading edge technology. At level 3, project management establishes an implementation model applicable to initiatives that are not only custom in nature, but also feature speculative applications of technology, materials and developmental functionalities. Through the ensuing investigative developments, the originally planned methods of project implementation must be dynamically altered to maintain the overarching mission objective. The nature of the challenge at level 3 is in setting the project strategy that will guide the evolution of the work to the required mission. The solution approach is cyclical in nature and usually features the trial and error application of developmental technologies. Level 3 objectives-based solutions feature high unpredictability, a relatively high rate of project failure to meet initial expectations.
• Level 4 is an evolution project. The end product deeply affects many people, changing, in a fundamental way the manner in which they conduct business and with that, their ability to achieve their objectives. These projects must simultaneously consider project issues and severe implications to routine business. At level 4, portfolio governance establishes a model for situating a new business approach within an open network of stakeholders. The external environment has more control on the situation than the internal project team and, for this reason, governance as opposed to management becomes the key approach. Through the interplay between the project practitioners and the environment stakeholders, objectives dynamically evolve under the overarching common reference of corporate principles. The nature of the challenges at level 4 is in seeking the right balance between project functionality and stakeholder common interest, while facilitating a cultural shift within the environment that will enable the new mode of operation. Level 4 principle-based solutions have consistently had no chance of achieving initial project expectations. The basis for proceeding is to revitalize the project organization within the surrounding business environment. Initial project targets of cost, time and functionality are not the basis for measuring performance and determining success.
• Level 5 are external governance projects operating outside the bounds of corporate principles. At level 5, public governance establishes an implementation model for the harmonization of traditional corporate culture with the overarching societal values. This level, though often beyond the realm of normal project concerns, establishes a familiar end point to the complexity continuum.

• To sum up: the lower the baseline in the DBM hierarchy, the simpler the project, vis-a-vis predictability, planning and the likelihood of success.

The following diagram illustrates the dynamic baseline model.

Excellence driven approach to major project measurement

Based upon the DBM classification, level 1 and 2 projects are measured using the excellence driven approach to major project measurement.

Definition

EDA is a philosophy, a process, a structure and a measurement system. It is:

• a philosophy of continuous learning in which feedback is used to identify achievements and to make adjustments in response to ongoing changes and risk;
• a process that begins with the development of a project plan that includes objectives, budget, schedule and deliverables and is followed by the development of a performance measurement (PM) system and an implementation strategy. It is a process in which all project plans are linked through a feedback process. The performance measures provide the feedback necessary to improve decision making in order for the project to progress towards the attainment of its objectives stage by stage; and
• a structure in which there are distinct roles and responsibilities that must be played by project management, clients and the project management office (PMO) to gather, analyze and report on project performance.

• Overall, the EDA to major project measurement is:

• a measurement system that provides a balanced and systematic attempt to assess the effectiveness of a project's progress from multiple points of view: financial, functional, technical, risk management and client satisfaction. It provides the feedback at all levels - (strategic, tactical or operational) on how well project deliverables are being carried out. It provides the essential feedback to improve decision making within the project by enabling proactive problem solving and by institutionalizing continuous improvement.

• EDA project measures

Under the excellence driven approach (EDA), major projects are measured using a project measurement index (PMI). The PMI specifies the design, content and structure of the performance measurement system.

The power of the PMI lies in its ability to aggregate and quantify a series of related quantitative and qualitative project performance measures to derive an overall score representing all the measures. It is able to match and measure the complexity of the major project in a simple and accurate way.

This EDA approach assesses major projects based upon a PMI which has five performance measures and accompanying indicators as outlined below.

Financial performance refers to project expenditure, schedule and scope measures:

• expenditure performance measures refer to actual versus planned expenditures as defined in the project plan. As the project progresses, decisions regarding procurement, design, development, deployment etc, will be assessed with respect to their impact on expenditures.
• schedule performance measures refer to the timely completion of project deliverables as compared to a baseline schedule defined in the project plan.
• scope performance measures are primarily concerned with product scope (the set of functions and features that characterize the product or service) and project scope (work that must be accomplished to deliver the product/service with the specified functions and features). Scope is measured based upon the degree of compliance of baseline product/service features and functions with proposed project deliverables (the means used for their delivery).

• Functional quality refers to the quality or correctness of the products and/or services functions/features delivered as a result of the project. Functional quality can be measured by comparing the quality or correctness of the baseline product and services features/functions to the proposed project deliverables. It can also be measured using the number of product and service change requests made, approved, and effectively implemented. Or it can be measured using the number of critical, serious and non-critical defects outstanding and resolved on a weekly basis.

Technical quality refers to the technical infrastructure that provides the foundation for product and service delivery. In the case of an information technology project, technical performance would be measured using such indicators as system availability, downtime, problem resolution, and response time and network utilization.

Issue management refers to the identification and resolution of issues or exceptions that are impacting the successful delivery of the project. The purpose of issue management is to ensure that all matters requiring resolution, decisions or direction are addressed as soon as possible to avoid negative consequences on project objectives and deliverables (cost, schedule, scope or products/services).

Client satisfaction performance measures. Often certain project services or products become operational during the life of the project. It is essential that these ongoing products or services be measured as well as ongoing project progress. Failure to deliver product or service offerings and/or effectively deal with problems during the course of a project will have disastrous results for ongoing project progress. Client satisfaction measures include client perceptions on various aspects of achieving a high degree of client satisfaction with implementation support and operational products/ services.

See diagram 2 for a graphic illustration of the EDA.

Application of the EDA and DBM approaches for project success

The DBM and EDA approaches to project performance measurement apply to any major project or to an inventory of projects undertaken by a public or private sector organization. In order to effectively adapt and implement this new approach to the management and measurement of a major project(s), an implementation strategy is needed. The implementation strategy consists of the following steps:

Step1: Diagnosis of DBM project complexity. It is necessary to conduct an initial diagnostic of the current level of project complexity using the DBM to determine the project classification level. This diagnostic focuses on determining DBM level based upon an examination of the following questions:

• What is the end product?
• Why is it being developed?
• Who is involved?
• Where are its partners?
• How do they plan to conduct the project? and
• When will it be completed and progress be determined

• Step 2: DBM project classification. Based upon this diagnostic, a project is classified at a level 1 to 5. Level 3, 4 and 5 projects cannot be successfully managed or measured using more traditional approaches. Their level of complexity warrants different approaches to stabilize the project requirements . Typically, this involves the development and implementation of a strategic project plan. The strategic plan identifies those parts of the project that can be defined within a stable set of requirements and that can be described in terms of an accurate budget, schedule with a consistent scope. These sub-projects are then reclassified as a level 1 or 2 projects and are measured using the EDA approach. This process continues until all elements of the project are stabilized and can be measured.

All level 3-5 projects that are refined to level 1 or 2 projects have a far greater probability of success.

Step 3: EDA project measurement. For projects or parts of projects classified as DBM level 1 and 2, the application of EDA employs the following steps:

• outline roles/responsibilities. There are three major sets of roles and responsibilities that must be developed with regard to the development, implementation and operation of a major project EDA: PM system-project management, client and PMO. These must be developed for the unique circumstances of the project and then presented, accepted and used. These roles and responsibilities are needed to ensure a high level of coordination, cooperation and communication in terms of planning, development, implementation and operation of the project PM system.
• refine performance measures. It is necessary to revise the PMI measures based upon feedback from management and stakeholders until there is consensus on their acceptability . Some PMI measures may apply to all projects while others will use only a portion. It especially important that client satisfaction measures be developed and accepted by the project management for project products or services that become operational during the life of the project.
• collect the performance data. The data collection strategies for the EDA project PM system include reviews conducted at various levels of the organization and at various intervals. The following are examples of methods used to collect performance information:
  • a project management system to provide information on project costs and schedule;
  • change management database to track scope change requests and their resolution;
  • an issue management system to track the identification and resolution of risks and exceptions;
  • a bug reporting system to provide information on defects and their resolution;
  • a requirements traceability matrix to ensure all product and service features and functions are complaint and functioning effectively; and
  • contractor status reports to provide more complete information on deliverables and their cost, schedule and scope, and functional/technical performance.
• analyze the performance information as a means to identify project accomplishments, issues and trends. Two types of analyses are required:
  • micro-level analysis of the project performance measures provides information on financial, functional/technical performance or risks by pinpointing accomplishments and specific areas of improvement necessary to better develop and implement the project; and
  • macro-level analysis permits the identification of accomplishments and issues/trends for the project as a whole. It helps determine whether there are gaps between client expectations and the quality of project deliverables. It provides a means to deal with macro issues that threaten the health of the entire project plan or strategy.

• Analysis of project performance will result in the emergence of accomplishments and several operational/strategic issues or exceptions that must be interpreted before they are reported to management. Project issues especially require interpretation to determine their relative and comparative importance. In other words, to determine What is normal for this particular project environment? Two approaches are used:

• management interpretation. Using a set of interpretation criteria (such as phase of the project, cost/schedule impact, affect on other deliverables) project management assesses the current state of project progress;
• confidence interval approach. Using the symbolic representation of red, yellow or green, a confidence interval is used to assess project progress. Different confidence intervals would be used to assess financial, functional and technical performance. For example, financial performance can be assessed using the following confidence interval approach.
  • green - 0 to 10 percent actual deviation from planned budget, schedule or scope results in a green, meaning everything is fine;
  • yellow -10 to 20 percent actual deviation from planned budget, schedule or scope results in a yellow meaning that it needs to be monitored and that there is an issue the must be addressed; and
  • red - greater than 20 percent actual deviation from planned budget, schedule or scope results in a red meaning that something is very wrong and must be highlighted for immediate and urgent attention.

• A combination of both approaches may be used. However, the Interpretation approach(s) must be specified in advance of the project and tailored to its specific circumstances.

• reporting. Communicating project accomplishments and issues in terms of financial status, functional/technical quality and issue management to both senior management and staff is critical. Without this type of information the ability to take corrective action to steer the project towards its product and/or service objectives is vastly diminished.

• Step 4: Integration strategies are essential to ensure that DBA and EDA become a self-correcting and self-sustaining element within project operations. These strategies include:

• knowledge transfer. Appropriate staff member(s) must be selected and trained to ensure they have the skills to properly use these approaches through coaching and mentoring.
• supportive organizational culture. The development and implementation of these approaches must begin with senior management and be implemented progressively downwards. It requires the full commitment and support of management in terms of providing a supportive environment, and in communicating the necessity and importance of this approach. It also requires a financial investment to develop, implement and maintain this performance measurement operation.

• Step 5: Demonstrate the effectiveness of the DBM and EDA on a pilot basisbefore full adoption across the project. The use of a pilot approach is necessary to tailor many of the DBM and EDA business processes for measuring and managing projects to the unique requirements of the project. These pilots provide the evidence of the effectiveness of the management and measurement approaches and identify problems/constraints that will impede the effective implementation of these approaches more widely across the project. Finally, pilots build acceptance/support for this endeavor and develop the management and measurement competencies of staff.

Diagram 3 presents a graphic view of the combined DBM and EDA approaches to major project management and measurement.

Conclusion

The use of the dynamic baseline model will greatly increase the likelihood of project success:

• It will improve decisions on whether to proceed with a project by providing a framework to analyze its level of complexity using five-level model. Those projects assessed at levels 3-5 are far less likely to succeed.
• For those projects already under way, the DBM provides a method to manage the project at whatever level of complexity it happens to be. In the case of level 3-5 projects , the focus is on governance and stabilizing the requirements for all or part of the project. The DBM provides the ability to focus management on those elements that need to be stabilized before issues of requirements, budget and schedule become the focus. Stable requirements in terms of deliverables, schedule, budget and scope have a much greater chance of success.

• The use of the excellence driven approach to project measurement for those projects stabilized at levels 1 and 2 provides the ability to identify project accomplishments and issues from financial, functional, technical, issue management and client satisfaction viewpoints. Cumulatively, this information provides a causal link between achieving project objectives and the strategic and operational issues interfering with project success.

More specifically, the EDA provides the following benefits:

• It provides a consistent understanding of project planning and the attainment of project goals. The focus is on identifying and dealing with exceptions that interfere with attaining project objectives, linking the project plan with decision-making, and on identifying and rewarding project achievement.
• It provides a process-oriented approach to developing and implementing a major project PM system that ensures its acceptance through a gradual process of change in organizational culture. Over time, this approach encourages the development of a culture that values and supports balanced and comprehensive feedback as an essential element in both rewarding achievement and providing the information necessary for effective project decision making.
• It provides a framework for decision making in which managers have a basis for making decisions that conform and support the project strategic directions.
• It facilitates proactive identification of issues that need to be considered and addressed by managers in order to deliver the project effectively.
• It facilitates improved communication and collaboration between project management, the contractor and its clients by identifying issues related to its effectiveness.
• It is a relatively inexpensive investment when implemented as part of an overall project management office, in comparison with the cost of major project failure.

• The information generated from the EDA PM system must be used to take corrective action to effectively manage and steer the major project to its successful implementation. Otherwise, the effort is wasted. Where it is demonstrated that project performance information is used to improve the functioning of the project, then EDA PM system becomes fully self-sustaining.