pmNERDS Community: Login

A+ A A-

Project Ranking Featured

   Why should we spend the time to rank projects? You shouldn’t unless you face a constraint. Constraints make it so we have to choose out of all the things we could do, what things should we do right now, given our constraints in time, money, or quality.

It is in the environment of constraints that we look at choosing one project over another.

   At this point, we ask ourselves, "If we only select the best people, can't they just execute against what they feel are the most valuable, strategically impactful, profitable, time sensitive, least risky, and least expensive projects?" How confident are you that everyone will have the same way of measuring each of these constraints? How does each team member weight the different criteria being used? Some of these may be conflicting constraints. If you want to maximize reward sometimes you must raise your risk threshold. These are just some of the things that can be addressed through ranking projects.

   The Integrated Project Management approach to ranking includes three tools used together to do initial project ranking and assessment. These tools are the Expected Commercial Value (ECV) analysis, Initial Project Assessment scoring model, and the Business Model Canvas.

   This evaluation is continuous and these tools are used to update the project ranking throughout the project’s lifecycle as more details are defined and the information is available. The ECV analysis helps us determine some of the financial constraints of the project and make decisions that impact the technical approach for the project. The Initial Project Assessment helps us evaluate the project from many different aspects, not just financial. This gives us a pretty good project ranking based on strategic fit, risk, and reward. The business model canvas is a communication tool used to collaborate with all stakeholders to clearly define all aspects of the project, from target clients and value propositions to cost structure and benefit streams. Using these three tools allows us to accomplish the three goals of portfolio management: maximize the value of the portfolio; balance risk and reward, and maintain strategic alignment.

Expected Commercial Value

   The Expected Commercial Value (ECV) method seeks to maximize the expected value or commercial worth of your portfolio of projects. Unlike the Net Present Value (NPV) method, ECV takes into consideration probability of technical and commercial success, which is a risk in every project. The decision tree below illustrates how the different factors of ECV map to the lifecycle of a project. You start off spending technology development budget on the project. If the project is a technical success it moves to Launch. The project then spends the Business or Commercialization budget to go to market. If project commercialization is successful it will result in future benefits represented by Estimated Benefit. ECV is a scoring model that can be expressed as a formula which captures the relationships of these estimation factors.

$ECV = Expected Commercial Value of the project
Pts = Probability of Technical Success or (Risk of not succeeding)
Pcs = Probability of Commercial Success (given a technical success)
$T = Technology Development Costs remaining in the project
$B = Business or Commercialization Costs

  With the typical ECV method, you are required to estimate five factors in order to determine ECV: Technology Cost, Business Cost, Probability of Commercial Success, Probability of Technical Success, and PV. With the initial project assessment, we lack a lot of the required information to make these kinds of estimates. So in the Integrated PM method, we use portfolio management principles to help us identify constraints to the project that will fix some of the variables and permit us to finish the equation without estimating some of the factors. For each of your primary portfolios, there are thresholds that are set. In this article, I will suggest three as an example. We can define these three portfolios as Maintenance & Utility, Enhancement & Improvement, and Transformation. These portfolios are defined by a set of state variables, thresholds, and strategic intent, which include accepted levels of risk and expected benefit. We can then fix the value of ECV, as a portfolio threshold, where the value is dependent on what portfolio the project is assigned to. Depending on industry and typical costs of your projects the thresholds will vary, but the ratio between portfolios is typical. The lowest performing projects might be assigned to a portfolio with an ECV below $25,000. Now, the next portfolio threshold would be set at, three times the last portfolio, or $75,000. There is nothing magical about 3, and there are other modeling methods that would rely on statistical processes to set threshold values.

   For our example, let’s set these ECV values for each portfolio as shown below:

Portfolio Minimum ECV
Maintenance & Utility  $25,000
Enhancement & Improvement  $75,000
Transformation  $150,000

 

   Remember, these thresholds depend on your organization and will usually be set by a portfolio manager, strategic planner, or someone else responsible for the overall success of your projects and strategy. By setting a portfolio threshold, you are saying all projects within this portfolio must have at least that ECV. A higher Fixed ECV makes the minimum Expected Benefit higher, the Business Model Canvas more challenging, and your value realization efforts more unlikely.

   These example portfolios will also have acceptable risk ranges set for Technical and Commercial Risk. For example, the portfolios could have the following ranges:

Portfolio Probability of Technical Success Probability of Commercial Success
Maintenance & Utility Greater than 90%, or a 10%risk threshold Greater than 90%, or a 10%risk threshold
Enhancement & Improvement Greater than 75%, or a 15% risk threshold Greater than 80%, or a 20%risk threshold
Transformation Greater than 75%, or a 25% risk threshold Greater than 70%, or a 30%risk threshold

   With a fixed ECV and a range for the probability of success, the estimator only has to come up with two values: Technology Cost and Business Cost, and our formula will calculate an expected benefit for a project assigned to a specific portfolio. Typically, the project sponsor will participate in the discussion about which portfolio to place the project in. So the project manager can do What If Analysis within the targeted risk and cost thresholds. Playing with the estimated costs as well as the amount of risk they can accept, they can see what the expected benefit for the project should be. This is a good gut check. Can this project deliver that large of an expected benefit, if so, with what approach? This will drive decisions in the business model canvas and help set budget risk constraints for the project. Below is an example of three scenarios a project manager evaluated for their project.

 Portfolio  Project Name  Portfolio Threshold (ECV)  Minimum Expected Benefit  Probability of Technical Success Probability of Launch Success  Development Cost  Launch Cost
 E&I  Alpha 1  $75,000  $135.185  .90  .90  $30,000  $5,000
 E&I  Alpha 2   $75,000  $168,289  .88  .85  $50,000  $1,000
 E&I  Alpha 3   $75,000  $225,735  .85  .85  $70,000  $10,000

   By playing around with the different technical approaches, I can see what would be expected of this project if I spent a little more on development cost and was close to the threshold of probability of technical success. In Scenario 3 we used a more complex and risky technical approach that has a higher cost associated with it. The expected benefit for this scenario is significantly higher, but it is doable with that budget. So now these cost estimates become my initial budget constraints and the expected benefit is our project target. The project decisions used in the ECV analysis will be used to perform the initial project assessment as well as develop the business model canvas.

Cost of Technology Development (T)

   Typically, this cost includes requirements gathering, project management, design, development, manufacturing, materials, and resource cost. What are the estimated costs of developing/producing the project deliverables?

Cost of Business Launch/Commercialization (B)

   This is the total Go-to-Market cost, capital costs, customer trials, etc., from market communications to training and delivery. What are the estimated costs of launching this project and enabling target clients to realize the value proposition? 

Probability of Technical Success (Pts)

   Typically, you look at things such as familiarity with technology, alignment to core competencies, existing use of technology within the company, or proven record developing similar solutions. Like the other factors of ECV, this gets further refined in the project assessment with a more detailed evaluation.

Probability of Commercial Success (Pcs)

This factor gets further refined in the project assessment with a more detailed evaluation. This is a percent probability of the commercial success of the project/initiative. Typically you look at things such as familiarity with the market segment, existing sales and distribution channels, or proven record delivering similar value propositions.

Expected Benefit

   This is the total estimated commercial benefit or the project at initial commercialization. It does not look at a 5-year period, it is just looking at the initial year. It is total gross revenue generated by project without taking into consideration development or commercialization costs.

Initial Project Assessment

   The Initial Project Assessment is a tool used to further refine the project details we began to put together during the ECV analysis. The Integrated Project Management approach continues to progressively elaborate on project concepts and assumptions throughout the project lifecycle, to continually gain more insight. This enables decision makers to continually assess projects and make decisions that will increase their competitive advantage; either through improved cost efficiencies or delivery of unique value. Unlike most project assessments and ranking methods, the integrated project management assessment is an effective tool that requires very little resources and effort in order to rank and assess the potential risk and reward of each project in the portfolio. This assessment uses a scoring model to evaluate projects in five areas: Business Strategy Fit, Strategic Leverage, Probability of Technical Success, Probability of Commercial Success, and Reward. In each area, we identify the key success factors that will lead to project success. Depending on the portfolio the questions and scoring values for each of the five key areas will be different. The Maintenance & Utility portfolio would have a much shorter assessment than the Transformation or Enhancement & Improvement portfolio. The Transformation portfolio would probably have a different weighting for certain factors than the Enhancement & Improvement portfolio. For this article, we’ll use an example scoring model for the Enhancement & Improvement portfolio.

 Project Name  Business Strategy Fit  Strategic Leverage  Probability of Technical Success  Probability of Commercial Success  Reward  Total Score
 Alpha  17  31  25  42  21  37
 Beta  20  34  28  54  27  44
 Gamma  5  16  13  19  12  17
 Delta  8  26  31  51  24  35
 Echo  11  19  26  33  15  27
 Foxtrot  14  28  8  9  18  24

   In order to compute a total score for each project we take the score for each area and divide it by the total possible in that area. We then multiply the total value by 10 to get a two digit Total Score value. In some organizations, they may add weights to different key areas or even individual factors. The assessment for each project should not take very long to do. Remember, this is the initial assessment when you don’t have a lot of detailed information. If will be refined as we progressively elaborate on the project details. Below is an example of Probability of Technical Success area with its four factors and scoring model.

 Key Success Factors  1 4  7  10  Rating   Comments
Technical "GAP" Large gulf between current practice and objective: must invent new science “Order of magnitude” change required    Step change short of “Order of magnitude” Incremental improvement; more engineering, less business focus      
Project Complexity Difficult to define; many hurdles Easy to define; many hurdles  A challenge but “do-able”  Straight forward    
Technology Skill Base Technology new to the company; (almost) no skills Some R&D experience but probably insufficient  Selectively practiced in company  Widely practiced in company    
Availability of People and Facilities No appropriate people / facilities will have to build or acquire Acknowledged shortage in key areas  Resources are available, but in demand; must plan in advance  People / facilities immediately available    

   Based on the scores given for each success factor, we are able to calculate a score for each of the five key areas. So for the Probability of Technical Success key area above, we would have a score of 22 out of a potential of 40. So this project is a fairly risky project in two areas and will need risk mitigation to address technology development, resource allocation, and competency development.

   The Initial Project Assessment scoring method enables teams to quickly assess and prioritize their projects without spending huge amounts of time and resources on analysis. If you leveraged the Delphi process you are able to gain valuable insights into the contributing factors of these scores. The Delphi process would have you identify a group of potential experts. These experts would use this scoring model to assess each project on their own. You would then discuss the scores of each expert with them individually to uncover the reasoning behind the scores they gave. This initial assessment will also foster thought around all aspects of the project, not just the financial side. This discussion could lead to additional project constraints or even innovative ideas on how to address concerns identified.

Business Strategy Fit

   When a Program Office asks if the project is strategic, they normally provide a definition which will include some type of scoring model for cost, schedule, and quality- which is a proven best-practice. We have found that prior to these assessments, while information about the project is still unknown, we have to still rank, gain approvals, and allocate time and resources to continue the project definition. In this case, the Business Strategy Fit is a meaningful abstraction of what the Program Office needs.

   There are two estimations, each taking about 30 seconds, to establish Business Strategy Fit. Estimate strategic alignment, and then the impact this project will have on the strategy. Later, you may or may not need to go into more detail.

Congruence (Alignment)

   The congruence factor is used to evaluate how well aligned the project is with long term strategies. This factor determines if a project is moving in the desired direction. Essentially we want to know if this is congruent with our selected strategic path, slightly off, or taking us in a completely different direction.

Impact

   Once we know we are going in right direction, we want to understand how fast this project will help us get there. The impact factor measures how much the project contributes to our long-term strategy. The impact is about how critical this project is to achieve strategic goals.

Strategic Leverage

   This is all about market pressure, where the market is the entire organization, not just your client. How much can the organization leverage the work you’re doing in this project in other projects and initiatives currently and in the future? Of course later, once more definition is available, your project may become part of an enterprise roadmap, where all kinds of projects will leverage your work. For now, the question is just, “Could this project contribute to the enterprise roadmap?” It raises a flag and helps the rest of the organization anticipate the project downstream as early as possible.

Proprietary Position

   Does the project help us establish a proprietary position that is hard for our competitors to imitate? This factor assesses how well protected this project makes our position. Does this project help to build our competitive advantage?

Platform for Growth

   Strategic leverage also looks at the overall opportunity for growth this project represents. If this project is a one-off project that we won’t be able to reuse or leverage for other future opportunities, we will rank it lower. A project with a strong Platform for Growth factor ranking, will open up new channels or provide access to many market segments.

Durability (Tactical and Market)

   Durability measures how long the value proposition of this project will continue to be valued. Essentially, it is determining how long before this project will become obsolete. In the case of new cell phones, for example, the durability of a new cell phone product initiative is only about 12-18 moths. This is the lifecycle of this project. Will it be something that will continue to hold value and be improved over a long period of time?

Synergy with Other Operations within Organization

   The synergy factor is all about how wide of an impact this project has. We use this factor to determine if the work products of this project can be leveraged by many customers, not just the initial project costumers.

Probability of Technical Success

   We build insight into the original estimate of Probability of Technical Success from the ECV by applying the principle of progressive elaboration and then define the general technology arena. Understanding the technology arena enables the project manager to select a high-level definition of the project technical approach. This is leveraged further by many innovation methods, budgeting and cost estimation processes, technology roadmapping, and risk management programs. To describe the technology arena we quickly estimate four technology dimensions.

Technical “Gap”

   The Gap factor, measures where we are technically and how big of a gap this project requires us to fill. Essentially we are measuring the amount of change risk this project has from a technical perspective. (Holistic market – New to World – Levels of Innovation)

Project Complexity

   The next factor measures the relative complexity of this project. Is the technical approach pretty straightforward, or difficult to identify and implement?

Technology Skill Base

   The technology skill base factor is a way of determining how aligned the technology required by this project is to your core competencies. Is this a technical skill that is used on a daily basis? Do you know any other people or organizations that could help mentor your team? If a project requires you to develop a brand new competency, the probability of technical success drops drastically.

Availability of People and Facilities

   For a project to be technically successful, all the needed resources, from people and materials to facilities, must be available for use. If the project requires a resource with very little to no availability and in high demand, the amount of technical risk increases.

Probability of Commercial Success

   Many managers completely ignore this estimation, yet it’s these same managers who walk away frustrated that they can’t get their arms around a business value realization program. It begins here, as you gain increased insight into your ECV’s, Probability of Business Success. In turn, this gained insight is used to drive the Business Canvas Model, which becomes the foundation of your business value realization program.

   From a value definition process, these estimates are critical. In a ranking process, of course, you’d want to load your project portfolio with those projects with the most business value, and likeliness to deliver the value, at the top. One of the great things about this process is that it builds on the previous discussion to provide further direction and insight, while at the same time providing an actionable ranking of projects early on, in an information-poor environment.

Client Need

   Does this project fulfill an existing customer need and is the 'product need' fit obvious. Client need is about measuring how actively clients are looking for a solution to fulfill this need. Will it require evangelizing and developing market, or are there other products in the space that customers are using?

Client Need Maturity

   Client needs have lifecycles similar to products. This factor is measuring the growth rate of this client need. Similar to product lifecycles, client needs go through the same stages: Embryonic (Introduction), Modest Growth, Rapid Growth, Maturity, and Decline.

Competitive Intensity

   Competitive intensity is measuring how much competition you will have to compete against. This is not just competitive saturation, but have aggressively they are targeting the same customer segments and client needs.

Commercial Applications

   This factor is all about how wide an impact and how many potential commercial applications this project has. We use this factor to determine how the work products of this project can be leveraged by many customers, not just the initial project costumers.

Regulatory/Social Political Impact

   Taking into consideration regulatory, social, and political issues, what kind of impact does this project have in relation to key issues? This ranking ranges from negative, neutral to positive impact.

Reward

   Well, did you get what you wanted? Was your project successful? What was it worth to the organization? What value does your project team deliver to the organization? You’d be surprised how many project managers can’t answer even one of these questions. Of course, this is the section that initiates the thinking that provides direction and insight to this type of thinking.

   In our ranked project portfolio, if we ignore reward, how will we maximize reward while minimizing risk within current business constraints? How will your project portfolio deliver increased competitive advantage? Your right, only accidentally, and no one wants to be in that position.

Contribution to Profitability

   This factor measures the profitability of the initiative and how much it will contribute to the organization’s profit margins.

Technology Payback

   This factor measures the amount of time it will take to recoup the cost of the technology investment the project requires.

Time to Commercial Start-up

   How long will it take the initiative to launch and start generating revenue from the time the project kicks off?

Business Model Canvas

   Essentially there are three reasons to complete the building blocks used in the Business Model Canvas (BMC), Focus, Focus, and Focus.

  1. FOCUS - Spin a business focus on the Primary Purpose of the project if there isn’t one already, and sharpen the focus if there is. In turn, this drives the focus of the remaining project planning activity, which is expected to drive the impact of the project.
  2. FOCUS - Leverage a structure of business decisions that lead to actionable activities in the project. The concepts of the BMC drive initial WBS activities, project budgets, the communication plan, and resource assignments with the focus provided by strategy, not convenience. The BMC captures the strategic intentions of the Sponsor and embeds strategically-aligned decisions into them. This results in a highly business focused structure to build the rest of the project plan on.
  3. FOCUS - The completed BMC provides a document that can quickly be referenced during the project execution to provide a business focus to day-to-day decisions, which assures a focused business alignment throughout the project’s life.

   The BMC is made up of 9 building blocks which help lead a business discussion, each building upon the next until answering that all elusive question, “Where is the estimated expected benefit coming from?” It is not an entire Business Case, but if later it becomes necessary to have one, the BMC is your ‘Go-To’ document and is seen as the initial business case: a summary of all the previous thinking, and a launch pad for future opportunity definition.

Value Propositions

   All projects cause change. The value proposition answers the question, “Why change”. It offers a benefit to making a change. In this building block, the change is stated, then the value is categorized, and the benefit source is identified.

We place the requested change into a familiar for quick recognition, and equitable comparisons.

“IF [desired change], THEN we [derived value], BUT [conflict]. “

If there were no conflict in terms of technology, budget, access, permissions, skills, usage, or something, then there would be no reason for the client to be requesting a project (or Change), they would already be doing it.

Value propositions can be categorized into three basic types: Scale of Economy, Familiarity, and Uniqueness.

Value comes from one of the following sources:

  • More value for less cost
  • More value for same cost
  • More value for more cost
  • Less value for less cost
  • Less cost for same value

Targeted Clients

   Within this building block, you want to describe the segments of who the project will benefit, in terms of their authority, organization, and what the client is seeking. Many people refer this block to this set of people as a client segment.

   The Authority component will be one of five possibilities; C-Level, Executive, Director, Manager, or Worker.

   The Business Unit should be something like Operations, Sales, Marketing, Engineering, Development, Distribution…

   You will also need to identify what they want:

  • Insight into the issues and alternatives of a solution or change
  • project plan for making the change
  • Execution of change
  • Insight & project plan for making the change
  • project plan for making the change & Execution of change
  • Insight & project plan for making the change & Execution of change
  • An example of a targeted client is “A Manager of Marketing who is seeking a project plan for making the change & Execution of change.”

Communication Channels

   Project information is delivered to project team members, targeted clients, sponsors, and stakeholders through this physical infrastructure. This is the technology and physical assets used to transfer information, both internal and external to the project. Examples of these communication channels are:

  • Document storage devices such as a local server, DMS, DAM, SharePoint, Dropbox, Box, and Google Drive
  • Network access, VPN, FTP, Website
  • Email
  • Phone System
  • Reports
  • Dashboards
  • Roadmaps
  • Project Plans

Relationship Building

   This block includes all the activities established and maintained with each targeted client segment, sponsor, and stakeholder. The analysis of these relationship building activities in the canvas includes looking at customer expectations, service levels, customer experience, and cost to maintain these relationships.

   Some potential relationship building activities that cause project stickiness are:

  • Project status meetings,
  • Steering committee meetings,
  • Webinars,
  • Training sessions,
  • Roundtables,
  • Focus groups,
  • Peer reviews
  • Gate reviews

   These activities are transferred to the project’s WBS and defined within the project’s communication plan. They provide the structure for more efficient information flow through your project. They help prevent the loss of executive sponsorship and controls the expectations of the clients. They increase the perceived value of the project.

Key Resources

   Key resources are those physical, financial, intellectual, or human assets of the organization that the project must have access to or it will fail. The job role “Welder” is not a Key Resource, but Jack Bennet who is a Welder is a key resource if no other welder will do.

   This is used by the capacity planner when scheduling the start date of this project. If other projects also require this resource, then they must be scheduled differently. The risk manager will identify this dependency as a risk, and normally require a mitigation plan before investing.

Key Activities

   Key activities are the most important actions a company must take to operate successfully and support all of the other building blocks in the business model canvas. These are activities that you must complete successfully for your project to be considered successful. Many times, your project deliverables would be considered “Key Activity”. Depending on the type of project, the performance of a service is a “Key Activity”. These are typically the activities or milestones places on the enterprise roadmap.

   Provides direction for the construction of the WBS, and enables coordination of lessons learned, shared assets, and go to market strategy. Facilitates what goes into the enterprise roadmap and levels of client endorsement of the organization.

Key Partnerships

   Most of the time a key partnership would be internal business units; e.g. Sales & Marketing may be required to develop a new CRM tool. However, they also include job roles that must be outsourced, consulting firms and rental agencies. They also would include strategic partners with other companies used to complete offerings that your organization only delivers a part of the complete product.

   Communication channels must be developed and Relationship building activities within the communication plan must be developed for this groups along with your targeted clients. A common mistake is to leave out the stakeholders and project sponsor who should be listed here.

Cost Structure

   The cost structure identifies the most important costs incurred while operating under this business model. The easiest places to identify these costs are the Key Resources, Key Partners, and Key Activities building blocks.

Benefit Streams

   Benefit streams result from value propositions being successfully delivered to targeted clients. This building block is about identifying all the potential ways this initiative could generate revenue. These are types of revenue source. Some example revenue streams are licensing, usage fees, subscriptions, and asset sales.

   The manager and sponsor should get together and ask the question, “Where do you see benefits coming from, and what is that benefit’s value to the enterprise. In general, there are six generic sources of value;

  1. Increases in Revenue
  2. Decreases in Costs
  3. Increases in Sustainability
  4. Increases in Efficiencies
  5. Decreases in Risk
  6. Increases in Client Endorsement

   Every benefit stream should be able to be put into one or more of these generic source buckets. Benefit Streams are specific, with lasting duration, types of benefit sources. Some examples of benefit streams are:

  • Increases in Revenue through software licensing.
  • Increases in client endorsement through continued relationship building activities
  • Increases in Sustainability through increased market access

Conclusion

   As project managers are continually asked to do more with less in order to increase competitive advantage, they must evolve and adopt practices from many different disciplines. This integrated project management approach avoids the pitfalls of many of the current methods used to rank and evaluate projects in their initial phases. As we progressively elaborate on our project definition, beginning with the ECV analysis, we are able to quickly and continually assess and rank our projects. Building on the information gathered from the ECV Analysis and the Initial Project Assessment, the building blocks of the Business Model Canvas help us to evaluate, plan, and refine our project concept, technical approach, and go to market strategy. The integrated project management approach to the initial ranking of projects requires very little time or resources, but it is very effective at getting to the information needed to help make decisions about projects. It allows us to focus, focus, focus in preparation for project planning, execution, and delivery.

 

Last modified onThursday, 11 May 2017 18:12
Derick Workman

Derick Workman is a product management and marketing professional with more than 13 years’ experience on the leading edge of product, project, and portfolio management. He has worked side-by-side with leaders at some of the most innovative companies in the world. Derick is the lead Product Manager of pmNERDS training courses. He works with Instructional Designers and Content Experts to deliver the best learning-experience to our members possible while balancing today’s business and technology constraints. Derick has received certification and training in Pragmatic Marketing, Progressive Elaboration, StageGate, PDMA certification in New Product Development Processes, Six Sigma Business Score carding, Project Management, Cost Estimation, Agile Development, Lean Six Sigma Logistics, and Process Improvement. He is a certified Workfront consultant. His interests reside in Supply Chain Management, E-Learning, logistics, Value “Stream” Chain analysis, Information Asset Management, Open Innovation, and Operational Research.

Website: www.linkedin.com/in/derickworkman

Leave a comment

Make sure you enter all the required information, indicated by an asterisk (*). HTML code is not allowed.

X