R&D Project Evaluation Tools
Each tool has defined inputs, a defined procedure and a defined output, and they run in sequence: the first three produce the numbers the last two consume. Set out in a 1994 methods synthesis and largely indifferent to the technology of its decade.
The three inputs every project decision requires
R9 treats project evaluation as the joint between two things usually managed separately: allocating resources, and executing the work. The highest-level project decisions — selection, continuation, modification and termination — all sit on that joint, and all of them require careful consideration of the same three things. That includes the decision to stop, which is why the same inputs appear again in Making better project termination decisions from an entirely separate empirical study.
What a decision needs, whatever the decision is
R&D cost and time to completion
Not cost to date and not the original budget. What it will take, from here, to finish. This is the quantity that changes most as a project proceeds and the one most often left as a stale figure in a plan.
Probabilities of success
Plural, and decomposed: technical success, implementation success, commercial success. The source treats these as separable estimates rather than one overall confidence figure.
Potential value given success
The value if the project works — assessed conditionally, not blended with the probability of getting there. Blending the two too early is what makes a business case unarguable.
How the five tools chain together
The tools are not alternatives to one another. Three of them generate structure and estimates; the last two consume those estimates and produce a decision. Run out of order, or with a link missing, the final number inherits nothing but the team's original opinion — dressed up. What the chain is trying to produce is a decision that would survive scrutiny on all six of the criteria in Decision quality in R&D.
- Strategy table
- Influence diagrams
- Sensitivity analysis
- Decision tree
- Expected value
THE FIVE TOOLS — INPUTS, PROCEDURE, OUTPUT
| Tool | Inputs | Procedure | Output |
|---|---|---|---|
| 1. Strategy table | Every major product design decision; the team's competing views of the product | Make each design decision a column; list a wide range of options beneath each; trace alternative product concepts as paths across the columns | A set of coherent alternative product concepts, and a visible map of where the team disagrees |
| 2a. Commercial influence diagram | The team's collective view of what drives commercial value | As a group, identify and prioritise all possible drivers of value and the factors creating uncertainty in it | A qualitative priority order of issues, and a blueprint for the business model that will value the project |
| 2b. Technical influence diagram | The technical hurdles that stand between here and a working result | Identify every hurdle that could block success; optionally assign a probability of overcoming each | A defined and, where quantified, measured probability of technical success |
| 3. Sensitivity analysis | Low, median and high values for each influential variable from the commercial diagram | Vary each in turn against the value model; sort the resulting bars by impact | A ranked picture of which uncertainties move commercial value most |
| 4. Decision tree | R&D cost, probability of success, potential commercial value, plus decision and outcome branches | Lay out decisions and chance events in sequence; value the endpoints; roll back | A structured comparison of the alternatives, including the no-R&D branch |
| 5. Expected value | The rolled-back tree | Probability-weight the outcomes and net off expected R&D cost to completion | A single comparable figure per project, and the productivity measure the portfolio view needs |
Tools 4 and 5 are fed by tools 1 to 3 plus the team's own experience and judgement. The source does not present them as a substitute for judgement.
Tool 1 — the strategy table
The strategy table is the cheapest of the five and the one most often skipped. Its purpose is to clarify a project team's vision of a new product, surface differences of opinion, clarify goals, and generate alternative product concepts.
Building and using one
Make every major product design decision a column
Not features — decisions. Each column is a dimension on which the product could differ.
List a wide range of design options under each column
This is the creativity aid. The width of the option lists is what opens the design space; a table with two options per column has not done its job.
Trace alternative concepts as paths across the columns
One choice from each column, joined up, is a coherent product concept. Several such paths are the alternatives you will actually decide between.
Have each function trace its own path
The source describes this second use as even more important than the first, because it makes disagreement visible instead of latent.
The worked instance given is a substantial divergence between an R&D manager's vision of a product and a marketing manager's vision of the same product — two paths across the same table, discovered before funding rather than after. The source notes that similar tables can be built for technical and market alternatives, not only product design.
The strategy table's role in setting up a portfolio decision, rather than its construction, is treated separately in R&D portfolio displays and strategy tables.
Tool 2 — influence diagrams, commercial and technical
Two distinct diagrams do two distinct jobs. Conflating them is a common error, because they look alike on a whiteboard and answer completely different questions.
Commercial influence diagram
- Question: where does commercial value come from, and what makes it uncertain?
- Explicitly a group product, not an analyst's deliverable.
- Must identify all important factors creating uncertainty in the value of the project if commercialised.
- Always consider market, competition, manufacturing and technical issues.
- Regulatory, environmental and financial issues sometimes emerge as well.
- Two outputs: a qualitative prioritisation of the issues, and a blueprint for designing the business model that will evaluate potential commercial value.
Technical influence diagram
- Question: what stands between us and a technical result, and how likely are we to get past it?
- Defines and quantifies the technical hurdles determining probability of technical success.
- Assists early identification of every hurdle that could block success.
- Hurdles can be identified qualitatively, or — the source says more accurately — by assigning probabilities of overcoming each one.
- Feeds the probability input directly.
- Its by-product is a work-sequencing argument: you now know which hurdles are decisive.
The technical diagram supports a decomposition that is worth adopting on its own. Probability of introduction success is defined as the probability that the project will be technically successful, be commercialised, and reach some specified level of profitability. That typically breaks down further into research success, development success, implementation success and commercial success.
Tool 3 — sensitivity analysis, and the two ways to read a tornado
Sensitivity analysis takes the variables identified in the commercial influence diagram and tests how much each one moves the answer. The output is the familiar tornado chart: one horizontal bar per variable, showing the impact on commercial value — project net present value if successful — sorted by size.
THE ASSESSMENT CONVENTION FOR EACH UNCERTAIN VARIABLE
| Assessment | What it means | What it is not |
|---|---|---|
| Low | The 10th percentile value | The worst case, or the lowest figure anyone in the room will defend |
| Median | The 50/50 value — as likely to be exceeded as not | The average of the low and high figures |
| High | The 90th percentile value | The best case, or the number in the original business case |
A stated assessment convention in the source, not a measured or benchmarked quantity. The source gives the convention but no elicitation procedure for arriving at the three values.
The two readings of the same chart are what make it worth building. From a decision-making viewpoint it shows which uncertainties matter most — where more information would change the answer. From a project management viewpoint it identifies opportunities: features that could be improved by adding resources, so the team can allocate incremental effort to whatever adds the most value. The second reading is why the source claims decision tools also improve execution.
The same logic appears in the financial tradition under a different name; the comparison is drawn out in Risk and uncertainty in R&D financial analysis.
Tools 4 and 5 — the decision tree and expected value
By this point the team has produced three credible assessments: time and cost to completion, probability of success, and potential commercial value. Three of those quantities — R&D cost, probability of success and potential commercial value — go into a decision tree, which the source says both improves the project decision and measures the project's productivity.
The two definitions to get right
- Potential commercial value
- The expected net present value if successful — averaged over the various market uncertainties, but assuming technical and introduction success. It is a conditional value, not a risk-adjusted one.
- Expected value
- Potential commercial value weighted by the probability of success. This is the figure that can be compared with the expected R&D cost to completion.
The illustrative tree in the source carries the columns R&D decision, introduction success, commercialisation strategy and potential commercial value, with endpoint values of 140, 210 and 50 million dollars, and a no-R&D branch labelled with no R&D cost. Those endpoint figures are illustrative values from one anonymised example, not indicative magnitudes. The structural point is the one to take: the no-R&D branch is on the tree, so doing nothing is an evaluated alternative rather than an unexamined default.
What the tree is for
The expected-value figure is also the input to portfolio-level ranking — expected value divided by expected cost — which is where these per-project numbers stop being an evaluation and start being an allocation. That step, and the governance around it, belongs to Governing R&D decisions organisationally.
What the toolkit does not cover
R9 is unusually clear about its own boundaries, and the boundaries are the part most likely to be lost when the tools are re-transmitted.
- Basic research is excluded by name. The source states that none of the discussion is intended to apply to, or constrain, basic knowledge-building research. The toolkit is for purposeful, planned activity exploiting science and technology to reach a commercially attractive objective.
- The vocabulary is loose on purpose. R&D, applied research, new product development, new process development and technology development are used somewhat interchangeably; no distinctions between them are drawn.
- The whole apparatus assumes a goal. It is conditional on taking shareholder value as the ultimate goal of industrial R&D, which the source states as an assumption rather than defends.
- Simple to understand, non-trivial to apply. The source says so directly, and adds that making the tools a way of life requires sustained effort.
- No elicitation method is supplied. The tools consume probabilities and percentiles; how to obtain them from experts, and how to correct for the biases in doing so, is not covered.
What to carry forward
- Cost to completion, probability of success and potential value given success are the three inputs. Any project decision made without all three is being made on two.
- Keep potential commercial value conditional — value if successful — and apply the probability separately. Blending them early destroys the ability to argue about either.
- Build the influence diagrams as a group. They are the source of the variable list that everything downstream depends on.
- Assess low and high as the 10th and 90th percentiles, not as worst and best case, and be able to say what frequency you mean.
- Read the tornado twice: once for which uncertainties matter to the decision, once for where extra effort would buy the most value.
- Put the no-R&D branch on the tree. An unexamined default is not an evaluated alternative.
Frequently asked questions
Do I need all five tools, or can I just build the decision tree?
You can build a tree from nothing, and the source's warning is precisely about what happens when you do. The first three tools exist to produce the variable list, the hurdle list and the ranges that the tree consumes. A tree fed by unexamined estimates reproduces the opinion it started with, which is the garbage-in-garbage-out experience the source says discredited earlier quantitative attempts.
What is the difference between potential commercial value and expected value?
Potential commercial value is the expected net present value if the project succeeds — averaged over market uncertainty but assuming technical and introduction success. Expected value applies the probability of success to that figure. Keeping them apart lets you argue separately about whether the prize is big enough and whether you are likely to win it.
How do I get probabilities of technical success that anyone will believe?
The route the source gives is the technical influence diagram: identify every hurdle that could block success, then assign a probability of overcoming each rather than guessing one overall figure. Credibility comes from the hurdle list being visible and arguable. The source does not supply an elicitation procedure beyond that.
Should the low and high values be worst and best case?
No. The stated convention is the 10th percentile for low, the 90th for high, and the 50/50 value for the median. Worst and best case widen every bar in the tornado chart and change the ranking, which defeats the purpose of sorting variables by impact.
Does the toolkit apply to basic research?
The source excludes it explicitly: none of the discussion is intended to apply to, or constrain, basic knowledge-building research. It is written for purposeful work aimed at a commercially attractive objective, so applying it to exploratory science means applying it outside its stated scope.
Are these tools still current, given the article is from 1994?
The mechanics are technology-independent and travel without difficulty. What does not travel is the article's characterisation of prevailing practice at the time, and its account of what computing made feasible then. Treat framework and commentary differently.
References and source attribution
- R9 - improving R&D decisions and execution. Practitioner methods article in a journal for R&D and technology managers, 1994; 8 printed pages; a synthesis of a decision-analysis toolkit illustrated with anonymised client displays, with nine figures and a four-item reference list. Not empirical research.
- Eleven copyrighted journal articles on R&D project management, supplied as a reading set for a literature review and profiled for this library. Front matter, abstracts, framework sections, tables and figures were read; article bodies were not reproduced, and all content here is paraphrase.
- Figures reproduced in outline from R9: a strategy table for a medical diagnostic system; commercial and technical influence diagrams; a sensitivity tornado chart from a materials lightweighting case; and a decision tree for an R&D project. All values in them are illustrative client data.
- Externally cited within R9 and not independently verified: a 1993 benchmarking study by a quality-directors network, a decision-quality association and a consulting firm, reported as revealing widespread use of these tools among leading R&D organisations, with no percentages given.
- Supplied teaching source for this library (research methods and research process materials). Used here for page conventions and voice only; it does not treat R&D project management.
Suggested questions for Ask KEVOS
- Build me a strategy table for a product I describe, with the design decisions as columns.
- Turn my project's technical hurdles into a probability of technical success.
- Which variables should go into a tornado chart for my project, and how do I set the 10th and 90th percentiles?
- Draft a decision tree structure for a project with a go, a delay and a no-R&D option.
- What is the difference between potential commercial value and expected value, in my numbers?
- Which of the five tools would most improve a business case I already have?