Decision Quality in R&D: Six Dimensions
In R&D the gap between a decision and its result runs to years, and the result depends on things nobody controlled. So judge the decision on its own terms: six dimensions, all of them assessable on the day you decide.
A decision is not a result
Every project decision is made with the information available on the day. The result arrives later, and it arrives contaminated: by a competitor's launch, a regulatory change, a departure, a market that moved. Judging the decision by the result therefore judges the decider on things they did not control and could not have known.
R&D makes the problem acute rather than merely philosophical. The lag between decision and outcome runs to years; the variance in outcomes is high by design, because the work is uncertain; and the projects that fail most visibly are often the ones it was most correct to attempt. If you can only evaluate decisions after their results, in R&D you can barely evaluate them at all.
The source does supply a closely related observation of its own, and it is worth having in hand because it shows what outcome-judging does to R&D specifically. Because low-probability projects are usually several years from commercialisation, a head of R&D running a portfolio loaded with high-probability projects will appear to be successful for the next several years — but the successor may be in for trouble. Current performance is a lagging signal of portfolio health, and a misleading one.
The four strategic questions the decisions have to serve
Before assessing a single decision, it is worth being clear about what quality strategic R&D management is trying to answer. The source names four questions, and notes that all four are resource-allocation questions.
The four questions
Do we have the right total R&D budget?
The size question. Note that it is posed as answerable, not as a matter of custom or of last year plus a percentage.
Are we allocating it to the right business and technology areas?
The distribution question. It asks about areas, not projects — a level above project selection.
Do we have the right balance?
Balance of risk and return; of long-term against short-term R&D; of research against development; of innovation against incremental improvement. Four distinct trade-offs, not one.
Are we working on the right projects and programmes, with the right levels of effort?
The selection question, and note the second half — effort level is part of the decision, not a consequence of it.
Questions 2 and 3 are answered at portfolio level and need portfolio-level instruments; those are set out in R&D portfolio displays and strategy tables. The six dimensions below apply to any decision at any of the four levels.
The six dimensions
The source presents six dimensions along which to measure and improve the quality of R&D decisions, stated as diagnostic questions. They are given in a fixed order, running from how the problem was set up to whether anyone will act on the answer.
THE SIX DIMENSIONS OF R&D DECISION QUALITY
| Dimension | The diagnostic question | What a failure looks like |
|---|---|---|
| 1. Frame | Is the project appropriately framed — strategic fit, a clear definition, and goals mutually agreed by all functions? | Two functions describing the same project differently, and no one having noticed |
| 2. Alternatives | Does a range of creative and feasible alternatives support pursuit of the project goals? | A single option presented for approval, with the real choice being yes or no |
| 3. Information | Do you have, or can you get, reliable information — including the appropriate range of uncertainty — on customers, markets, competitors and technologies? | Point estimates with no ranges, or ranges nobody can explain the basis of |
| 4. Values | Are the organisation's values clear — time preference such as cost of capital, risk preference, and non-financial objectives? | An argument about the discount rate happening inside a project decision, years too late |
| 5. Logic | Can you logically combine the information to reliably evaluate project results, including time, cost, probability and value? | A model whose steps cannot be reproduced, or whose conclusion does not follow from its inputs |
| 6. Commitment | Can you do the preceding five in a way that generates organisational credibility, acceptance of the results, and commitment to action? | A rigorous evaluation, filed, with the decision made informally elsewhere |
Dimensions and diagnostic questions are close paraphrase of the source. The failure column is our characterisation, offered to make the dimensions concrete; it is not in the source.
Note what the six are all doing. Each names something you could establish, argue about and fix before the decision is taken. None refers to whether the project subsequently worked. That is what makes it a diagnostic of decisions rather than an appraisal of outcomes.
Why the sixth dimension is not like the other five
Frame, alternatives, information, values and logic are analytical criteria. You can satisfy all five in a room by yourself, given enough time and data. Commitment cannot be satisfied that way, because it is a criterion about the organisation rather than about the analysis.
This is why the source pushes the evaluation work into cross-functional teams rather than into an analytical function — the organisational consequences of that are worked through in Governing R&D decisions organisationally. It also explains a barrier the source names elsewhere: management rarely has credible assessments on which to base termination decisions. The missing ingredient is not usually the analysis — it is credibility, which is dimension six failing upstream of every stop-or-continue conversation. The consequences of that are followed through in Making better project termination decisions.
Improving dimensions 1 to 5
- Buy better information, or buy ranges rather than points.
- Generate more alternatives before narrowing.
- Settle values — discount rate, risk preference, non-financial objectives — at organisational level, once.
- Use an explicit model so the logic is inspectable.
- Largely doable by the analyst or the project team.
Improving dimension 6
- Involve the functions that will have to act, while the analysis is being built rather than after.
- Make the assumptions visible and contestable rather than defensible.
- Agree the criteria before the numbers exist, so the decision rule is not chosen to fit the result.
- Accept that a slightly rougher analysis everyone owns can outperform a better one nobody does.
- Not doable by the analyst alone, at any level of effort.
Running the diagnostic on a live decision
The six work as a structured conversation held before a funding decision, not as a form. Ask them in order; the earlier ones constrain the later ones, since better information cannot rescue a project that was framed wrongly and better logic cannot rescue a choice between one alternative and nothing.
Six questions to put to a project decision before it is taken
- Frame — can every function state the same project definition and the same goals, in their own words?
- Alternatives — how many genuinely different, feasible options are on the table, and who generated them?
- Information — for each key uncertainty, do we have a range as well as a number, and can we say where the range came from?
- Values — is the time preference, the risk preference and the set of non-financial objectives already agreed, or are we about to argue about them here?
- Logic — can someone outside the team follow how time, cost, probability and value were combined into the conclusion?
- Commitment — will the people who have to act on this accept it, and how do we know?
Where the dimensions bite in practice
Three of the source's own gating rules are the six dimensions in operational form. Each refuses a decision on a decision-quality ground rather than on a forecast.
Gating rules that enforce decision quality
The shared-vision rule is where dimensions 1 and 6 meet: the strategy table exists partly to make competing visions visible, and the mechanics of building one are in R&D project evaluation tools. What makes the rule a gating criterion rather than good advice is that the source treats absence of shared vision as grounds to withhold funding.
What the diagnostic does not give you
- No weighting. Nothing tells you whether a frame failure is worse than an information failure. In practice frame and alternatives constrain everything downstream, but the source does not rank them.
- No threshold. There is no stated level at which a decision counts as high quality, so the diagnostic identifies weaknesses rather than certifying decisions.
- A stated goal assumption. The whole apparatus is conditional on taking shareholder value as the ultimate goal of industrial R&D. The source states this as an assumption. Where an organisation's R&D serves a different ultimate objective, dimension 4 is doing much more work than it appears to.
- Basic research is out of scope. The source excludes basic, knowledge-building research from the discussion by name.
- Not a quality-management framework. The source positions decision-quality tools as complementary to, and distinct from, the total quality management toolkit — while borrowing its argument that clear definition and measurement are essential to improving any process.
One further limitation is structural rather than stated. A diagnostic of decisions needs a feedback loop, or it becomes an unexamined ritual — and R9 does not supply one. The obvious candidate is a review held after the fact that asks how the decision was made rather than how the project turned out; that is the subject of Post-project reviews in R&D.
What to carry forward
- Assess the decision on the day it is made, on the six dimensions. The result will arrive years later carrying influences nobody chose.
- Frame and alternatives come first for a reason: better information and better logic cannot repair a badly framed choice or a single option presented as a decision.
- Insist on ranges, not point estimates, and on knowing where a range came from. That is dimension 3 in one sentence.
- Settle time preference, risk preference and non-financial objectives at organisational level, before they distort an individual project case.
- Treat commitment as a quality criterion, not as change management. An evaluation nobody accepts has failed, however good it is.
- Introduce the six as a structured practitioner framework with no published study behind it — because that is what they are.
Frequently asked questions
Isn't a decision that leads to a failed project a bad decision?
Not necessarily, and in R&D usually not. A portfolio of only projects that succeed is a portfolio that was not attempting anything uncertain. The six dimensions exist so you can assess whether a decision was well made at the time, independently of an outcome that will be shaped by events nobody decided.
Can I turn the six dimensions into a scored assessment?
You can, but you would be adding something the source does not contain. It supplies six diagnostic questions with no scale, no weighting and no threshold. If you build a scoring instrument, say internally that the scale is yours, so nobody mistakes a local convention for an established standard.
Which dimension fails most often?
The source does not rank them or report frequencies. What it does say is that failure to build a shared product vision from the start is one of the most common and most destructive failure modes, which is a frame failure — and that management rarely has credible assessments on which to base termination decisions, which is an information and commitment failure.
Why is commitment treated as part of decision quality rather than as implementation?
Because on this account an evaluation that nobody accepts has not produced a decision at all, however rigorous it is. Commitment asks whether the first five dimensions were achieved in a way that generates credibility, acceptance and commitment to action. It is a criterion about the process, and it cannot be satisfied by improving the analysis.
How do the six dimensions relate to the four strategic questions?
The four strategic questions are what R&D management is trying to answer: total budget, allocation across areas, balance, and the right projects at the right effort levels. The six dimensions are how you assess the quality of any decision made in answering them. One is the agenda, the other is the standard.
Does an article from 1994 still describe how R&D decisions are made?
The six dimensions are technology-independent and do not depend on the state of computing or markets in 1994. What is period-bound is the article's characterisation of how common quantitative assessment was at the time. Keep the framework, date the commentary.
References and source attribution
- R9 - improving R&D decisions and execution. Practitioner methods article in a journal for R&D and technology managers, 1994; 8 printed pages; a synthesis of a decision-analysis toolkit illustrated with anonymised client displays. Not empirical research, and no study is reported for the six dimensions.
- Eleven copyrighted journal articles on R&D project management, supplied as a reading set for a literature review and profiled for this library. Front matter, abstracts, framework sections, tables and figures were read; article bodies were not reproduced, and all content here is paraphrase.
- Externally cited within R9 and not independently verified: a 1992 conference paper reporting that one chemical company's estimated research value nearly tripled in two years against the average of the previous ten, after new and improved product and process concepts were established as the main output of research.
- Supplied teaching source for this library (research methods and research process materials). Used here for page conventions and voice only; it does not treat R&D project management.
Suggested questions for Ask KEVOS
- Run the six decision-quality dimensions over a funding decision I am about to make.
- How do I tell whether a project is genuinely framed, or just described?
- My team has one option and a yes-or-no choice. How do I generate real alternatives?
- What would a credible assessment for a termination decision have to contain?
- How should I explain to a governance board that a failed project can follow a good decision?
- Draft agenda questions for a pre-funding decision-quality review.