Exploration & Exploitation

Everything until now assumed you knew the odds. Part III removes that comfort. And the moment you don't know the odds, one tension dominates every choice: exploit, or explore?

Parts I and II handed you a model — transition probabilities, rewards, all given. Real projects rarely do. When you're uncertain how good your options actually are, every decision does double duty: it earns a result and it teaches you something. Exploitation means taking the option you currently believe is best; exploration means trying an uncertain option to learn more, accepting a possibly worse result now for better decisions later. Getting this balance right is the whole of decision-making under an unknown model.

1The bandit: the tension in its purest form

The clean abstraction is the multi-armed bandit: several options, each with an unknown payoff you can only learn by trying it. Pull an arm, get a reward, update your belief. Pour every pull into the arm that looks best so far and you might never discover that a barely-tried option was better all along. Spread your pulls to learn, and you waste them on options you already suspect are poor. The cost of getting it wrong is measured as regret — the gap between what you earned and what the best possible strategy would have earned.

Figure 1. The proven option pays a confident, moderate amount. The untried one looks worse on average — but its uncertainty leaves a real chance it's much better. Optimism in the face of uncertainty says: judge the untried option by its plausible upside, and it may be worth a trial.

2Strategies for striking the balance

ε-greedy exploits the best-known option most of the time but explores at random a small fraction of the time. Crude, but effective and trivial to run.
Optimism under uncertainty (the upper-confidence-bound idea) judges each option not by its average so far but by the top of its plausible range, and picks the highest. Under-tried options carry wide uncertainty, so they get the benefit of the doubt — until enough trials either confirm or deflate them.
Posterior sampling (Thompson sampling) is elegantly simple: draw one plausible value for each option from your current belief, then act greedily on that draw. Options you're unsure about get sampled optimistically often enough to be tried, and the exploration self-tunes as beliefs sharpen.

Key idea

When you don't know the odds, the average result so far isn't enough to decide by — you must weigh what an option could be, given how little you've tested it. Give promising unknowns the benefit of the doubt, in proportion to your uncertainty about them.

3This is the pilot-versus-commit decision

Strip away the mathematics and this is the most common strategic question in project delivery. Do you commit to the incumbent supplier, method, or contractor you trust — or run a pilot with the promising newcomer who might be substantially better? Every trial costs time and money and might disappoint; every trial also buys information you can't get any other way. The bandit framing gives that instinct a spine: an untried option deserves a trial when its plausible upside, weighted by how little you know, beats sticking with the proven choice.

What it means in practice

Treat "should we try the new approach?" as an explicit explore-exploit call, not a matter of temperament. Don't judge an unproven supplier, tool, or method solely on a thin track record — weigh its plausible upside against your uncertainty, and size the trial accordingly. Run cheap, reversible experiments early while there's time for the information to pay off, and commit hard once the uncertainty has genuinely collapsed. Persisting with the familiar option out of habit is a decision too — usually an expensive one.

Handbook application: from concept to controlled practice

Purpose. This expanded section turns the original page into a practical handbook. It preserves the supplied material and adds a repeatable way to apply, check and review Exploration & Exploitation. It does not replace a contract, legislation, a controlled standard, competent engineering judgement or specialist advice.

The operating aim is to convert the subject into a governed decision, owned work, usable evidence and a reviewable outcome. Read the original explanation first, then use the workflow and checks below to convert knowledge into evidence.

Use Exploration & Exploitation as a decision instrument rather than an administrative form. The subject terms—exploration, exploitation, bandit, balancing, acting—need an explicit connection to the project objective, business value and stakeholder commitments. Before completing the artefact, write one sentence stating who will use it, what decision it supports and when that decision is required.

Apply a disciplined information model. Separate facts supported by evidence, forecasts derived from a method, assumptions awaiting validation, constraints that limit choice, risks that may occur, issues that already exist and actions assigned to people. Each material entry should have an owner, date, status and next review point. Where probability or impact scores are used, define the scale so different reviewers interpret it consistently.

A baseline is useful only when changes are visible. Give the artefact an identifier, version, approval state and effective date. Define which changes require reapproval, how superseded versions are retained and where supporting evidence is stored. During reviews, focus on exceptions, decisions and trends rather than reading every field aloud. Record the decision and rationale, not merely that a meeting occurred.

Close the loop beyond delivery. Confirm acceptance criteria, unresolved items, transferred responsibilities and operational ownership. Where benefits are expected, identify the outcome measure, baseline, target, observation period and owner who remains accountable after the project team disbands. Lessons should describe the condition, consequence and reusable action; a generic statement such as “communicate better” cannot improve the next project.

Step-by-step operating method

Clarify the decision. Name the outcome, sponsor, affected stakeholders and decision that this work must enable.
Set boundaries. Record scope, assumptions, constraints, dependencies, tolerances and escalation conditions.
Plan the evidence. Define deliverables, measures, owners, due dates and acceptance criteria before execution.
Control delivery. Compare actual performance with the baseline, assess changes and manage risks and issues explicitly.
Close the loop. Confirm acceptance, transfer ownership, capture lessons and track benefits beyond handover.

Completion and governance protocol

Start with a short drafting workshop involving the accountable owner and the people who hold the evidence. Complete high-consequence fields first: objective, scope, owner, baseline, acceptance, dependencies and escalation. Mark unknowns as assumptions or actions rather than hiding them behind vague prose. Circulate a review draft, resolve conflicting interpretations, baseline the approved version and place the next review date in an owned schedule.

Information type	Minimum useful content	Review test
Outcome	Observable change and intended recipient	Not merely a deliverable or activity
Measure	Definition, baseline, target, frequency and source	Two reviewers would calculate it the same way
Ownership	One accountable role plus contributors and approver	Authority matches responsibility
Uncertainty	Assumption, risk or issue with response and trigger	Status reflects current reality
Control	Version, approval, review date and change rule	Current baseline is identifiable

Common failure modes and recovery actions

1. Watch for

Producing a document with no named decision or accountable owner.

Recovery: Return to the governing definition or requirement and restate the decision in one sentence.

2. Watch for

Mixing risks, current issues, assumptions and actions in one unstructured list.

Recovery: Separate evidence from assumption, assign an owner and set a date for validation.

3. Watch for

Measuring activity or output while leaving the intended outcome undefined.

Recovery: Run a small counterexample, boundary test, pilot or independent check before proceeding.

4. Watch for

Accepting changes without evaluating effects on value, scope, schedule, cost and risk.

Recovery: Record the consequence, decision and rationale, then update the controlled baseline.

5. Watch for

Closing the project at delivery even though benefit ownership has not transferred.

Recovery: Escalate when the issue affects safety, compliance, acceptance, material value or an agreed tolerance.

Review checklist

Which decision or commitment does this artefact support?
Who owns each action, risk, acceptance and post-project benefit?
What is the baseline and what variance triggers escalation?
Where is the evidence that the result was accepted and transferred?
Are mandatory requirements distinguished from recommendations and illustrative values?
Are sources, assumptions, units, dates and versions recorded closely enough to reproduce the decision?
Have safety, legal, ethical, stakeholder and operational consequences been considered at the appropriate level?
Is there a named owner and a trigger for review, escalation, change or retirement?

Questions for deeper application

What is the most important distinction a practitioner must preserve when applying Exploration & Exploitation?