Policy Search, Policy Gradients and Actor-Critic Methods

When policy search is useful

Direct policy optimisation is attractive when actions are continuous, the policy can be represented compactly or value-function maximisation is difficult.

Instead of solving for the optimal value of every state and deriving a greedy action, policy search defines a family πθ(a|s) controlled by parameters θ. The objective is expected cumulative return under that policy. Simulation or observed trajectories estimate how changes to θ affect performance.

The policy class creates both efficiency and limitation. A compact rule may be easy to deploy and interpret, but the optimal policy might not be representable within that class. Choose a policy architecture that reflects operational structure while retaining enough flexibility.

Stochastic policies and the policy-gradient idea

A differentiable stochastic policy assigns action probabilities or densities that depend on parameters.

A central result allows the gradient to be estimated from trajectories using the gradient of the log policy multiplied by observed return or an advantage signal. The estimator does not require differentiating the environment transition model, which is useful when the environment is simulated or unknown.

The estimate is unbiased under suitable assumptions but can have high variance. A few unusually good or bad trajectories can dominate learning.

Variance reduction with baselines and advantage

Subtracting a baseline that does not depend on the selected action can reduce variance without changing the expected gradient.

A state-value estimate is a common baseline. The difference between action return and state baseline is an advantage: how much better or worse the chosen action performed compared with what was expected in that state. This centres the learning signal and can make optimisation substantially more stable.

Gradient estimation choices

Returns can be estimated using full episodes, bootstrapped value estimates or combinations.

Full Monte Carlo returns avoid value-function bias but can have high variance and require waiting until the outcome is known. Bootstrapped methods update earlier using a learned critic but introduce approximation error. Multi-step estimators trade these properties. The appropriate choice depends on horizon, noise, data availability and tolerance for bias.

Normalise or clip signals cautiously. Numerical stabilisation can improve training, but hidden transformations can also change the effective objective. Keep an audit trail of reward scaling, discounting, constraints and termination logic.

Policy optimisation

A gradient direction still requires an optimisation method and step-size discipline.

A step that is too small wastes data; a step that is too large can collapse a useful policy. Adaptive optimisation can help, but there is no substitute for monitoring actual policy performance and distribution shift. Some approaches constrain the size of each policy update so that the new policy remains close to the behaviour that generated the data.

For business systems, operational constraints should often be encoded explicitly rather than learned through huge penalties. Examples include inventory non-negativity, legal limits, approved supplier lists or hard safety bounds. This reduces the chance that exploration discovers an economically attractive but prohibited behaviour.

Actor-critic architecture

Actor-critic methods combine a policy model with a value estimator.

The critic can learn faster than waiting for complete episodes, while the actor can represent continuous or stochastic policies naturally. Their interaction can also be unstable: a poor critic teaches a poor actor, whose changed behaviour then changes the data distribution seen by the critic. Learning rates and update schedules therefore matter.

Worked example: dynamic prioritisation

A generic operations team must decide which job to process next from a changing queue.

A parameterised policy might score each job using urgency, processing time, customer impact and downstream blocking. Rather than hand-setting weights, simulation can estimate which weights produce better long-run service and cost. A stochastic policy explores near-tied alternatives during training, while the critic estimates expected future backlog and lateness from each state.

Before deployment, compare the learned policy with established dispatch rules under peak load, missing data and unusual job mixes. Preserve a deterministic fallback and impose hard constraints for jobs that must legally or contractually take precedence.

Failure modes and controls

Policy optimisation can improve the metric it is given while damaging the purpose behind the metric.

Operational readiness for a learned policy

A technically correct method still needs an auditable operating translation.

A learned policy should enter operations only after its training objective has been translated into business-facing acceptance criteria. Define the minimum performance improvement, maximum tolerated downside, prohibited action set, computation limit and override process. Compare multiple independently trained policies because stochastic optimisation can converge to different behaviours even when average training reward looks similar. Inspect representative trajectories rather than relying only on aggregate reward. If two policies achieve similar return, prefer the one with simpler behaviour, stronger constraint compliance and easier diagnosis. Preserve the training configuration, random seeds, reward definition and validation evidence so a later performance change can be traced to data drift, model changes or operating conditions rather than guessed after the event.

Application checklist

• Choose a policy class that fits operational structure and action type.
• Define the reward so it reflects the real objective and constraints.
• Use baselines or advantage estimates to reduce gradient variance.
• Select return estimation and bootstrapping according to horizon and noise.
• Control policy step size and monitor performance after each update.
• Keep hard operational constraints outside or alongside the learned objective.
• Validate actor and critic stability across multiple runs.
• Deploy with monitoring, override logic and a safe fallback policy.