Learning Probability Models: Parameters and Structure

Separate model form from model parameters

A model may specify the variables and relationship type while leaving numerical parameters unknown.

For a categorical conditional distribution, parameters may be probabilities for each outcome given parent states. For a Gaussian model, parameters can include means, variances and regression coefficients. Learning estimates these quantities from observations. The estimation method should match the data-generation process and the way the model will be used.

Before fitting, define sampling unit, measurement process and missingness. Historical databases often contain selection effects: only accepted orders, surviving machines or approved customers may be observed. A technically correct estimator applied to biased data can still produce a poor decision model.

Maximum-likelihood estimation

Maximum likelihood chooses parameter values that make the observed data most probable under the assumed model.

For many standard distributions, maximum-likelihood estimates are intuitive. A Bernoulli success probability is estimated by the observed success fraction; a Gaussian mean by the sample average under the basic independent model. The method is attractive because it is direct and often computationally convenient.

The limitation is that MLE returns a point estimate and can overfit when data are sparse relative to model complexity. Regularisation, simpler models, cross-validation or Bayesian methods can reduce this risk.

Bayesian parameter learning

Bayesian learning starts with a prior distribution over parameters and updates it with observed data.

The posterior represents uncertainty about the parameter rather than collapsing immediately to one value. With little data, prior information can stabilise estimates; with abundant informative data, the likelihood often dominates. Priors should be documented and sensitivity-tested, especially when they materially affect decisions.

For some distribution families, conjugate priors produce analytical updates. In more complex models, numerical inference is required. The practical benefit is decision-aware uncertainty: predictions can average over parameter uncertainty rather than pretending the estimated parameters are known exactly.

Non-parametric approaches

Parametric models assume a specific distributional form. Non-parametric methods allow more flexibility, often at the cost of more data or computation.

Empirical distributions, kernel methods and flexible density estimators can represent shapes that a single Gaussian or other fixed family cannot. This is useful when data show skew, multiple modes or irregular tails. However, flexible models can extrapolate poorly beyond observed regions and can overfit small datasets.

Choose flexibility based on decision need. If only a threshold probability matters, an elaborate density model may be unnecessary. If tail shape controls a high-consequence decision, a simplistic distribution can be dangerous.

Missing data and latent variables

Real datasets frequently omit variables or contain hidden states that are not directly observed.

The effect of missing data depends on why values are missing. Randomly missing sensor records differ from measurements omitted because a process was already known to be failing. Treating missing entries as if they were a random subset can bias the model. Document the missingness mechanism and consider whether the observation process itself needs to be modelled.

Iterative methods can estimate latent or missing quantities and parameters together. A common pattern alternates between estimating the distribution of hidden variables using current parameters and then updating parameters using those inferred quantities. Convergence to a local solution rather than a global best fit is possible, so initialisation and multiple runs matter.

Learning network structure

When dependency structure is not fixed, the model can search over candidate graphs.

A structure-learning method usually balances goodness of fit with a complexity penalty. Adding edges can always make a model more flexible, so scoring must discourage unnecessary complexity. Search is challenging because the number of possible graphs grows rapidly and directed cycles are not allowed in a Bayesian network.

Different graph structures can represent the same set of conditional-independence relationships, creating equivalence classes. Therefore a unique arrow direction may not be identifiable from observational data alone. Domain constraints and intervention evidence are important when the direction has causal meaning.

Worked example: demand model update

Imagine a generic firm forecasting whether a customer enquiry will convert, with features such as segment, lead time and quoted price.

A maximum-likelihood model can estimate conversion probabilities from historical records. If some segments have very few observations, the raw estimates may be extreme. A Bayesian model can partially pool evidence through priors or hierarchical structure, preserving uncertainty for sparse segments. Missing follow-up outcomes must be investigated: if lost enquiries are less likely to be recorded, the training data are systematically biased.

If the dependency structure is uncertain, structure learning can suggest relationships, but management should not conclude that changing a feature will cause conversion to change merely because the learned graph points that way. Intervention decisions require stronger causal justification.

Governance for learned probability models

Learning is not complete when an optimisation routine finishes.

Application checklist

• Define the target population and observation process before fitting.
• Use maximum likelihood where a point estimate is adequate and data support the model.
• Represent parameter uncertainty when sparse evidence can affect decisions.
• Select distribution flexibility according to the features of uncertainty that matter.
• Analyse why data are missing rather than applying blind imputation.
• Penalise unnecessary structure complexity and use domain constraints.
• Validate calibration and predictive performance on unseen data.
• Monitor model drift and re-estimate or retire models when context changes.