What Fitting a Model Really Means
A simulation model makes a mathematical claim: the outputs of the real system lie on (or near) a specific surface M in the space of observations.
Let the real system produce observations y₁, y₂, ..., yₙ. The model predicts values ŷ₁, ŷ₂, ..., ŷₙ.
Residuals as distances: rᵢ = yᵢ - ŷᵢ. Each residual measures the distance between an observation and its corresponding model prediction. In n-dimensional observation space, the residuals form a vector r = y - ŷ.
Least-squares fitting: choose model parameters to minimize ||r||² = Σrᵢ². Geometrically, find the point ŷ on the model surface M closest to the observation vector y in Euclidean distance.
When Residuals Mislead
Small ||r||² does not guarantee a valid model. Two systematic failure modes:
1. Systematic bias: residuals rᵢ are small but all positive (or all negative). The model consistently under- or over-predicts. Geometrically: ŷ lies on a parallel offset surface to the true data manifold — close in distance, wrong in structure.
2. Wrong manifold: residuals are small because the model has enough free parameters to fit the training data exactly (overfitting). The model surface threads through the data points, but curves wildly between them. Predictions on new data are poor.
Detecting Systematic Bias
A model with zero mean residual may still have systematic bias that varies with an input variable.
Example: a weather simulation that underestimates temperature by 2°C in summer and overestimates by 2°C in winter has mean residual ≈ 0 across a full year, but a clear seasonal bias.
Residual diagnostic: plot rᵢ against each input variable. A flat pattern (no trend) suggests no systematic bias from that variable. A trend pattern reveals a missing dimension in the model.
Hamming's validation question — 'Could a small but vital effect be missing?' — translates geometrically: does the residual vector have a component in a direction not spanned by the model's parameter space?
Systematic Offset vs Random Noise
The Hawthorne effect: subjects in a study change their behavior because they know they are being observed, not because of the experimental treatment.
Geometric Interpretation
Let the true data manifold M live in a space spanned by the variables (x₁, x₂, ..., xₖ, observation_context).
The model ignores observation_context. It fits a surface to observations in (x₁, ..., xₖ) alone.
When observation_context = 'being studied,' the actual data points shift along the observation_context axis. The model's surface — fixed in (x₁, ..., xₖ) space — now fits displaced data. The residuals appear small (the surface still fits well within the study context), but predictions in the unobserved context are systematically wrong.
The geometry: the model surface is close to the study-context data manifold, but far from the reality manifold. The distance between them: the Hawthorne offset along the observation_context axis.
Hamming's double-blind requirement: prevent observation_context from becoming correlated with treatment. This keeps the reality manifold and the study-context manifold coincident — eliminates the geometric offset.
Other Hidden-Dimension Effects
Any variable that affects the system but is excluded from the model creates the same geometric structure:
- Seasonal effects omitted from economic models
- Operator behavior excluded from manufacturing simulations
- Software version state absent from performance models
The model fits a lower-dimensional surface to data that lives on a higher-dimensional manifold. Residuals will be small in directions the model measures, large in the unmeasured directions.
Validation as Geometric Alignment
Hamming's validation checklist, reframed as geometry:
Does the background theory support the assumed laws? Do the dimensions of the model's parameter space span the true data manifold? If key variables are missing (excluded dimensions), the model surface cannot be aligned with reality.
Are internal checks available? Conservation laws are geometric constraints: the data must lie on a specific submanifold defined by mass conservation, energy conservation, etc. If the simulation violates these, its trajectory has left the valid submanifold.
Cross-checks against known past experience: the model surface must pass through historical validation points — not just fit training data, but generalize to out-of-sample observations.
Is the simulation stable? A stable simulation stays near the true solution manifold despite small perturbations. An unstable simulation leaves the neighborhood of the manifold and cannot be called a valid model.
When Prediction Becomes Projection
Hamming endorsed the scenario method for domains where prediction is impossible: instead of claiming 'the system will do X,' present a set of possible trajectories under different assumption sets.
Geometric Interpretation
The model surface M(θ) depends on parameters θ (assumptions about laws, constants, boundary conditions). Different assumption sets θ₁, θ₂, ..., θₖ define different surfaces M(θ₁), ..., M(θₖ).
The scenario envelope is the union of these surfaces: the region of output space that any of the scenario models could produce.
A single prediction claims: the true outcome lies near M(θ) for the best estimate θ. The scenario method claims: the true outcome lies somewhere inside the envelope.
When the Envelope Is Useful
If the envelope is narrow — all scenarios agree on the output despite different assumptions — confidence in the prediction is high. If the envelope is wide — different assumptions produce very different outputs — the model is highly sensitive to assumptions. That sensitivity is the output, not a failure mode.
Hamming's claim about his own predictions: he was giving scenarios, not point predictions. The future he described was 'what is likely to happen, in my opinion,' not a precise forecast.
Overlap with Reality
A scenario model is validated when reality falls inside the envelope. This is a weaker test than point prediction but more honest about what the model can claim.
Putting It Together: Valid Models & Their Geometry
The geometry of valid simulation comes down to three alignments:
1. Parameter space covers the true manifold: the model's dimensions include all variables that drive the system. Hidden-dimension gaps produce systematic offsets.
2. Stability keeps the trajectory near the true manifold: a convergent direction field means errors shrink. A divergent field means the simulation leaves the valid region.
3. Residuals are small AND unstructured: random, uncorrelated residuals suggest the model captures the true manifold. Structured residuals (trends, patterns) signal a missing dimension.
Hamming's 'Why should anyone believe the simulation?' translates geometrically: how close is the model surface to the reality manifold, in how many dimensions, with how much stability, validated on how many out-of-sample points?