The Central Question
Hamming opened Chapter 19 with a direct quote from the 1975 Summer Computer Simulation Conference:
> Since important decisions must rely on simulation, it is essential that its validity be tested, and that its advocates be able to describe the level of authentic representation which they achieved.
His formulation of the same question, sharper:
> Why should anyone believe the simulation is relevant?
Do not begin a simulation until you have given this question serious thought. Often there are practical pressures to start computing immediately — time, money, organizational momentum. Those pressures do not change the question.
Hamming's experience: when he raised validity questions about simulations, the answers invariably cited:
- How much manpower went into the work
- How large and fast the computer
- How important the problem
None of those answers address validity. They are social arguments, not technical ones.
The space-flight story: a simulation director claimed 99.44% reliability for launches. Hamming pressed: reliability of what? After being cornered, the director admitted his figure was reliability of the simulation, not the actual flight. He further admitted the NASA director who approved launches did not understand the distinction — and he had not corrected that misunderstanding.
The conflation of simulation reliability with system reliability is one of the most dangerous errors in engineering practice.
What Validation Actually Requires
Hamming assembled a checklist for evaluating simulation reliability. Each question must be answered before the simulation output earns decision weight:
1. Does the background theory support the assumed laws? Physics simulations rest on verified equations; economic simulations often do not.
2. Could a small but vital effect be missing? The pilot trainer example: excellent for known physics; dangerous if a new aircraft has interactions not yet incorporated.
3. Is the input data reliable? Uncertainty in inputs propagates through computation. Know where the data came from.
4. Is the simulation stable or unstable? Unstable direction fields amplify input errors; stable ones damp them.
5. What cross-checks against known past experience are available?
6. Can internal checks be produced? Conservation of mass, energy, angular momentum, or other invariants. Without redundancy, no error detection is possible.
7. Do the marks on paper get into the machine accurately? Programming errors persist for years in production code. An automated card-reader that translated reaction descriptions into differential equations — Hamming's solution for the atmospheric chemistry simulation — reduces transcription errors without relying on human vigilance.
Hamming's Law of Economics
Physics simulations rest on verified, mathematically expressed laws. Chemistry simulations require verified rate constants. Biology simulations require known population dynamics.
Economics has, in Hamming's view, one reliable law: you cannot consume what is not produced. Every other claimed economic law is either a mathematical tautology or is sometimes false.
The implication for simulation: when the background laws do not hold reliably, the entire foundation of the simulation is suspect. High precision in the computation does not compensate for unreliable laws in the model.
Hamming's response when asked to run ecological simulations: he asked for mathematically expressed rules for every interaction, with numerical constants, and for real data to compare test runs against. The requesters soon found someone more willing to run 'very questionable simulations which would give the results they wanted.'
Hamming's rule: keep your integrity. Do not allow yourself to be used for other people's propaganda. Refuse simulations that are structurally designed to confirm a conclusion.
Simpson's Paradox in Simulation Data
Hamming's Berkeley story: more men than women were accepted to graduate school, suggesting discrimination. No individual department showed discrimination. The explanation: women applied disproportionately to departments with low acceptance rates; men applied disproportionately to departments with high acceptance rates. The aggregate statistic was real, and misleading.
This is Simpson's Paradox: combining data can create apparent effects not present in any subgroup. Simulation outputs that aggregate over groups, regions, or time periods can exhibit the same pathology.
Simulations That Change What They Model
A special class of simulation failure: the subject alters behavior in response to the model.
Stock market example: if a widely known trading strategy reliably produces returns above market, participants will adopt it, and competition will eliminate the excess return. The model's success destroys the model's validity.
Medical example: the double-blind experiment exists because doctors who knew which patients received treatment and which received placebo found improvement where they expected it. The doctors' knowledge contaminated the observation.
Simulation for high-stakes decisions — economic policy, military planning, public health — faces this problem whenever the simulation's outputs are made public or guide behavior.
Sampling & Accuracy
The Nyquist theorem sets a fundamental constraint on digital simulation of continuous systems: you need at least two samples per period of the highest frequency present in the signal.
In practice, Hamming noted, single-sided sampling and aliasing push the requirement to seven to ten samples per period of the highest frequency of interest.
Accuracy tradeoff: analog computers (in Hamming's era) achieved at most one part in 10,000 accuracy per component. Digital computers can carry arbitrary precision. But digital accuracy requires adequate sampling — and for signals with sharp high-frequency content, that sampling rate drives computation cost.
Analog advantage: speed and physical integration. You can wire a physical component directly into the analog circuit and measure its behavior without finding a mathematical description of it. For the early guided missile designs, this allowed rapid feel for the system before a full mathematical model existed.
The analog machines used for early missile work derived from M9 gun director parts. Hamming built a differential analyzer from such equipment. Solution time: about 30 minutes per trajectory. That pace forced deep thinking between runs — a constraint Hamming came to regard as a structural advantage.
Feel vs Accuracy: The Mars Lander Problem
Hamming posed a sharp design question: for a Mars lander flight trainer, should the simulator optimize for positional accuracy or for the feel of the vehicle?
Classical numerical analysis optimizes position trajectory using local polynomial approximations. The path is accurate, but the acceleration changes discontinuously at each polynomial interval — an unnatural 'feel'.
Frequency-domain methods optimize the frequency response of the simulator. Position may be less accurate, but the pilot perceives the vehicle's dynamic response correctly. The pilot's muscle memory forms on the right transfer function.
Hamming's judgment: a pilot trainer should optimize for feel. During actual flight, the pilot can compensate for position errors via feedback control. If the feel is wrong, the pilot has been trained for a vehicle that does not exist.
The deeper principle: the choice of numerical method encodes a claim about what matters in the simulation. That claim is a domain judgment, not a mathematical one.