What is model calibration?

Model calibration is the process of tuning the values of a model's parameters so that its results line up with real measurements. It is used when some inputs, such as a conversion rate or a delay time, cannot be measured directly. It matters because a model with poorly set parameters can look convincing while giving wrong forecasts.

How it works

Calibration starts with data on outcomes you already know and a list of parameters that are uncertain. You choose a measure of the gap between model and data, for example the sum of squared differences. You then change the parameters, by hand or with a search method, until the gap is small. Parameters should stay within plausible limits, because a good fit with absurd values is a warning sign.

Fit alone does not prove a model is right. Many different parameter sets can fit the same data, and the model may then fail on new conditions. For that reason, keep some data aside that is not used in calibration and use it for model validation. Calibration is closely linked to parameter sweeps and simulation optimization, which can automate the search, and to sensitivity analysis, which shows which parameters matter most.

Example

Say monthly sales for the last 6 months were 100, 130, 170, 215, 260, 300 units, and the model has an unknown adoption rate. With a rate of 0.2 the model gives 100, 120, 144, 173, 207, 249, which is too low by 50 units at month 6. With a rate of 0.28 it gives 100, 128, 164, 210, 269, 344, too high. A rate near 0.25 stays within about 20 units each month. This is an illustration with round numbers.

How to model it in Machinations

Put the real data in a Register and the uncertain parameter in another Register. Build the model with a Pool and chart it with the data line next to it. Use a Lever to move the parameter and Play repeatedly until the lines match. For a systematic search, run many Plays at once with a Forecast and compare the range to the data.