Cultures of Model Use

by Ekaterina Svetlova

The Economist (14 November 2020) highlighted the, sometimes tetchy, debate between epidemiologists and economists over the role of models in the policy response to the covid-19 crisis. It was largely an academic debate about familiar themes such as models’ assumptions, uncertainties, and the meaning of model-based scenarios in both disciplines.

However, the rambling discussion missed an important point. To understand how models influence political decision-making, one must remember that decisions are not made by scientists. When models enter into political discourse they are interpreted, contested, misunderstood, misrepresented and sometimes just ignored by practitioners. So any debate about the role of models in policy advice should recognise how politicians, central bankers, regulators and investors use models in their decision-making. In times of crisis, such insights become particularly pertinent.

My research on financial markets and a number of projects funded by Rebuilding Macroeconomics investigate cultures of model use in deep empirical detail. They suggest that models do not ultimately dictate decisions and actions to practitioners. Preliminary findings from the RM project “What do policymakers want from macroeconomics?” suggest that practitioners do not take model output at face value but treat it as an anchor, or a reference point for decisions, that is adjusted by judgment calls and stories. Another RM project shows that, in setting monetary policy at the Bank of England, statistical models play a role of heuristic tools that are combined with carefully constructed narratives about what is going on in the economy.

Models fulfil a number of pragmatic functions that facilitate – but do not determine decisions. They serve as communication devices, opinion proclaimers or legitimization tools.

Models might help to initiate or sustain the communication between diverse groups of professionals. For example, DCF models’ projections are not treated as forecasts in portfolio management but rather allow for a meaningful discussion of scenarios: what will happen to the company’s profit if the oil price goes up, or if interest rates go down? Similarly, the aforementioned research on monetary policy highlights that models are “devices to be played with” and are used to elaborate possible futures in the MPC meetings. This flexible application of models is the very condition for their usefulness.

Sometimes, models can be applied to express market participants’ opinions about the economy, the market, or a security. In the words of Mercier and Sperber, they confirm our intuitions. For example, financial analysts often form their views about a company first and then identify the numerical parameters that should be inserted into a model in order to support their pre-formed view (“target price”). In this case, models serve as opinion proclaimers.

Models – often seen as rigorous and objective - frequently function as persuasion devices and provide legitimization for decisions and policies. The findings of the project “Who is leading the change?” show how the insufficiency of modelling of systemic risks immediately after the financial crisis exposed regulators to charges of arbitrariness and mistrust. The need for scientific expertise in form of mathematical models led to the formation of research alliances between central bankers and academic economists. The developed measures and models of systemic risks provided justifications for the new incremental macroprudential regulations and interventions. Hence, models are instruments not just of decision-making but also of decision-selling.

All those examples demonstrate that there are patterns of model use in the practice of policymaking. The systematic research into those patterns might help to better understand what is going on in the economy. Remember the terrific scene in the “Big Short” movie in which Mark Baum and Vinny Daniel ask the representative of Standard & Poor’s why subprime bonds were not downgraded although the quality of the underlying loans was clearly deteriorating. The official starts by arguing about models and ends up stating that the rating agency simply had to keep its customers happy. The necessary link between the final product (the rating) and customer satisfaction is the model that is used as an opinion proclaimer: the rating agencies know which rating they want to produce and use the model to do it. The same can be said for regulatory agencies.

The failure of risk management in some banks occurred not because the risk models were insufficient (accommodated wrong parameters and assumptions) but because bank executives decided to overlay, or ignore, the models that have been specifically installed to warn them. The same practice happens today if we only were to enquire.

There are specific cultures of model use within all political, regulatory and financial organizations that drive the success or failure of policies and decisions. More research on those cultures might help to address some questions for how we really re-build macroeconomics. How do policy makers use macroeconomic models? When do those models guide decisions? When are they neglected and why?

With respect to the recent covid-19 crisis, we need more research and transparency on how models were applied in the discussions at various political boards such as the Scientific Advisory Group for Emergencies or Independent Sage. The type of models or their assumptions may be of lesser interest. Rather, it would be crucial to know whether models were used as future predictors, expressed a pre-formed judgment or functioned as legitimation tools for actions. Such information would tell a lot about the quality of decisions made.