Modeler's Hippocratic Oath

This will be familiar to some people already. In an essay put out in 2009 Emanual Derman and Paul Wilmott write a manifesto aimed at the financial modelers, but with points that can apply to any modelers. The highlight of the article is the Modeler's Hippocratic Oath. Of course, the original Hippocratic oath was the medical doctors pledge (to the Greek gods) to do no harm (sort of). What would it look like if modern modelers were forced to make such a pledge? Maybe something like this:

• I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
• Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
• I will never sacrifice reality for elegance without explaining why I have done so.
• Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
• I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.

What a great idea! Even the parts that are more clearly aimed at economic modelers would work pretty well for psychological modelers. In the second line, we might change 'value' to 'values' and in the final line we might or might not remove explicit mention of the economy. In any case, we would have an oath that in no way stops mathematical modeling, but serves to remind us to be explicit about our assumptions, and to dissuade others from having false confidence in our models.

Psychology has its share of formal mathematical models of the type of the type Derman and Wilmott are concerned with, but it also has other types of models. Psychologists increasingly rely on agent-based models to simulate behavior, and in the future (one hopes) will rely on agent-based models to update their hypothesis-testing methods. Psychologists also rely on many forms of statistical modeling, such as factor analysis, structural-equation modeling, hierarchical linear modeling, neural network models. Some of these contain painfully arbitrary elements. For example, there is no hard and fast rule for determining the number of factors indicated by a factor analysis, and structural equation models are often built to test a specific theory in a way that makes unpredicted results unlikely to create problems. Also, almost all of these methods have a tenuous connection to the psychology of an individual person. Even a very good structural equation model, quite worthy of publication, might be very poor at predicting any individual outcome - e.g. whether a particular person might become a drug user.

I am not against doing any of this work, but I would like a bit of that oath mixed in. Make the assumptions and oversights explicit, remind people about the inaccuracies of the model if they seem to be getting to confident. 

The neural-network models are particularly troublesome. They have the additional flaw of containing the work "neural" in the name, so people get confused and think that they work like brains. But neural network models are just fancy regression equations, very useful for some purposes, but not at all a biologically sophisticated simulation of neural functioning. For example, one critical element of brain function is the synchronous or asynchronous arrival of action potentials at given locations (e.g. what gets screwed up in Multiple Sclerosis, when you start loosing your mylen sheaths). Again, I am not advocating that we stop neural-network modeling; I am advocating that the modeler be taught to feel obliged to interrupt and correct anyone who starts to imply that the models work in exactly the same way that actual nervous systems work.

By the way, another line from the essay is:

The most important question about any financial model is how wrong it is and how useful it is despite its assumptions.

Yes, yes! Not true models are useful, but only if we remember that they are not true. One of Skinner's criticisms of the use of mental terms as place-holders (e.g. using them as we await further investigation) was that the next generation of scientists always seemed to forget they were place-holders. This has happened with several major psychological phenomenon in recent times. For example, the "central executive" was supposed to be a place-holder term for all the things the brain did that we couldn't yet put in a more specific part of a model. It was kind of like cognitive psychology's "black box", only with the hope that the box would shrink over time as we got better models for specific mental functions. However, very quickly people forget that the central executive wasn't a thing and there are now countless studies about the role and function of the central executive in various processes.

That last part might not be on topic completely, I am a bit feverish... but I'd love to know what people think about the idea of the modeler's oath.