A blog about problems in the field of psychology and attempts to fix them.

Wednesday, August 8, 2012

New Trends for Finding Fraud in Experimental Psychology

I have just returned from my first American Psychological Association meeting that I thought was worth attending. I noted in a previous post that APA meetings now feature a wide array of division activities, and that those are worth going to, regardless of the main convention. This time, however, there were a handful of talks (out of many) which were valuable enough to make me think the main convention was starting to turn things around. The highlight was a session organized by Joseph Simmons, and featuring talks by Leslie John and Uri Simonsohn.
 

You might remember Simmons and Simonsohn as two of the three authors of last year's killer article False-Positive Psychology. Simonsohn is also the "data detective" responsible a few recent resignations. Simonsohn's methods in the latter cases looking for anomalous published results (e.g. effect sizes well above other effects in the same liturature), then looking a layer deeper into the data than most fraudsters would worry about (e.g., it is easy to make a correlation matrix look right, but who bothers to fake a realistic covariance matrix if they are not reporting it). -- For more details, see Neuroskeptic's good discussion here. -- All three talks in the session were solid, and the broader project forming here is really important for our field. The session finale was Simonsohn's explanation of his new fraud detection tool.

Recall that in the "false-positive" paper, no one was actually accused of acting maliciously. Instead, it was shown that series of fairly standard moves by psychologists made it much easier than it should to find an effect when there was not one. These common techniques included the adding and subtracting of covariates, running tests with and without outliers, and testing "just one more" participant over and over again. All of these are procedures that are not to bad when used in moderation, and might well be essential during the exploratory phase of a research program. However, if used too much, or too carelessly, and without follow up replication, they provide ample opportunity for false positives (i.e. type 1 error).

I won't get too specific, as he is has it under submission, but I am pretty sure I can get the point across in a way that is not "scooping" anything. The idea is to look, either within a lab or across a wider literature, at the distribution of published probability values. If a person, or lab, or literature, is routinely using techniques that generate false positives, you would expect a distribution biased towards the barely-good-enough-to-publish side of things (i.e., lots of p values near .05). This is because there is no reason to keep massaging the data (innocently or not) after you have passed the publishability threshold. Simonsohn had an excel spread sheet where you could enter the t or F values from studies, along with their degrees of freedom, determine the distribution, and calculate a probability that people were (innocently or not) using techniques that bias towards false positives. There was also a check to see if the published values fit with honest reporting in a situation where the null hypothesis was true and a test of fit for honest reporting in a situation where the null hypothesis was false. Like Simonsohn's data detective work, this involves dropping to a level of statistical analysis below where most people usually look: The probability of probability distributions! Still resisting the urge to scoop his paper, I can only say that it was brilliant. Simple, straightforward, obvious once you hear it, and potentially devastating.

This has the potential to disrupt several prominent labs in psychology, and possibly several whole literatures. Simonsohn seemed less interested (though not uninterested) in the meta-analytic implications. Attending this session and being able to talk with the Simmons and Simonsohn afterwards was almost enough on its own to make attending the entire APA conference worth it.

What I have missed at APA in the past was the feeling that I was hearing novel brilliant ideas, and gaining the opportunity to meeting brilliant people I didn't already know. This APA, it happened a few times. I am really close to looking forward to the next conference.

2 comments:

  1. Sounds like that will be a really interesting paper, I'll have to make sure I read it when it comes out. It makes sense that detecting fraud is possible by examining the distribution of p-values across a researchers published findings. Of course publication bias will likely exist (most published findings are generally significant), which is no big surprise. However, the distribution of p-values below 0.05 should otherwise follow the expected distribution of p-values under the alternative, given a large enough sample of publications. I would think a method that examines deviation from that expected distribution should be able to detect when something questionable is going on. I'm really interested in seeing how the actual method works.

    It kind of reminds me of the logic behind the false discovery rate method for multiple testing correction.

    ReplyDelete
  2. Since neuroskeptic's blog is having some of my comments evaporating, I send you here 2 comments of mine who appeared only briefly on his blog on the post you commented too.

    http://www.blogger.com/comment.g?blogID=2733981550095578188&postID=6618721415047067322Ivana Fulli MD said...
    Dear Andrew,

    Broadly speaking, I am on your side on that post and i am grateful to neuroskeptic for letting you express your opinion slightly out of topic- or putting the problem in perspective...

    But be careful when you write things like:

    ///(...) non-diagnostic symptoms that are part of the syndrome (...)///

    For a lawyer or you average neuroscientist I cannot answer; but for any MD it sounds rather difficult to accept the notion of "non-diagnostic symptoms" being part of a medical syndrome".

    May be you meant " non pathognomonic symptoms "

    Nb: This is just a friendly remark since I like people with the courage to marry for love outside their own culture .Reasonable busy people know when they are just writing a comment on sb's blog and not writing a blog post and they write in a hurry.

    28 August 2012 08:32

    Ivana Fulli MD said...

    Andrew,

    "Replication and replicability aren't so terribly important when the hole discipline is lost in the wilderness and using methods that have far too low resolution to capture what is going on in a complex world."

    How can a USA lawyer imply that giving public or benefactor money to fraudsters is no sin ?

    The "bestest best best" (Eric Charles 26 08 12 at 00:10 )are rare by definition and neurosciences in general are lacking good innovative ideas but at least you can expect integrity as a minimum in a profession that claim to be scientific because they use stats!

    28 August 2012 12:52

    You are indeed inspiring and checking to see if our Neuroskeptic has censored my last comment is an incitation.

    No offense intented though since my eldest son, Samuel Fulli-Lemaire , a law PhD candidate, is a great admirer of USA lawyers and I trust his superior intelligence he took from his father!

    8/28/2012 5:58 AM

    ReplyDelete