RSS
You may know that there is a reproducibility crisis in psychology. Or is there? Wired and Slate both have pieces up reviewing the current debates on whether there is, or isn’t, a crisis. Perhaps the media is biased, but the behavior and explanations who assert there isn’t a crisis seems informative to me. From the Wired piece:
Emotions are running high. Two groups of very smart people are looking at the exact same data and coming to wildly different conclusions. Science hates that. This is how beleaguered Gilbert feels: When I asked if he thought his defensiveness might have colored his interpretation of this data, he hung up on me.
And now, from Slate:
In his lab, Baumeister told me, the letter e task would have been handled differently. First, he’d train his subjects to pick out all the words containing e, until that became an ingrained habit. Only then would he add the second rule, about ignoring words with e’s and nearby vowels. That version of the task requires much more self-control, he says.
Second, he’d have his subjects do the task with pen and paper, instead of on a computer. It might take more self-control, he suggested, to withhold a gross movement of the arm than to stifle a tap of the finger on a keyboard.
If the replication showed us anything, Baumeister says, it’s that the field has gotten hung up on computer-based investigations. “In the olden days there was a craft to running an experiment. You worked with people, and got them into the right psychological state and then measured the consequences. There’s a wish now to have everything be automated so it can be done quickly and easily online.” These days, he continues, there’s less and less actual behavior in the science of behavior. “It’s just sitting at a computer and doing readings.”
Of course even those who accept and promote a replication crisis often feel their own unreplicated work is an exception to the rule. Basically what this is telling us is that psychologists are subject to the sorts of cognitive biases they themselves study. Some researchers though seem to be facing the problems head-on, Reckoning with the past:
To be fair, this is not social psychology’s problem alone. Many other allied areas in psychology might be similarly fraught and I look forward to these other areas scrutinizing their own work—areas like developmental, clinical, industrial/organizational, consumer behavior, organizational behavior, and so on, need an RPP project or Many Labs of their own. Other areas of science face similar problems too.
During my dark moments, I feel like social psychology needs a redo, a fresh start. Where to begin, though? What am I mostly certain about and where can my skepticism end? I feel like there are legitimate things we have learned, but how do we separate wheat from chaff? Do we need to go back and meticulously replicate everything in the past? Or do we use those bias tests Joe Hilgard is so sick and tired of to point us in the right direction? What should I stop teaching to my undergraduates? I don’t have answers to any of these questions.
This blogpost is not going to end on a sunny note. Our problems are real and they run deep. Okay, I do have some hope: I legitimately think our problems are solvable. I think the calls for more statistical power, greater transparency surrounding null results, and more confirmatory studies can save us. What is not helping is the lack of acknowledgement about the severity of our problems. What is not helping is a reluctance to dig into our past and ask what needs revisiting.
From what I have hear psychological experiments are relatively cheap. Replication is feasible, even if it’s not glamorous. In contrast in biomedicine replication is more expensive. There might therefore be bigger problems lurking out there….
English
Arabic
Chinese (Simplified)
Dutch
French
German
Hebrew
Hindi
Italian
Japanese
Korean
Persian
Polish
Portuguese
Russian
Spanish
Swedish
Turkish
Vietnamese
My impression is that the replication crisis is not driven by a lack of replication, but by selective reporting of results. There are admissions from researchers, evidence from individual studies comparing survey questionnaires to reported results, and funnel plots, all of which indicate or suggest that some social science researchers selectively publish results. The less expensive a study, the easier it is for a researcher to omit that study and conduct another study testing the same hypothesis. If that’s the case, then fields in which studies are more expensive to conduct would have less of a replication crisis, all else equal.
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0010068
but it's not just file-drawer effect. it might be a lot of psych stuff isn't that robust and conceptual replications are too easy....
in support of your contention
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0010068
but it’s not just file-drawer effect. it might be a lot of psych stuff isn’t that robust and conceptual replications are too easy….
Most psychological studies are based on Multiple Regression Analysis which can be very misleading in the first place.
https://www.edge.org/conversation/richard_nisbett-the-crusade-against-multiple-regression-analysis
Thanks for the Inzlicht quote
In that vein
Warning from the Amer. Statistical Assn.
http://www.eurekalert.org/pub_releases/2016-03/asa-asa030116.php
“…”The p-value was never intended to be a substitute for scientific reasoning,” said Ron Wasserstein, the ASA’s executive director. “Well-reasoned statistical arguments contain much more than the value of a single number and whether that number exceeds an arbitrary threshold. The ASA statement is intended to steer research into a ‘post p<0.05 era.'"
"Over time it appears the p-value has become a gatekeeper for whether work is publishable, at least in some fields," said Jessica Utts, ASA president. "This apparent editorial bias leads to the 'file-drawer effect,' in which research with statistically significant outcomes are much more likely to get published, while other work that might well be just as important scientifically is never seen in print. It also leads to practices called by such names as 'p-hacking' and 'data dredging' that emphasize the search for small p-values over other statistical and scientific reasoning…."
Social Psychology is an agar medium for for projection of one's ideology. is too often little more then a Soapbox with, as the ASA suggests a pseudo-scientific veneer of dubious statistics. But as Inzlicht suggests the rest of Psychology is little better.
But I think Inzlicht's answer is too reductionistic a
(sorry, tapped publish by mistake)
But I think Inzlicht’s answer is too reductionistic a procedure. Psychology needs its own methodology based on looking at the Mind. The old Introspectionists were on to something but were way ahead of themselves. Only with modern brain scanning equipment can mental activities actually be studied. It may be cumbersome and expensive for a while but verbal reports etc are not true proxies for how the mind works.
I’ve said it before and I’ll say it again: as a major in behavioral psychology, at best, the field is made up. Even if the studies are replicable the material itself is basically like learning a religion.
Psychology isn’t a real science so who cares. Anyone with the wits to observe is “psychologist”.
My own expectation is that the replication crisis in social psychology — and similar issues in other sciences — is multifactorial.
Any of these might be a source of systematic error:
–unconscious p-hacking
–conscious p-hacking
–publication bias
–unconscious manipulation of subjects
–conscious manipulation of subjects
–unconscious alteration of data
–conscious alteration of data
My own Unified Theory of Scientific Error is this: each source of systematic error will be increased in rough proportion to the stakes of the experimenters in the outcome of the experiment.
So, if an experimenter is simply trying to make a name for himself on a given subject, he will (on average) be more liable to commit each kind of error. If he is part of a certain school of thought, he (and others in his school) will be more liable to commit each sort of error to favor results which would support that school. And, in the possibly worst case, if a large group of experimenters have a particular ideological axe to grind with a certain result, then each member will be more liable to make each kind of error, and in aggregate the group will commit each kind of error in great quantities, will be least likely to catch or expose such errors, and each member will increase the viability of every other member’s errors through a feedback process. If the ideological bias is everywhere in the profession, the error will grow like Topsy.
Probably the only way to counteract such errors is to recognize explicitly the power of a stake in a experiment to bias the published outcomes, and take corrective measures at every level — and, if possible, to introduce experimenters with opposite or no stakes in the outcome.
I, like Steven Pinker, find it very instructive to note that IQ research has mostly avoided this replication crisis.
The best source I’ve found on that whole replicability kerkuffle is Sanjay Srivastava, aka @hardsci on twitter
Especially recommended :
https://hardsci.wordpress.com/2015/09/02/moderator-interpretations-of-the-reproducibility-project/
https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-critique-of-the-reproducibility-project/
Also lost in the public discussion is that some social psychology effects did replicate, very strongly and very reliably (especially anchoring).
From the original ManyLabs replication project:
Should it be a requirement for every science/social science Ph.D to have an accompanying master’s (at least) in mathematical statistics (not the cheap business statistics)? I know, a M.S. in mathematical statistics is a lot of math as well as statistics, much more than most Ph.Ds are familiar with, and it would pile on to an already heavy course load, but such a requirement could help avoid the problem of so many bogus studies.
The issue is not statistics here; the issue is reproducibility of data. If the data cannot be replicated, a masters in statistics will do you no good.Replies: @Daniel H
This is ridiculous. Getting a degree of any kind in math is a challenge for most scientists. Asking for a masters in statistics for any scientist, would be very difficult.
The issue is not statistics here; the issue is reproducibility of data. If the data cannot be replicated, a masters in statistics will do you no good.
The issue is not statistics here; the issue is reproducibility of data. If the data cannot be replicated, a masters in statistics will do you no good.Replies: @Daniel H
So much of science is statistics. Do many scientists truly understand statistics. Facility with statistical software packages, or programming in R does not ensure that one necessarily understands the output. Maybe if the scientist had a true understanding of statistics they would realize that the results of their investigation were meaningless. Yeah, math and statistics are a challenge to learn, but it is the life they chose.