The Unz Review: An Alternative Media Selection
A Collection of Interesting, Important, and Controversial Perspectives Largely Excluded from the American Mainstream Media
 TeasersiSteve Blog
The Replication Crisis: Is Psychology More Like Astronomy or Marketing Research?
🔊 Listen RSS
Email This Page to Someone

 Remember My Information



=>

Bookmark Toggle AllToCAdd to LibraryRemove from Library • BShow CommentNext New CommentNext New ReplyRead More
ReplyAgree/Disagree/Etc. More... This Commenter This Thread Hide Thread Display All Comments
AgreeDisagreeLOLTroll
These buttons register your public Agreement, Disagreement, Troll, or LOL with the selected comment. They are ONLY available to recent, frequent commenters who have saved their Name+Email using the 'Remember My Information' checkbox, and may also ONLY be used once per hour.
Ignore Commenter Follow Commenter
Search Text Case Sensitive  Exact Words  Include Comments
List of Bookmarks

Marketing!

There has been much discussion lately of the “Replication Crisis” in psychology, especially since the publication of a recent study attempting to replicate 100 well-known psychology experiments. From The Guardian:

Study delivers bleak verdict on validity of psychology experiment results

Of 100 studies published in top-ranking journals in 2008, 75% of social psychology experiments and half of cognitive studies failed the replication test

For more analysis, see Scott Alexander at SlateStarCodex: “If you can’t make predictions, you’re still in a crisis.”

(By the way, some fields in psychology, most notably psychometrics, don’t seem to have a replication crisis. Their PR problem is the opposite one: they keep making the same old predictions, which keep coming true, and everybody who is anybody therefore hates them for it, kill-the-messenger style. For example, around the turn of the century, Ian Deary’s team tracked down a large number of elderly individuals who had taken the IQ test given to every 11-year-old in Scotland in 1932 to see how their lives had turned out. They found that their 1932 IQ score was a fairly good predictor. Similarly, much of The Bell Curve was based on the lives of the huge National Longitudinal Study of Youth 1979 sample up through 1990. We now have another quarter a century of data with which to prove that The Bell Curve doesn’t replicate. And we even have data on thousands of the children of women in the original Bell Curve sample. This trove of data is fairly freely available to academic researchers, but you don’t hear much about findings in The Bell Curve failing to replicate.)

Now there are a lot of reasons for these embarrassing failures, but I’d like to emphasize a fairly fundamental one that will continue to plague fields like social psychology even if most of the needed methodological reforms are enacted.

Consider the distinction between short-term and long-term predictions by pointing out two different fields that use scientific methods but come up with very different types of results.

At one end of the continuum are physics and astronomy. They tend to be useful at making very long term predictions: we know to the minute when the sun will come up tomorrow and when it will come up in a million years. The predictions of physics tend to work over very large spatial ranges, as well. As our astronomical instruments improve, we’ll be able to make similarly long term sunrise forecasts for other planetary systems.

Why? Because physicists really have discovered some Laws of the Universe.

At the other end of the continuum is the marketing research industry, which uses scientific methods to make short-term, localized predictions. In fact, the marketing research industry doesn’t want its predictions to be assumed to be permanent and universal because then it would go out of business.

For example, “Dear Jello Pudding Brand Manager: As your test marketer, it is our sad duty to report that your proposed new TV commercials nostalgically bringing back Bill Cosby to endorse your product again have tested very poorly in our test market experiment, with the test group who saw the new commercials going on to buy far less Jello Pudding over the subsequent six months than the control group that didn’t see Mr. Cosby endorsing your product. We recommend against rolling your new spots out nationally in the U.S. However, we do have some good news. The Cosby commercials tested remarkably well in our new test markets in China, where there has been far less coverage of Mr. Cosby’s recent public relations travails.”

I ran these kind of huge laboratory-quality test markets over 30 years ago in places like Eau Claire, Wisconsin and Pittsfield, MA. (We didn’t have Chinese test markets, of course.) The scientific accuracy was amazing, even way back then.

But while our marketing research test market laboratories were run on highly scientific principles, that didn’t necessarily make our results Science, at least not in the sense of discovering Permanent Laws of the Entire Universe. I vaguely recall that other people in our company did a highly scientific test involving Bill Cosby’s pudding ads, and I believe Cosby’s ads tested well in the early 1980s.

But that doesn’t mean we discovered a permanent law of the universe: Have Bill Cosby Endorse Your Product.

In fact, most people wouldn’t call marketing research a science, although it employs many people who studied sciences in college and more than a few who have graduate degrees in science, especially in psychology.

Marketing Research doesn’t have a Replication Crisis. Clients don’t expect marketing research experiments from the 1990s to replicate with the same results in the 2010s.

Where does psychology fall along this continuum between physics and marketing research?

Most would agree it falls in the middle somewhere.

My impression is that economic incentives push academic psychologists more toward interfacing closely with marketing research, which is corporate funded. For example, there are a lot of “priming” studies by psychologists of ways to manipulate people. “Priming” would be kind of like the active ingredient of “marketing.”

Malcolm Gladwell discovered a goldmine in recounting to corporate audiences findings from social sciences. People in the marketing world like the prestige of Science and the assumption that Scientists are coming up with Permanent Laws of the Universe that will make their jobs easier because once they learn these secret laws, they won’t have to work so hard coming up with new stuff as customers get bored with old marketing campaigns.

That kind of marketing money pushes psychologists toward experiments in how to manipulate behavior, making them more like marketing researchers. But everybody still expects psychological scientists to come up with Permanent Laws of the Universe even though marketing researchers seldom do. Psychologists don’t want to disabuse marketers of this delusion because then they would lose the prestige of Science!

 
Hide 50 CommentsLeave a Comment
Commenters to Ignore...to FollowEndorsed Only
Trim Comments?
  1. Pittsfield, MA? Dude, my hometown. Not as diverse 30 years ago (when I lived there) as it is now, where it’s only 87% white.

    • Replies: @Steve Sailer
    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    And we had seven other similar test markets at our peak.

    These were amazing real world laboratories, especially considering how long ago the early 1980s were.

    We used the best experiment principles devised by scientists to conduct our experiments. We employed quite a few Ph.D.s, whom we paid better than academia.

    But were the results we discovered Science?
  2. graduate degrees in science, especially in psychology.

    I have a feeling that people with actual graduate degrees in science are insulted you included marketing in this.

    • Replies: @ScarletNumber
    I meant to say psychology, but the point still holds.
    , @Realist
    Psychology is not science
  3. I once worked at the local office of the U.S. Congressman in Pittsfield (during Clinton’s impeachment, as it happens). It seemed like a very static place. Might be that perception had more to do with the nature of Congressman’s offices than with the city.

    • Replies: @Steve Sailer
    Pittsfield, MA is home to the Norman Rockwell Museum (thanks to major donations from Steven Spielberg, by the way). It's in the Berkshires and seems to be a pretty nice place (I was there for a few hours on business in the 1980s and drove through last summer. It's near the summer home of the Boston Symphony.)

    The residents probably want to keep it static.

  4. After reading the various articles about the replication problem over the last few weeks, I would think that there is a niche somewhere for an Amazing Randi type to tear down the entire rotten edifice. Priming is bullshit. Stereotype threat is bullshit. But it is pleasing bullshit, and one can make a career of telling folks what they want to hear. Jimmy Swaggart, for example.

  5. @ScarletNumber

    graduate degrees in science, especially in psychology.
     
    I have a feeling that people with actual graduate degrees in science are insulted you included marketing in this.

    I meant to say psychology, but the point still holds.

  6. I have long wondered whether psychologists’ ideological refusal to recognize group differences undermined the bulk of these studies. They rarely identify the race or ethnicity of subjects and blindly assume that these factors would not have any effect on the outcome. It seems very unlikely. The replication failure may be linked to this, at least on some significant level.

    • Agree: Stan D Mute
  7. I think two other major factors are affecting the reproducibility of psychology studies, and social sciences studies generally. First, there’s an enmeshment between popular outlets that publicize these studies’ results, and the scientists who perform the studies. The scientists want the publicity of having their study referenced in some magazine, and the magazine writers love having the veneer of Science fronting for their biases. This puts pressure on the research to find “interesting” results, rather than particularly honest ones – and this is especially true if the study can be related to political alignments, however spuriously (see e.g. “Conservatives scare easily”). The second factor is the biases of the scientists themselves. It’s been repeatedly shown that social science peer groups have remarkably similar worldviews. This of course pushes results away from the unfettered truth, toward an ideologically preferable one. There is research about this that has been published, particularly by Haidt et al. Although who knows if it will prove reproducible.

    • Replies: @Steve Sailer
    "First, there’s an enmeshment between popular outlets that publicize these studies’ results, and the scientists who perform the studies."

    We could call it the Gladwell Effect.

    Malcolm didn't invent it, but his example encouraged psychologists to look for stuff that would sound cool to marketers.

  8. Al Qaeda has put out a hit list on Warren Buffet, Michael Bloomberg, and Bill Gates. Al Qaeda should view these 3 billionaires as natural allies and not enemies. After all these 3 billionaires want to flood White Western Christian nations with masses of 3rd world Muslims. These 3 billionaires want to destroy White Western civilization just like Al Qaeda does.

  9. I wonder how much priming goes into national surveys on the public’s attitudes toward controversial issues like immigration?

    • Replies: @Steve Sailer
    There exist outstandingly talented primers -- Taylor Swift, Oprah Winfrey, Donald Trump, the Kardashian mom, etc. -- but very few of them are academic psychologists.
  10. Anonymous • Disclaimer says:

    Where does psychology fall along this continuum between physics and marketing research? Most would agree it falls in the middle somewhere.

    Yes, it does, but the most pertinent question is “should it?” The answer is categorical “no”. The entire goal of psychology is to discover fundamental rules that govern human behavior. Those cannot vary significantly within decades and if the are found to do so then the only conclusion is that the research in question was garbage.

    • Replies: @Steve Sailer
    "the only conclusion is that the research in question was garbage"

    Or, maybe, they discovered something that was true for some college students in 1996 but isn't true in other times and places. Like it wouldn't have been too hard to prime college students in 1996 into dancing the Macarena but now it would be hard.

    Marketing researchers discover pretty often that marketing plans are working, but that doesn't mean they've discovered a permanent truth of the universe, just something that is contingent to some time and place.

  11. A Black thug in Detroit was recently in the news because he openly admitted that he only robs White people and never robs Black people.

    He must not make too much money as a mugger in 83 percent Black Detroit if he only mugs White people at gunpoint.

    Detroit is only 7 percent Non Hispanic White. It is even less White than that if you only include Northwest Europeans as White and exclude Detroit’s Arab population.

    This mugger is missing out on a lot of financial opportunities by not robbing any Blacks in Detroit, the city’s largest racial group by a huge landslide.

    This Black mugger should venture out to lily White suburbs like Livonia if he wants to make a decent living mugging White people at gunpoint. Livonia is Whiter than sour cream. It is the complete opposite of Detroit’s city limits.

  12. @EriK
    Pittsfield, MA? Dude, my hometown. Not as diverse 30 years ago (when I lived there) as it is now, where it's only 87% white.

    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    And we had seven other similar test markets at our peak.

    These were amazing real world laboratories, especially considering how long ago the early 1980s were.

    We used the best experiment principles devised by scientists to conduct our experiments. We employed quite a few Ph.D.s, whom we paid better than academia.

    But were the results we discovered Science?

    • Replies: @Lugash
    I'd say Not Science. It's an interesting use of scientific principles and experimentation, but unless you can move from short life span pattern recognition to durable scientific laws, it doesn't count.

    Thinking of sociology, anthropology, psychology etc as short term pattern recognition is useful idea as well.
    , @FactsAreImportant

    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    ...

    But were the results we discovered Science?
     

    This sounds very much like science to me. Careful experimental design crafted to cleanly focus on the effect of a well-defined independent variable.

    Your criticisms of marketing research stem more from the fact that human behavior is complicated and evolves over time. That doesn't mean marketing research is not science, it just means that it is difficult science.

    The crisis in academic psychology comes from too many researchers chasing too few worthwhile results within a shoddy peer-review system where there are way too many ways to nudge the data into making the desired result statistically significant. These distortions render much psychology research non-scientific. The replication study you cite pretty much proves this. Good scientific research is replicable. Bad research where the data is nudged is not replicable because a second, non-nudging researcher will not get the same result.

    I have to respectfully disagree with the thrust of this entire post.

    , @Buzz Mohawk
    To make some money before starting college, I worked for six months as a lab tech at the R&D center for a major building products manufacturer. My job involved applying scientific testing methods to measure the properties of new formulas for things like fiberglass roofing mat, insulation and boards.

    I ripped things apart, burned them, and poked holes in them. I subjected them to extreme simulated climate changes over time in a programmable environmental chamber. I blew air through them in a wind tunnel. Then I calculated standard deviations and other things for my data. Engineers used my results to determine if their new formulas and products were good enough.

    This was applied science, engineering if you will. It was not science because I was not discovering anything applicable outside our specific cases.

    Your work in market research sounds the same as this, applied to people and advertising formulas.

    You were doing engineering!

    , @NeonBets
    What you're referring to from your work in the 1980s is simply the analogue version of Google's entire Business Plan. And if that's not "science" it is certainly "scientific".

    Reproduction is the flip-side to "Is the conclusion falsifiable?" And it's important to keep this in mind, especially when dealing with jumps from the specific to the general (ie induction).

    "63 out of 100 college students exhibited a loss of empathy when given the power to deliver electric shocks, therefore all humans tend to lose empathy when given power".

    How can one refute this statement? And, look at the burden placed on potential opponents. If I'm Stanley Milgram I spent very little time performing stupid, useless "experiments", but it would take an opponent damn near a lifetime to gather enough samples and re-tests to see if Milgram's conclusions have any inductive force. And at the end of it all, we still have nothing definitive--only percentages.

    Social "science" has turned science upside-down where credulity and acceptance are easy and skepticism and doubt are difficult.
  13. @Luke Lea
    I wonder how much priming goes into national surveys on the public's attitudes toward controversial issues like immigration?

    There exist outstandingly talented primers — Taylor Swift, Oprah Winfrey, Donald Trump, the Kardashian mom, etc. — but very few of them are academic psychologists.

  14. @Anonymous

    Where does psychology fall along this continuum between physics and marketing research? Most would agree it falls in the middle somewhere.
     
    Yes, it does, but the most pertinent question is "should it?" The answer is categorical "no". The entire goal of psychology is to discover fundamental rules that govern human behavior. Those cannot vary significantly within decades and if the are found to do so then the only conclusion is that the research in question was garbage.

    “the only conclusion is that the research in question was garbage”

    Or, maybe, they discovered something that was true for some college students in 1996 but isn’t true in other times and places. Like it wouldn’t have been too hard to prime college students in 1996 into dancing the Macarena but now it would be hard.

    Marketing researchers discover pretty often that marketing plans are working, but that doesn’t mean they’ve discovered a permanent truth of the universe, just something that is contingent to some time and place.

  15. @StoneKicker
    I think two other major factors are affecting the reproducibility of psychology studies, and social sciences studies generally. First, there's an enmeshment between popular outlets that publicize these studies' results, and the scientists who perform the studies. The scientists want the publicity of having their study referenced in some magazine, and the magazine writers love having the veneer of Science fronting for their biases. This puts pressure on the research to find "interesting" results, rather than particularly honest ones - and this is especially true if the study can be related to political alignments, however spuriously (see e.g. "Conservatives scare easily"). The second factor is the biases of the scientists themselves. It's been repeatedly shown that social science peer groups have remarkably similar worldviews. This of course pushes results away from the unfettered truth, toward an ideologically preferable one. There is research about this that has been published, particularly by Haidt et al. Although who knows if it will prove reproducible.

    “First, there’s an enmeshment between popular outlets that publicize these studies’ results, and the scientists who perform the studies.”

    We could call it the Gladwell Effect.

    Malcolm didn’t invent it, but his example encouraged psychologists to look for stuff that would sound cool to marketers.

  16. @Anonymous
    I once worked at the local office of the U.S. Congressman in Pittsfield (during Clinton's impeachment, as it happens). It seemed like a very static place. Might be that perception had more to do with the nature of Congressman's offices than with the city.

    Pittsfield, MA is home to the Norman Rockwell Museum (thanks to major donations from Steven Spielberg, by the way). It’s in the Berkshires and seems to be a pretty nice place (I was there for a few hours on business in the 1980s and drove through last summer. It’s near the summer home of the Boston Symphony.)

    The residents probably want to keep it static.

    • Replies: @jjbees
    Here to shill again for western mass!

    If you get to New England (in the summer or fall) you really need to go up to Lenox and Great Barrington.

    Tanglewood is the summer home of the boston symphony orchestra and it is a great place to sit out on their grass fields having a picnic and wine, listening to some nice ye olde violin or somesuch.

    The area is incredibly picturesque, and the reason, in my opinion, for the love people have for the area is that it has four seasons and tends to be nice and dry.

    Lenox/Great Barrington are never humid! On the hottest days of the year it will be 80F with a light breeze.

    I went for lunch and a long drive in the countryside with my father last weekend and it was the highlight of my summer. Western mass is god's country, I only wish I still lived there!
    , @Buzz Mohawk

    Pittsfield, MA is home to the Norman Rockwell Museum.
     
    That's actually in nearby Stockbridge, where he lived, but close enough. I hiked through on the Appalachian Trail back in ancient times...
  17. “Game” would probably fall somewhere in the middle as well, leaning toward the short-term, since distilled down it’s mainly psychology leaning towards the marketing side of things. I think it probably has some long-term permanent principles but a lot of the nuts and bolts can be vary a lot depending on the situation, time, and place.

    What are some examples from sociology/psychology other than psychometrics where simple theories have proven to have long-term predictive power?

    • Replies: @G Pinfold
    The USP of game seems to be that it relies on evo psych laws which are more universal than the cultural ephemera of romance and all that.
  18. The Cosby commercials tested remarkably well in our new test markets in China

    “in Detroit” would have been funnier. And more likely.

  19. @Steve Sailer
    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    And we had seven other similar test markets at our peak.

    These were amazing real world laboratories, especially considering how long ago the early 1980s were.

    We used the best experiment principles devised by scientists to conduct our experiments. We employed quite a few Ph.D.s, whom we paid better than academia.

    But were the results we discovered Science?

    I’d say Not Science. It’s an interesting use of scientific principles and experimentation, but unless you can move from short life span pattern recognition to durable scientific laws, it doesn’t count.

    Thinking of sociology, anthropology, psychology etc as short term pattern recognition is useful idea as well.

  20. In Pittsfield, Melville wrote Moby Dick at a desk next to a window with a view of Mount Greylock.

  21. GS VS. IS
    Great Satan is favorite to defeat Islamic State. European Lurch: https://theconversation.com/why-did-europe-lurch-right-because-the-recovery-is-a-farce-27363
    IS is a farce. The trouble is not in the stars it is in ourselves thanks to Great Satan.

  22. No idea if it’s still true, but as an undergrad as part of Psychology 101 we were required to be part of numerous surveys and experiments for graduate level students and professors. Pretty clear any study depending on bored, sleep-deprived, indifferent often hungover 17-22 year olds is not going to produce any data nor analysis of it you could call scientific. Love the sex and drug ones. How many teen boys that age are going to say they haven’t fornicated nor smoked dope?

  23. @Jokah Macpherson
    "Game" would probably fall somewhere in the middle as well, leaning toward the short-term, since distilled down it's mainly psychology leaning towards the marketing side of things. I think it probably has some long-term permanent principles but a lot of the nuts and bolts can be vary a lot depending on the situation, time, and place.

    What are some examples from sociology/psychology other than psychometrics where simple theories have proven to have long-term predictive power?

    The USP of game seems to be that it relies on evo psych laws which are more universal than the cultural ephemera of romance and all that.

  24. @Steve Sailer
    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    And we had seven other similar test markets at our peak.

    These were amazing real world laboratories, especially considering how long ago the early 1980s were.

    We used the best experiment principles devised by scientists to conduct our experiments. We employed quite a few Ph.D.s, whom we paid better than academia.

    But were the results we discovered Science?

    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    But were the results we discovered Science?

    This sounds very much like science to me. Careful experimental design crafted to cleanly focus on the effect of a well-defined independent variable.

    Your criticisms of marketing research stem more from the fact that human behavior is complicated and evolves over time. That doesn’t mean marketing research is not science, it just means that it is difficult science.

    The crisis in academic psychology comes from too many researchers chasing too few worthwhile results within a shoddy peer-review system where there are way too many ways to nudge the data into making the desired result statistically significant. These distortions render much psychology research non-scientific. The replication study you cite pretty much proves this. Good scientific research is replicable. Bad research where the data is nudged is not replicable because a second, non-nudging researcher will not get the same result.

    I have to respectfully disagree with the thrust of this entire post.

  25. It’s not just the replicability of results. The entire field of psychology / psychiatry is just complete dog shit.

    Do you know that most PCPs and insurance companies do NOT recommend psycho-therapy for serious PTSD patients? Because the patient outcomes for those directed into pyscho-therapy are WORSE than the outcomes for those just left to their own devices. If you suffer a horrible injury / death of a loved one / traumatic experience, you’re more likely to attempt suicide, self-medicate through drugs, or suffer other permanent catastrophic effects if you go see a shrink. As opposed to being told “just try to, you know, deal with it.”

    That’s pretty much the one thing that psycho-therapy should be able to help with. If the AMA came out and said “we recommend seeing your doctor for anything … except a serious injury, in which case you’re better trying to handle it yourself,” would anyone take doctors seriously?

  26. In the ongoing iSteve effort of bringing everything back to golf, I was part of the data of Steve Leavitt’s Freakonomic golf book.

    The thesis was forget ball striking, chips, pitched and putting is the game. Which it is. Dave Pelz proved that 30 years ago.

    This was 3 years ago. The thesis failed.

  27. anonymous • Disclaimer says:

    Marketing research, science or just divining fads? So why are there so many blacks featured in commercials, usually as all-wise types? Does having blacks in ads really sell well to the majority? It’s hard to believe they are more persuasive than white actors would be in those roles. It’s understood that the government forces them to hire blacks and to showcase them but they’re way overrepresented well beyond meeting the minimum requirements.

    • Replies: @Realist
    "Marketing research, science or just divining fads? So why are there so many blacks featured in commercials, usually as all-wise types?"

    This is something I have noticed as well. Also blacks are given alpha roles, in commercials, to wimpy whites.
    , @Jim Don Bob
    "So why are there so many blacks featured in commercials, usually as all-wise types?"

    Because in the early 90s, the New York Times (!) got sued for not having any/enough black faces in their real estate ads. Everybody got the message.
  28. @Steve Sailer
    Pittsfield, MA is home to the Norman Rockwell Museum (thanks to major donations from Steven Spielberg, by the way). It's in the Berkshires and seems to be a pretty nice place (I was there for a few hours on business in the 1980s and drove through last summer. It's near the summer home of the Boston Symphony.)

    The residents probably want to keep it static.

    Here to shill again for western mass!

    If you get to New England (in the summer or fall) you really need to go up to Lenox and Great Barrington.

    Tanglewood is the summer home of the boston symphony orchestra and it is a great place to sit out on their grass fields having a picnic and wine, listening to some nice ye olde violin or somesuch.

    The area is incredibly picturesque, and the reason, in my opinion, for the love people have for the area is that it has four seasons and tends to be nice and dry.

    Lenox/Great Barrington are never humid! On the hottest days of the year it will be 80F with a light breeze.

    I went for lunch and a long drive in the countryside with my father last weekend and it was the highlight of my summer. Western mass is god’s country, I only wish I still lived there!

  29. @Steve Sailer
    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    And we had seven other similar test markets at our peak.

    These were amazing real world laboratories, especially considering how long ago the early 1980s were.

    We used the best experiment principles devised by scientists to conduct our experiments. We employed quite a few Ph.D.s, whom we paid better than academia.

    But were the results we discovered Science?

    To make some money before starting college, I worked for six months as a lab tech at the R&D center for a major building products manufacturer. My job involved applying scientific testing methods to measure the properties of new formulas for things like fiberglass roofing mat, insulation and boards.

    I ripped things apart, burned them, and poked holes in them. I subjected them to extreme simulated climate changes over time in a programmable environmental chamber. I blew air through them in a wind tunnel. Then I calculated standard deviations and other things for my data. Engineers used my results to determine if their new formulas and products were good enough.

    This was applied science, engineering if you will. It was not science because I was not discovering anything applicable outside our specific cases.

    Your work in market research sounds the same as this, applied to people and advertising formulas.

    You were doing engineering!

    • Replies: @Steve Sailer
    That sounds about right.
  30. Over at ClimateAudit today, I ran across the Wagenmaker’s “Reasearchers degree of freedom”. It’s hard to reproduce something if you don’t know how heavily the original author applied their thumb to one side of the scale or the other

  31. University professors studying physics in the 18th and 19th centuries began uncovering “laws” of nature. The really neat thing about a “law” like “F=ma” is that you can solve an amazing variety of problems without having to look at nature again – all you need is the law! Just fill in the variables. Physicists started getting a lot of respect, because their laws made amazing things possible, like radio, cars and appliances. And, of course, they kept doing direct experiments, because it’s the best way, even if you have solid laws.

    In the first part of the 20th century, medical doctors started discovering things that actually made medicine useful and desirable. Doctors started to get a lot of respect, because their uncovered “laws” started really saving lives (especially vaccines and antibiotics and all the surgeries that became possible).

    This was all good.

    But then, other departments in the university started glomming onto this idea, which we might call “science”; they used words like “force” or “law” or “experiment” and they too uncovered what they felt were “laws”. They started to demand equal respect; a “doctor” of sociology was the equal of a doctor of medicine or a guy with a doctorate in physics. Their argument was that if one takes a “scientific” approach, one necessarily gets to truth as a result. Unfortunately, this turns out to be bullshit.

    This is where the problem started. Sociology, psychology and political science professors came up with laws like “Men and women are equal!” And once you have a law, you can solve all kinds of problems without ever having to check with reality.

    If you really want to start an argument with an academic, insist that some kinds of science are better than others. Like, “physics is the strongest science, because you can always get direct experimental verification whenever you want” or “sociology is a weak science, since the supposed results change constantly”.

    • Replies: @Realist
    Excellent points. But I would call sociology a pseudoscience.
    , @Anonymous

    professors came up with laws like “Men and women are equal!”
     
    professor profess :"to take a vow" (in a religious order)
    doctor: ..., from Medieval Latin doctor "religious teacher, adviser, scholar,"
    dean : ... from Late Latin decanus "head of a group of 10 monks in a monastery,"
    rector : ... ruler ... by 18c. generally restricted to clergymen and college heads

    They were here first. Universities started as religious institutions, and now they returned to their roots.
  32. One thing psychology, as a scientific discipline, has going against it is that people are outstanding natural psychologists; we evolved that way.

    The finest natural psychologists are our great writers. You can learn alot more from them, at least on subjects that matter, than from all the peer-reviewed psychology papers ever published.

    In contrast, people have no innate intuition about electricity, magnetism, or quantum mechanics. So physics is worth studying formally.

  33. @Steve Sailer
    Pittsfield, MA is home to the Norman Rockwell Museum (thanks to major donations from Steven Spielberg, by the way). It's in the Berkshires and seems to be a pretty nice place (I was there for a few hours on business in the 1980s and drove through last summer. It's near the summer home of the Boston Symphony.)

    The residents probably want to keep it static.

    Pittsfield, MA is home to the Norman Rockwell Museum.

    That’s actually in nearby Stockbridge, where he lived, but close enough. I hiked through on the Appalachian Trail back in ancient times…

  34. @Buzz Mohawk
    To make some money before starting college, I worked for six months as a lab tech at the R&D center for a major building products manufacturer. My job involved applying scientific testing methods to measure the properties of new formulas for things like fiberglass roofing mat, insulation and boards.

    I ripped things apart, burned them, and poked holes in them. I subjected them to extreme simulated climate changes over time in a programmable environmental chamber. I blew air through them in a wind tunnel. Then I calculated standard deviations and other things for my data. Engineers used my results to determine if their new formulas and products were good enough.

    This was applied science, engineering if you will. It was not science because I was not discovering anything applicable outside our specific cases.

    Your work in market research sounds the same as this, applied to people and advertising formulas.

    You were doing engineering!

    That sounds about right.

  35. Anon • Disclaimer says:

    Social Psychology also has the serious problem that only people on the extreme left are allowed into the field. So you really can’t do anything that would have any conservative implications, and nobody who is allowed through those PHD programs would pursue such research.

    Economics is a bit healthier because its always had some support from right wing individuals and foundations and therefore even the Krugmans have to answer criticisms from Tyler Cowen.

    • Replies: @Realist
    Economics is like reading tea leaves....just not as accurate.
  36. For those curious the first figure of that paper is ridiculous on its own. If the psychologists were honest there would need to be some introspection and major changes to the field to combat this. What’s especially damning was these were 100 studies from their top tier journal, and the original authors provided their instructions and assistance to the reproduces. And yet still these results were about as likely to be statistically significant as any random deep thought from a stoned psych freshman. Look at this:

    I’m sure in my field of biology that it is similarly bad, although I doubt quite as extreme. There are too many papers and too many journals for them all to be well conducted. I would start with that. I would hope that the key findings from the top labs in my field would be more then 50% reproducible. I bet they would be.

  37. Psychology could only dream about attaining the intellectual heights reached by marketing research. The basic concept of investigating what the world is like, rather than demonstrating it is the way you think it is already, is usually beyond the psychologist’s purview.

  38. @ScarletNumber

    graduate degrees in science, especially in psychology.
     
    I have a feeling that people with actual graduate degrees in science are insulted you included marketing in this.

    Psychology is not science

  39. @anonymous
    Marketing research, science or just divining fads? So why are there so many blacks featured in commercials, usually as all-wise types? Does having blacks in ads really sell well to the majority? It's hard to believe they are more persuasive than white actors would be in those roles. It's understood that the government forces them to hire blacks and to showcase them but they're way overrepresented well beyond meeting the minimum requirements.

    “Marketing research, science or just divining fads? So why are there so many blacks featured in commercials, usually as all-wise types?”

    This is something I have noticed as well. Also blacks are given alpha roles, in commercials, to wimpy whites.

  40. @Anon7
    University professors studying physics in the 18th and 19th centuries began uncovering "laws" of nature. The really neat thing about a "law" like "F=ma" is that you can solve an amazing variety of problems without having to look at nature again - all you need is the law! Just fill in the variables. Physicists started getting a lot of respect, because their laws made amazing things possible, like radio, cars and appliances. And, of course, they kept doing direct experiments, because it's the best way, even if you have solid laws.

    In the first part of the 20th century, medical doctors started discovering things that actually made medicine useful and desirable. Doctors started to get a lot of respect, because their uncovered "laws" started really saving lives (especially vaccines and antibiotics and all the surgeries that became possible).

    This was all good.

    But then, other departments in the university started glomming onto this idea, which we might call "science"; they used words like "force" or "law" or "experiment" and they too uncovered what they felt were "laws". They started to demand equal respect; a "doctor" of sociology was the equal of a doctor of medicine or a guy with a doctorate in physics. Their argument was that if one takes a "scientific" approach, one necessarily gets to truth as a result. Unfortunately, this turns out to be bullshit.

    This is where the problem started. Sociology, psychology and political science professors came up with laws like "Men and women are equal!" And once you have a law, you can solve all kinds of problems without ever having to check with reality.

    If you really want to start an argument with an academic, insist that some kinds of science are better than others. Like, "physics is the strongest science, because you can always get direct experimental verification whenever you want" or "sociology is a weak science, since the supposed results change constantly".

    Excellent points. But I would call sociology a pseudoscience.

  41. @Anon
    Social Psychology also has the serious problem that only people on the extreme left are allowed into the field. So you really can't do anything that would have any conservative implications, and nobody who is allowed through those PHD programs would pursue such research.

    Economics is a bit healthier because its always had some support from right wing individuals and foundations and therefore even the Krugmans have to answer criticisms from Tyler Cowen.

    Economics is like reading tea leaves….just not as accurate.

  42. Anonymous • Disclaimer says:
    @Anon7
    University professors studying physics in the 18th and 19th centuries began uncovering "laws" of nature. The really neat thing about a "law" like "F=ma" is that you can solve an amazing variety of problems without having to look at nature again - all you need is the law! Just fill in the variables. Physicists started getting a lot of respect, because their laws made amazing things possible, like radio, cars and appliances. And, of course, they kept doing direct experiments, because it's the best way, even if you have solid laws.

    In the first part of the 20th century, medical doctors started discovering things that actually made medicine useful and desirable. Doctors started to get a lot of respect, because their uncovered "laws" started really saving lives (especially vaccines and antibiotics and all the surgeries that became possible).

    This was all good.

    But then, other departments in the university started glomming onto this idea, which we might call "science"; they used words like "force" or "law" or "experiment" and they too uncovered what they felt were "laws". They started to demand equal respect; a "doctor" of sociology was the equal of a doctor of medicine or a guy with a doctorate in physics. Their argument was that if one takes a "scientific" approach, one necessarily gets to truth as a result. Unfortunately, this turns out to be bullshit.

    This is where the problem started. Sociology, psychology and political science professors came up with laws like "Men and women are equal!" And once you have a law, you can solve all kinds of problems without ever having to check with reality.

    If you really want to start an argument with an academic, insist that some kinds of science are better than others. Like, "physics is the strongest science, because you can always get direct experimental verification whenever you want" or "sociology is a weak science, since the supposed results change constantly".

    professors came up with laws like “Men and women are equal!”

    professor profess :”to take a vow” (in a religious order)
    doctor: …, from Medieval Latin doctor “religious teacher, adviser, scholar,”
    dean : … from Late Latin decanus “head of a group of 10 monks in a monastery,”
    rector : … ruler … by 18c. generally restricted to clergymen and college heads

    They were here first. Universities started as religious institutions, and now they returned to their roots.

  43. Ouch.

    As a psychologist this has been an interesting discussion to read. My own view would be that at the ‘rats and stats’ end of psychology (biologically-based) the science is good. Social Psychology made some terrible choices 40 years ago – as an undergraduate in the early 80’s, it was still a strong strand of Psychology, but the gradual move away from biological explanations (in line with the fashions of the time) undermined the field. I would be hard pressed to name a prominent social psychologist (i.e. who uses that label) now.

    True, the general profession has become extremely politically correct (as have most professions), but the sorts of psychology matters discussed by Steve Sailer would be reasonably generally understood – some younger psychologists would not have been taught, and some older ones prefer not to know, however. Even if most psychologists stay very quiet about these issues in public.

    There is, however, a general preference for good science, so it will be interesting to see how the profession responds, if the evidence keeps going in the direction it seems to be at present. Incidentally, I came to this site because of my interest in IQ – many years of testing has convinced me of the usefulness of this concept.

    • Replies: @hodag
    By virtue of his radio show, the only living social psychologist I am familiar with is Milt Rosenberg, emeritus at U of C. He is in his 90s and still does his radio show albeit at a small station in Evanston.
  44. I was about to say that of course psychology is a Science because I considered studying psychology when I was young. But then I also considered studying economics, and that’s a conclusive counter-example. Or, at least, all the economics that isn’t footnotes to Smith and Ricardo ain’t a science.

  45. @Steve Sailer
    In 1980, the marketing research firm that I worked for, on and off, from 1982-2000, bought checkout laser scanners for all the supermarkets in Pittsfield in return for getting their cooperation in providing us with data on the 2500 volunteers we recruited in town who we rewarded each time they identified themselves to the cashier so we could see what they had bought. On top of that, we gave each of the 2,500 volunteer households a special cable converter box allowing us to manipulate exactly which commercial they would see. So we could do scientific gold standard blind experiments on consumers of television marketing campaigns for consumer packaged goods.

    And we had seven other similar test markets at our peak.

    These were amazing real world laboratories, especially considering how long ago the early 1980s were.

    We used the best experiment principles devised by scientists to conduct our experiments. We employed quite a few Ph.D.s, whom we paid better than academia.

    But were the results we discovered Science?

    What you’re referring to from your work in the 1980s is simply the analogue version of Google’s entire Business Plan. And if that’s not “science” it is certainly “scientific”.

    Reproduction is the flip-side to “Is the conclusion falsifiable?” And it’s important to keep this in mind, especially when dealing with jumps from the specific to the general (ie induction).

    “63 out of 100 college students exhibited a loss of empathy when given the power to deliver electric shocks, therefore all humans tend to lose empathy when given power”.

    How can one refute this statement? And, look at the burden placed on potential opponents. If I’m Stanley Milgram I spent very little time performing stupid, useless “experiments”, but it would take an opponent damn near a lifetime to gather enough samples and re-tests to see if Milgram’s conclusions have any inductive force. And at the end of it all, we still have nothing definitive–only percentages.

    Social “science” has turned science upside-down where credulity and acceptance are easy and skepticism and doubt are difficult.

  46. So, having a currently popular celebrity endorse your product doesn’t increase sales? This myth sure does seem to persist.

    Methodology flaws indeed.

    Psychology studies have shown stepchildren have a significantly higher chance of drowning in the family pool than those who live with their natural parents. I’m going to predict that in the future, stepchildren will have a significantly higher chance of drowning in the family pool than those who live with their natural parents. I’m also going to predict that having a currently popular celebrity endorse your product will increase sales.

    While there is little doubt that the field of psychology is beset with hucksters and charlatans taking advantage of the widespread lack of understanding about psychology, summarily dismissing the entire field is absurd and asinine. It’s reminiscent of primitive, superstitious natives fearing automobiles because they don’t understand them.

  47. @Paolo
    Ouch.

    As a psychologist this has been an interesting discussion to read. My own view would be that at the 'rats and stats' end of psychology (biologically-based) the science is good. Social Psychology made some terrible choices 40 years ago - as an undergraduate in the early 80's, it was still a strong strand of Psychology, but the gradual move away from biological explanations (in line with the fashions of the time) undermined the field. I would be hard pressed to name a prominent social psychologist (i.e. who uses that label) now.

    True, the general profession has become extremely politically correct (as have most professions), but the sorts of psychology matters discussed by Steve Sailer would be reasonably generally understood - some younger psychologists would not have been taught, and some older ones prefer not to know, however. Even if most psychologists stay very quiet about these issues in public.

    There is, however, a general preference for good science, so it will be interesting to see how the profession responds, if the evidence keeps going in the direction it seems to be at present. Incidentally, I came to this site because of my interest in IQ - many years of testing has convinced me of the usefulness of this concept.

    By virtue of his radio show, the only living social psychologist I am familiar with is Milt Rosenberg, emeritus at U of C. He is in his 90s and still does his radio show albeit at a small station in Evanston.

  48. @anonymous
    Marketing research, science or just divining fads? So why are there so many blacks featured in commercials, usually as all-wise types? Does having blacks in ads really sell well to the majority? It's hard to believe they are more persuasive than white actors would be in those roles. It's understood that the government forces them to hire blacks and to showcase them but they're way overrepresented well beyond meeting the minimum requirements.

    “So why are there so many blacks featured in commercials, usually as all-wise types?”

    Because in the early 90s, the New York Times (!) got sued for not having any/enough black faces in their real estate ads. Everybody got the message.

  49. Interesting. That’s about what I figured was going on, after I read that popular response that went around (in the atlantic?) denying a crisis and calling it science, and noting that failure to replicate generally implies an imprecise hypothesis, not a useless experiment. And my first thought was, If psychology is testing under very particular conditions and requires more particular explanations, isn’t that by definition a pursuit of the trivial?

    I took the two intro courses to psychology with a great professor and a great textbook, and what I got out of it most of all, I think, and this was from the textbook, is that the only thing left to solve in psychology of interest to other than the hyper-specialist is the very strange and very real stuff of para-psychology.

    But then, the best stuff I’ve read says those explanations are tied up with quantum physics, and maybe a philosopher named Berkeley…

    BTW, Steve, I recall reading that excellent speech on your epistemology, WHICH is no longer available by the way.

  50. […] by some psych prof with her fingers in her ears. Steve Sailer, a former market researcher, wonders if psychology is more like astronomy, or marketing research? The latter only wants results now; the former (in a pretend quest to be physics) seeks Permanent […]

Comments are closed.

Subscribe to All Steve Sailer Comments via RSS