Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Which is more probable?
1. Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement.
Personally, after reading the above description, I have Linda in my mind’s eye, and I can just see her lecturing me on what sort of yoghurt I should eat. If I ever met her, I would not dream of admitting that I drive a diesel car, and that I have very recently taken up sketching nude women. Of course she is a feminist, and against nuclear weapons! That is obvious. (Actually, if Linda is very attractive, it might be worth my while telling her about my book against nuclear war).
“Linda” is the tricky question Kahneman and Tversky made famous. They implied that people who chose answer 2 were being irrational, because, wait for it, it is more likely from a statistical point of view that Linda is a bank teller (answer 1) than that she is a bank teller with a particular political interest (answer 2). This is because there will be at least some bank tellers who are not feminists, and even if there is only one such bank teller, then the category “bank teller and also feminist” will be smaller than the category “bank teller”. So, it is more likely that she is just a bank teller.
However, the introductory remarks have lead you into getting the sucker punch. The woman is SINGLE for God’s sake, despite being 31 years of age. Some problem there. Despite being a woman, she is OUTSPOKEN and VERY BRIGHT. She studied PHILOSOPHY which I can testify puts you on a hiding to nothing. She was DEEPLY CONCERNED with issues of DISCRIMINATION and SOCIAL JUSTICE. ANTI-NUCLEAR completes the picture. Answer 2 is the better match with the female of this species.
For many years some people have been asserting that tests of rationality show that IQ tests leave out an important aspect of mental ability. If you are clever, but screw up on the Linda Question, then that shows that you lack rationality. Like most things, this is an old debate, best presented by Peter Cathcart Wason, who started this horse running with his four card logic problem in 1966. I heard him lecture on it at The National Hospital, Queen Square, in 1968 or thereabouts, and he demurred when the neuro-surgeon introducing him said that he thought the problem “somehow unfair”. That comment caught the essence of the matter very well.
The other interpretation is that, far from being a test of rationality, whatever that is, the Linda Question is tricky, and teaches you more about the specific question form than about general human thinking processes. Question forms are interesting, and should be studied, particularly by those writing instruction manuals, government advice brochures, and examination questions. Confusing people is easy. Writing clearly and honestly is more difficult.
I have explained all this before, and will probably keep explaining it for ever, because many people, discouraged by clear evidence of their palpably limited intellects, would like a Get Out of Jail card from Multiple Intelligence, Emotional Intelligence, Practical Intelligence and now, Rational Intelligence.
As you will have seen if you read the second list, I wished the authors good luck in making their test, but was implying it would be difficult.
Stanovich, West and Toplak have now published their book: “The Rationality Quotient: Towards a test of rational thinking”. This is a “Towards” book, and we should not be too impatient, because scholarship takes time. However, it is a bit frustrating. The book gives an erudite account of the issues, but then only selected gems from the relevant work, much of it unpublished. What happened to the old-style test manuals? They had short introductions, a much longer section on standardization samples and general procedures, full instructions about how to administer and score each test, and then correlation matrixes, some factor analyses, and standard score conversion tables. You knew where you were with these manuals. You had to read them to understand what you were doing, and had to hang on to reliability and validity measures for dear life: without those the test was useless. Presumably, we will have to wait for that test manual. Daniel Kahneman, however has seen enough to aver that “it makes a compelling case for measuring rationality independently of intelligence”. Should we be compelled to that conclusion?
There is no a priori reason why a new test of rationality, or practical intelligence, or emotional intelligence or multiple intelligence should not supplant the current range of psychometric tests developed over the last 110 years. Improvements happen. All the new test has to show is that it is a better predictor of real life outcomes than the former test. In an instant, the old test is toast.
Why do the developers of CART, heavily trailed as reaching the reasoning zones that ordinary psychometric tests cannot reach, not bother with validity tests? I pondered this and decided that, prior to reading it, I would sketch out what validity tests would be required. I needed a real life measure of reason, and a strong motive for people to exercise their reason in attaining their goal. Finally, the answer struck me: most people work for a living, and most reasonable people save some money for long retirement. Would tests of rationality, I asked myself, be better predictors of investment strategies and outcomes than ordinary tests of intelligence?
No sooner do I pose the question than I find that Stuart Ritchie has already reviewed the book, and found out that the authors have not bothered with validity tests.
— Stuart Ritchie (@StuartJRitchie) January 11, 2017
So, I no longer have to review the book. Ritchie has done the job, calmly, kindly, and lethally. He has lanced the pretensions of the “rationality tests are better than intelligence tests” by showing they have not yet provided any validity data, and in terms of their association with intelligence tests, they correlate 0.7 Here is an illustrative table.
Naturally, the individual subtests vary considerably in their correlations with intelligence, but the final test result is closely linked with IQ. Indeed, from a psychometric viewpoint it is no more than a long-winded and very cumbersome intelligence subtest, and so it will remain, until validity data proves otherwise. However, these correlations are instructive, so make a mental note of the ones that most rely on intelligence, and then compare with my comments below.
Here are a few reflections as a post-script to Ritchie. If Rationality is a thing, it must hang together. Tests of rationality should correlate with each other because they have rationality in common. As a rule of thumb, subtest correlations should be better than 0.6, ideally 0.7 and above. Absent such correlations, the subtests would be no more than a hodge-podge of curiosities.
Furthermore, inspection of the correlations between the subtests shows that many of the subtests are apparently not testing the same mental skill of rationality. Correlations of 0.28 hardly suggest a common factor. I think these are disparate tests, corralled under a one-size-fits-all banner. Wechsler subtests look pristine by comparison. From a psychometric point of view the following subtests (ordered by correlations including, and then excluding, the subtest in the total score) seem promising:
1 Probabilistic reasoning .78 .71
2 Scientific reasoning .78 .70
3 Reflection versus Intuition .77 .71
14 Financial literacy .72 .65
4 Syllogistic reasoning .68 .62
13 Probabilistic numeracy .67 .62
Is it just me, or do these have an IQ-like look to them? In fact, the ones that correlate which each other are ones which require intelligence as shown in the correlations with cognitive ability listed above. This is something of a crisis for any test of rationality. If they pick the best 6 sub-tests and drop all the others, they could get the whole test down to manageable proportions and quick enough to use. Currently, the full form is far too long, but this is part of the test development process. Neither subjects or test givers like long tests. There are many quick IQ tests against which a rationality test will have to compete. However, how much Rationality will reside in such a test? It will be just another IQ test. To stick true to their own mission they should reject the list above and concentrate on those tests which do not correlate with intelligence. This will be the pure rationality they seek. The fact that those tests don’t correlate with each other is a problem they will have to solve.
Validity measures must be proffered for this new test, or the whole thing will be no more than a burst bubble, a party balloon that popped.
I think that Kahneman has been premature in concluding that we should measure rationality independently of intelligence. Currently it is no more than a cumbersome subtest, with no demonstrated advantages.
What I cannot refute is that many show a deep desire to supplant tests of intelligence. Far from doing so, this “Towards” book has inadvertently done a good job of showing that Rationality is very probably not separate from Intelligence, and that the tests of a presumed common factor of rationality are a very mixed bunch. I suspect they are mostly a collection of tricky questions organized into general types. Refined down to the best subtests, and further refined down to the most discriminating items, it might be possible to construct a test which clinicians and researchers will use. That remains to be demonstrated, and depends validity measures. It would be good to see if a Rationality Quotient was a better predictor of long-term investment performance than an IQ test.
Grant proposal, someone?