The Unz Review - Mobile
A Collection of Interesting, Important, and Controversial Perspectives Largely Excluded from the American Mainstream Media
Email This Page to Someone

 Remember My Information

 TeasersiSteve Blog

Bookmark Toggle AllToCAdd to LibraryRemove from Library • BShow CommentNext New CommentNext New Reply
🔊 Listen RSS

Countries in sub-Saharan Africa and the West Indies only occasionally participate in the big international school achievement tests, PISA and TIMSS. Fortunately, a number of countries in Eastern and Southern Africa have their own test: SACMEQ. (Here are the math scores for SACMEQ III in 2006-2011. SACMEQ IV doesn’t appear to have been released yet.)

Here’s a paper by Justin Sandefur comparing SACMEQ scores to TIMSS scores:

Internationally Comparable Mathematics Scores for Fourteen African Countries

Internationally comparable test scores play a central role in both research and policy debates on education. However, the main international testing regimes, such as PISA, TIMSS, or PIRLS, include very few low-income countries. For instance, most countries in Southern and Eastern Africa have opted instead for a regional assessment known as SACMEQ. This paper exploits an overlap between the SACMEQ and TIMSS tests—in both country coverage, and questions asked—to assesses the feasibility of constructing global learning metrics by equating regional and international scales. I compare three different equating methods and find that learning levels in this sample of African countries are consistently (a) low in absolute terms, with average pupils scoring below the fifth percentile for most developed economies; (b) significantly lower than predicted by African per capita GDP levels; and (c) converging slowly, if at all, to the rest of the world during the 2000s. While these broad patterns are robust, average performance in individual countries is quite sensitive to the method chosen to link scores. Creating test scores which are truly internationally comparable would be a global public good, requiring more concerted effort at the design stage.

Here’s Sandefur’s list of SACMEQ scores from the year 2000:

Mauritius 585
Kenya 562 (teachers 967)
Seychelles 554 (teachers 868)
Mozambique 530 (teachers 789)
Swaziland 518 (teachers 804)
Tanzania 517 (teachers 792)
Botswana 513 (teachers 751)
Uganda 506 (teachers 827)
South Africa 483
Zanzibar 476 (teachers 686)
Lesotho 448 (teachers 737)
Malawi 434 (teachers 770)
Zambia 432 (teachers 749)
Namibia 431 (teachers 731)

Sandefur writes:

Focusing exclusively on the set of common items administered to both pupils and teachers, pupils answered an average of 28% of items correctly, while teachers answered 73% correctly. Across countries, pupils’ percent correct ranged from just over 20% in Malawi and Zambia to 41% in Mauritius (the latter country did not administer the teacher test). Teacher scores ranged from just 58% in Zanzibar to 91% in Kenya.

Sandefur’s three methods of comparing SACMEQ to TIMSS scores suggest that the African test is inflated by 150-225 points or so. I think it’s perfectly reasonable for African countries to have an easier test that encourages them to try to improve to outdo their peers, just as Ivy League football teams measure themselves against Ivy League football teams rather than against, say, the U. of Alabama.

Random comments:

Mauritius isn’t really an African country, it’s more like an Indian Ocean version of Trinidad, with mixed South Asian, African, French, and East Asian populations.

Kenya’s students scored the best in 2000 and their teachers did quite well. Sandefur estimates that Kenyan teachers would have done better than world-leading Singaporean 15-year-olds on the TIMSS. Way back in 1981 I had a summer job involving Africa. Back then, Kenya’s GDP per capita was quite low by African standards. But most of what I read for the job that was of a qualitative nature implied that, GDP statistics be damned, Kenya, as Olympic medals and foreign tourism would suggest, was the best country in black Africa. Employing smart math teachers would seem to me to be a good measure of a country’s quality.

The Seychelles are similar to Mauritius demographically, with African, South Asian, European, and East Asian ancestries, although perhaps more African than Mauritius.

Both Mauritius and the Seychelles islands were uninhabited before Europeans arrived in the early 1500s, although it’s likely they were visited earlier. The flightless, easy-to-eat dodo bird lived in Mauritius so it’s safe to assume that no humans had made Mauritius their home for long previously.

Modern scientists say the dodo was probably about as smart as its relative, the pigeon. It’s just that the dodo lacked an instinct to fear newcomers. So by 1662 the dodo was extinct.

• Tags: PISA 
🔊 Listen RSS

PISA 2015 by Steve Sailer

Here are the overall 2015 PISA scores (averaging the Science, Reading, and Math scores equally), with color coding to put the various American scores (red bars) in perspective.

Keep in mind that some countries didn’t do a good job of rounding up everybody who was supposed to take the test, which probably serves to boost their scores slightly.

American whites went up 1 point since 2012′s PISA, while American Asians were down 31 points (roughly 3 tenths of a standard deviation). I don’t know why. In general, East Asian countries were down a little from 2012.

There were a number of big changes in the PISA from 2012 to 2015. I think they switched from paper to computer and made some other changes as well. So it’s probably best not to worry too much about score changes from 2012 to 2015.

You can read my earlier coverage of PISA results over the years here.

Data for this graph can be found here for countries and here for U.S. ethnic groups.

• Tags: PISA 
🔊 Listen RSS

Screenshot 2016-12-06 03.18.50

This table is sorted in order of scores on Science, the subject most emphasized in this 3 year period.

Here’s the big PDF of data.

Argentina for the win!

In the past, the Argentines always complained that they were doing a better job of rounding up their dimmest 15 year olds to take the international PISA exam sponsored by the OECD than some countries they could mention, cough Mexico cough, which was dragging their scores down relative to other countries, cough Vietnam cough. I don’t know what the Argentines did to better manipulate their sample this time, but it definitely worked.

Here’s a graph of the Science results (this was the year PISA emphasizes Science), with a column of what percentage of the expected 15 year olds weren’t tested:

pisa science 2

The US missed 16% of its expected coverage, which is kind of lousy. Not as bad as Vietnam (51% missing), Mexico (38%), or China (36%), but still …

Here are my posts about older PISA tests, such as 2012 and 2009.

India doesn’t take the PISA test. It tried it out on a couple of states a number of years ago and did very badly.

China is rolling the PISA out to more provinces. It did pretty well this time, but not as well as last time when it reported only the Shanghai results and came in first in the world. China’s coverage in those provinces is 3rd World level too (missing 36% of 15-year-olds). So, we’ll see…

In the past, I’ve broken out American scores by race: Asian Americans typically have done about as well as the wealthy Asian countries; white Americans do better than many European countries but not as well as Finland; Latino Americans do better than Latin countries (although Argentina boosted its scores enormously this time for as of yet unexplained reasons); African Americans do better than any African countries and most West Indian countries that bother to take the PISA test (although Trinidad is climbing). The US government usually posts that data buried in documents here.

Screenshot 2016-12-06 05.55.02

So, in Science, white Americans (531) scored like Finland, black Americans (433) scored about a standard deviation lower like Cyprus, Hispanic Americans (470) scored between Iceland and Israel, Asian Americans (525) scored like Vietnam (although Vietnam’s score is likely inflated by dropping low scorers from having to take the test), and Multiracial Americans (503) score like Ireland.

Similarly, here’s my graph of the 2012 results (averaging the three subjects equally), showing American groups (red) in racial context:

Note, this colorful graph is for 2012, not for the new 2015 results. Also, they made some scaling changes between 2012 and 2015, so the scores from this 2012 graph aren’t exactly comparable to the scores from 2015.

One question I’ve never seen an answer to is how hard different countries try on this low-stakes test and how much does it matter. One past report included a notation that Austria’s score that year was probably artificially low because teachers were striking and may have told students to do badly on the test to stick it to management. Other countries might give rousing pep talks. Who knows?

Here’s Anatoly Karlin’s comments on the PISA results at the Unz Review.

• Tags: PISA 
🔊 Listen RSS

From The Spectator:

Immigration helps explain Sweden’s school trouble
Gabriel Heller Sahlgren 10 August 2015 17:56

Sweden’s education performance has faltered in the past decades, with scores tumbling in the OECD’s international Pisa survey since the early 2000s. Both the Guardian and the BBC have recently looked into this phenomenon.

I’m a fan of the giant international PISA tests, but I’m not a huge fan of trusting minor differences between countries or differences over time in one country to support whatever your favorite theory might be. What goes into influencing PISA scores is extremely complicated and murky, and I’m far from convinced that things as ephemeral as, say, how big of a pep rally they hold for the students taking the test don’t matter.

There’s no shortage of explanations for the poor results, but British enemies of school reform have latched on to one of them: the free schools reform of the early 1990s, which they claim sent the system into chaos.

You’ll notice how the naysayers never give any evidence to back up their claims. …

Why, then, are Swedish schools struggling? There are many hypotheses floating around, some more plausible than others. Yet there has been no hard evidence for any of them. Until now.

On the other hand, there are often ways to perform all-else-being-equal analyses on PISA scores. For example, I’ve long pointed out that although America’s PISA scores are notoriously mediocre overall, on a race-by-race basis they’re closer to Not Bad.

Similarly, the PISA tests break out the scores of first and second generation immigrants, allowing the following type of quite legitimate analysis to be done:

Two weeks ago, I published a paper [in Swedish] at the Research Institute of Industrial Economics and an op-ed in the largest Swedish daily about the impact of immigration on Sweden’s Pisa scores. Quite a row followed. Why? Well, it turns out that the change in pupil demographics due to immigration explains almost a third of the average decline between 2000 and 2012: 19 per cent in mathematical literacy, 28 per cent in reading literacy, and 41 per cent in scientific literacy. The effect is especially pronounced in recent years, coinciding with accelerating refugee immigration. Indeed, between 2009 and 2012, 43 per cent of the average Pisa score decline is explained by altered demographics: a full 29 per cent in mathematical literacy, 45 per cent in reading literacy, and 62 per cent in scientific literacy.

In the paper, I also calculate that 11 per cent of the overall decline can be attributed directly to the decreasing share of pupils with a Swedish background, while 18 per cent can be attributed to the fact that results among pupils with an immigrant background have fallen more significantly. The latter is clearly displayed by the next graph. While scores among pupils with a Swedish background have declined by 22 points between 2000 and 2012, scores among pupils with an immigrant background have fallen by 39 points – and the performance gap between the two groups has grown from 36 points to 53 points.

These are strong effects. In fact, the change in pupil demographics is the only factor that we know for sure has contributed to Sweden’s falling scores. Furthermore, the full impact is probably somewhat larger. This is because immigration may also have lowered performance among pupils with a Swedish background, for instance through the redistribution of resources to immigrant pupils. While this is a more difficult subject to study, the most relevant paper suggests modest negative effects in this respect. It’s therefore probable that the total impact of changed pupil composition is higher than reported here.

Even with this effect included, immigration is nevertheless unlikely to be the only reason behind Sweden’s falling educational performance. While the left’s arguments have been debunked, there are more plausible explanations, explored elsewhere by myself and my colleague Dr Tino Sanandaji. In short: the rise of an overly progressive educational culture is likely to be an important culprit. Incidentally, as my recent monograph shows, this is also relevant for understanding declining Pisa scores in Finland, the longstanding pin-up model of school choice and accountability critics.

• Tags: PISA 
🔊 Listen RSS

With Puerto Rico in the news for threatening to go broke, Paul Krugman is worried that hedge funds want to “destroy the island’s education system in the name of fiscal responsibility.” But it turns out that Puerto Rican school administrators have largely done that already. Although Puerto Rico spends more per public school student than Utah and Idaho, and more per student on certain obscure but lucrative categories of school administration than any of the 50 states, its test scores are horrific.

It’s interesting to discover that Puerto Rico actually participated — on a preliminary basis — in the 2012 international PISA test of 15-year-olds, although PR’s scores were not released in 2013 like most places’ were.

Instead, a report on Puerto Rico’s PISA scores surfaced a half year later. This is similar to how quietly Puerto Rico’s scores on the federal NAEP test were released.

The PISA is scored like the SAT with an intended average (in wealthy OECD countries) of 500 and a standard deviation of 100.

As on the NAEP, Puerto Rico did really bad on the Math portion of the PISA, worse than Jordan, and better than only Colombia, Qatar, Indonesia, and Peru. (No doubt there are countries that would score worse than Peru, but they don’t participate in PISA.)

Puerto Rico averaged a 382, 99 points or about a standard deviation behind the U.S.’s 481.

Screenshot 2015-08-02 21.17.11

On PISA’s 0 to 6 scale of proficiency in math, 34% of Puerto Rican students scored at the lowest (0) level, and not enough students out of the sample size of 1668 scored at any of the three highest levels (4 > 544, 5>607, or 6>669) to report a statistically reliable percentage.

I’m estimating 0.8% scored in any of the top three ranges. In contrast, almost 1/4th of U.S. students scored at the 4-6 levels.

Public schools in Puerto Rico are notoriously ineffectual and corruptly administered, so 23% of all students in the commonwealth are sent to private schools. The PISA test included private school students in its sample, however.

More math score details from the report:

The U.S. Hispanic average score (455) was significantly higher than the Puerto Rico average score (382) by 73 scale score points.

The U.S. Hispanic female average score (450) was significantly higher than the Puerto Rico female average score (377) by 73 scale score points. Similarly, the U.S. Hispanic male average score (460) was significantly higher than the Puerto Rico male average score (387) by scale score points.

The U.S. Hispanic public school student average score (455) was significantly higher than the Puerto Rico public school student average score (370) by 85 scale score points. The U.S. Hispanic private school student average score (477) was significantly higher than the Puerto Rico private school student average score (424) by 53 scale score points.

Puerto Rico’s 10th percentile did badly, of course, but not superbad relative to other backwards places. Puerto Rico’s 90th percentile, however, scored as badly as the 90th percentile in any country.

This phenomenon of the Apathetic Elite seems more common in Latin America than most other places. For example, Mexico and Turkey are fairly similar overall, except that the really smart kids in Turkey, while relatively few in number, are still really smart, while Mexico just doesn’t seem to have much of a high end at all.

But Puerto Rico’s 90th percentile is way below even Mexico’s 90th percentile. My guess is that the high end Puerto Rican families get out of Puerto Rico, but that’s not particularly true for high end Mexicans, so I don’t know.

In defense of Puerto Rico, however, the test administrators rounded up a quite reasonable 91% of the 15-year-olds who were supposed to take the test, which is comparable to the coverage in the U.S., although not as good as in Finland or Netherlands. In contrast, Mexico somehow or other lost almost half of the youths who were supposed to take the test, and Costa Rica skimmed the cream even harder.

(Argentina’s miserable performance is related in part to the test administrators conscientiously rounding up about 4/5ths of the eligible youths. Conversely, Shanghai’s stratospheric scores may be related to test administrators not being all that diligent about rounding up the city’s huge population of children of proletarian migrants without legal permission to reside in Shanghai.)

Puerto Rico did slightly less awful on Science, scoring 401, 96 points behind the U.S. Two percent of Puerto Ricans scored at Level 4 on the 0 to 6 scale, versus 26% of Americans scoring 4, 5, or 6.

Puerto Rico did best in reading, scoring 404, which is 94 points behind the U.S.

• Tags: Education, NAEP, PISA, Puerto Rico 
🔊 Listen RSS

From the NYT:

Closing Education Gap Will Lift Economy, a Study Finds

Study after study has shown a yawning educational achievement gap between the poorest and wealthiest children in America. But what does this gap costs in terms of lost economic growth and tax revenue?

That’s what researchers at the Washington Center for Equitable Growth set out to discover in a new study that concluded the United States could ultimately enrich everybody by improving educational performance for the typical student.

When it comes to math and science scores, the United States lags most of the other 33 advanced industrialized countries that make up the Organization for Economic Cooperation and Development, ranking 24th, far behind Korea, Poland and Slovenia.

Moving up just a few notches to 19th — so that the average American score matched the O.E.C.D. average — would add 1.7 percent to the nation’s gross domestic product over the next 35 years, according to estimates by the Washington Center, a nonpartisan, liberal-leaning research group focused on narrowing inequality. That could lead to roughly $900 billion in higher government revenue, more than making up for the cost of such an effort.

White Americans outscored the OECD average on the 2012 PISA 518 to 497.

If Americans were able to match the scores reached in Canada, which ranks seventh on the O.E.C.D. scale, the United States’ gross domestic product would rise by an additional 6.7 percent, a cumulative increase of $10 trillion (after taking inflation into account) by the year 2050, the report estimated.

White Americans trailed Canadians by only 4 points. White and Asian Americans combined probably about equaled Canada.

Here’s Anatoly Karlin on 2009 PISA scores for immigrants (first and second generation) versus natives. Canada is much more selective than the U.S. and thus benefits from smarter average immigrants.

The words “immigration” or “immigrant” are not mentioned in the New York Times article.

Robert G. Lynch, an economist who wrote the Washington Center report, explained why he took the trouble to make these what-if calculations.

… One point of this exercise, Mr. Lynch explained, is to show that the added cost of improving educational achievement at the bottom would be more than made up for by the rise in economic output and tax revenue.

The study used math and science scores from the 2012 Program for International Student Assessment, a test widely used around the world for measuring and comparing educational achievement. The average combined score for the United States is 978, while the O.E.C.D. average is 995. The Canadian average is 1,044.

Eliminating the achievement gap in America would require raising the country’s average to 1,080, so that it would rank third behind South Korea (with an average score of 1,092) and Japan (with a 1,083 average). That stunning improvement, according to the center, would raise the total output in the United States by another 10 percent. Lifetime earnings of the poorest quarter would jump by 22 percent in this event.

As I’ve been saying for a long time, the conventional strategy of Closing the Gap by raising the bottom half’s performance by roughly a standard deviation while somehow not allowing the upper half to improve seems like a worse idea than my suggestion that our goal should be to raise everybody’s performance by half a standard deviation.

After all, the reality of diminishing marginal returns suggests that improving a half standard deviation is considerably less than half as difficult and more than twice as plausible as improving a full standard deviation. Moreover, my goal works under various combinations of the importance of nature and nurture, while the conventional wisdom’s goal is only possible if nurture is all important and nature is meaningless.

• Tags: PISA 
🔊 Listen RSS
Because Finland scores well on PISA tests, there has been much interest in that remote Northern land’s rather laidback public education system. But that raises a problem: Finland hasn’t been very diverse until recently despite having a huge border with a much poorer country (secret: land mines). So, many accounts of Finland’s education system in America simply assert that immigrants do great in Finland. Only problem: not true. From a Google Translate version of a Finnish government account of the results of the latest PISA test, this one on “problem-solving:”

Immigrants fared poorly compared to the native population differences between the native population and the migrant pupils’ problem-solving skills were high in all the participating countries. In Finland, the main population, representing the students ‘scores averaged 526 points, while second-generation immigrant students’ backgrounds ¬ an average of 461 points and a first-generation 426 points. In Finland, migrant and native population, the difference between success was greater than in the participating countries on average.


• Tags: PISA 
🔊 Listen RSS
The tireless PISA folks are back with the results of a test of math-related real world problem solving among 15 year olds in 44 upscale countries. (Check here for sample questions like how to find the quickest route on a map or how to adjust an air conditioner). The U.S. did not bad, scoring a little above the average for rich countries, but not as good as the Asians or the white countries with smart immigration policies (Canada, Australia, Finland). 

OECD average 500
Singapore 562
Korea 561
Japan 552
Macao-China 540
Hong Kong-China 540
Shanghai-China 536
Chinese Taipei 534
Canada 526
Australia 523
Finland 523
England (United Kingdom) 517
Estonia 515
France 511
Netherlands 511
Italy 510
Czech Republic 509
Germany 509
United States 508
Belgium 508
Austria 506
Norway 503
Ireland 498
Denmark 497
Portugal 494
Sweden 491
Russian Federation 489
Slovak Republic 483
Poland 481
Spain 477
Slovenia 476
Serbia 473
Croatia 466
Hungary 459
Turkey 454
Israel 454
Chile 448
Cyprus1, 2 445
Brazil 428
Malaysia 422
United Arab Emirates 411
Montenegro 407
Uruguay 403
Bulgaria 402
Colombia 399

Shanghai came down to earth after its stratospheric scores on the last two PISAs. Poland was also down v. its PISA scores. Otherwise, there would appear to be a fairly high degree of correlation at the national level between the triennial PISA test of book smarts and the new PISA test of real world smarts, which is what the g Factor theory of intelligence would predict.
(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
Click to enlarge

In the 2010 paper The High Cost of Low Educational Performance: THE LONG-RUN ECONOMIC IMPACT OF IMPROVING PISA OUTCOMES, Stanford economist Eric A. Hanushek and Ludger Woessmann take a pretty scientific SWAG at what would be the economic benefits of your country enjoying the higher cognitive skills associated with higher PISA scores. They mull over the results of a series of mostly forgotten international math and science exams going back to the 1960 and find a surprisingly high, probably causal relationship between having a cognitively gifted student body in the past and an economically productive one today. 

This report uses recent economic modelling to relate cognitive skills – as measured by PISA and other international instruments – to economic growth. This relationship indicates that relatively small improvements in the skills of a nation’s labour force can have very large impacts on future well-being. Moreover, the gains, put in terms of current GDP, far outstrip today’s value of the short-run business-cycle management. This is not to say that efforts should not be directed at issues of economic recession, but it is to say that the long-run issues should not be neglected. 

A modest goal of having all OECD countries boost their average PISA scores by 25 points over the next 20 years – which is less than the most rapidly improving education system in the OECD, Poland, achieved between 2000 and 2006 alone – implies an aggregate gain of OECD GDP of USD 115 trillion over the lifetime of the generation born in 2010 (as evaluated at the start of reform in terms of real present value of future improvements in GDP) (Figure 1).

A number of years ago, I suggested that the American Establishment drop its obsession with Closing the Gap — in effect, boosting black and Hispanic scores by close to a standard deviation while not allowing whites and Asians to improve — in favor of a fairer and far more feasible goal of improving all groups by an average of a half standard deviation. Hanushek looks at just boosting everybody by a quarter of a standard deviation on test scores (25 points on a PISA test or on a SAT test) and finds the net present value for the U.S. would be $20 trillion.

Bringing all countries up to the average performance of Finland, OECD’s best performing education system in PISA, would result in gains in the order of USD 260 trillion (Figure 4). The report also shows that it is the quality of learning outcomes, not the length of schooling, which makes the difference.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS

The nice people at provide a free tool for making a “heat map,” like this one I made of average overall 2012 PISA scores. Click on the map to enlarge it.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
Three years ago, Andreas Schleicher and the other well-funded folks at PISA were media darlings. This year … not so much. You can sense that the bloom is off the rose. 

A big part of PISA’s new PR problem is that the results were so similar from 2009 to 2012. Now, you might think that stability is a good sign that suggests that the PISA people aren’t just pulling these numbers out of thin air. But accuracy is boring. The media likes change for the sake of change. Who’s up? Who’s down? A school test that’s more or less a giant budget IQ test doesn’t produce enough random changes to maintain media interest.

Decades ago when the news magazine US News & World Report was launching their college ranking system, there was much interest from year to year as they improved their methodology, frequently casting overlooked colleges toward the top. But, after awhile, USNWR got pretty good at measuring as much as could be conveniently measured … and then what? Colleges, it turns out, don’t change much from year to year, so the future looked a lot like the present. And without trends, we don’t have news. 

So, USNWR came up with the idea of changing some of the fairly arbitrary weights in its formula each year to generate a new #1 frequently. One year, for example, Caltech shot up to #1, which generated a lot of press coverage. But it was almost all just churn for the sake of churn. Caltech was pretty much the same place before, during, and after its sudden rise and fall.

But spectators like churn. In fact, one side effect of bad quantitative methodologies is that they generate phantom churn, which keeps customers interested. For instance, the marketing research company I worked for made two massive breakthroughs in the 1980s to dramatically more accurate methodologies in the consumer packaged goods sector. Before we put to use checkout scanner data, market research companies were reporting a lot of Kentucky windage. In contrast, we reported actual sales in vast detail. Clients were wildly excited … for a few years. And then they got kind of bored. 

You see, our competitors had previously reported all sorts of exciting stuff to clients: For example, back in the 1970s they’d say: of the two new commercials you are considering, our proprietary methodology demonstrates that Commercial A will increase sales by 30% while Commercial B will decrease sales by 20%. 


We’d report in the 1980s: In a one year test of identically matched panels of 5,000 households in Eau Claire and Pittsfield, neither new commercial A nor B was associated with a statistically significant increase in sales of Charmin versus the matched control group that saw the same old Mr. Whipple commercial you’ve been showing for five years. If you don’t believe us, we’ll send you all the data tapes and you can look for yourselves.


It was pretty amazing that we could turn the real world into a giant laboratory (and this was 30 years ago). But after a few years, all this accuracy and realism got boring. 

It turned out that clients kind of liked it back in the bad old days when market research firms held a wet finger up to the breeze and from that divined that their client was a creative genius whose new ad would revolutionize the toilet paper business forever. (New ads and bigger budgets mostly work only if your ad has some actual message of value to the consumers to convey: e.g., “Crest now comes with Unobtanium, which the American Dental Association endorses for fighting Tooth Scuzz.”)

These parallels between the consumer packaged goods industry in the 1980s and the educational reform industry in the 2010s are not really coincidental. Everybody says they want better tests, but what they really want is more congenial results. So, when they get better tests, they aren’t as happy as they thought they’d be.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
How can you be confident that local officials didn’t pull any fast ones with their PISA results? Well, you can’t, but you can get some sense of how much room there is to pull the wool over your eyes by looking at the response rate. 

Large countries have to test at least 4,500 students, and the sample is supposed to be carefully designed to represent the entire country’s 15-year-olds. But projected coverage usually turns out less than perfect. For example, countries can exclude students with disabilities. This sounds reasonable — it’s hard for a blind person to take a pencil and paper test. But, what about cognitive disabilities, such as not being very bright? From the federal government’s website on PISA:

PISA 2012 is designed to be as inclusive as possible. The guidelines allowed schools to be excluded for approved reasons (for example, schools in remote regions, very small schools, or special education schools). Schools used the following international guidelines on student exclusions: 

Students with functional disabilities. These were students with a moderate to severe permanent physical disability such that they cannot perform in the PISA testing environment. 

Students with intellectual disabilities. These were students with a mental or emotional disability and who have been tested as cognitively delayed or who are considered in the professional opinion of qualified staff to be cognitively delayed such that they cannot perform in the PISA testing environment. 

Students with insufficient language experience. These were students who meet the three criteria of not being native speakers in the assessment language, having limited proficiency in the assessment language, and having less than 1 year of instruction in the assessment language. 

Overall estimated exclusions (including both school and student exclusions) were to be under 5 percent of the PISA target population.

Buried in a PISA appendix entitled Annex 2A are PISA figures for what percentage of the target populations of 15-year-olds didn’t get tested. America didn’t come close to getting 95% representation, and many Third World countries were far worse.

“Coverage Index 3: Coverage of 15-year-old population” shows what percentage of the cohort are represented if the test taking sample was projected to the whole country. I subtracted this percentage from 100% to come up with the % Missing index. For example, Costa Rica only managed to test half the people they were supposed to, and Albania only tested 55%. Vietnam, which made a splashy PISA debut with high scores, somehow couldn’t find 44% of their 15-year-olds. At the other end, the dutiful Dutch managed to test slightly more students than were thought to be around.

% Missing
Costa Rica 50%
Albania 45%
Vietnam 44%
Mexico 37%
Colombia 37%
Indonesia 37%
Turkey 32%
Brazil 31%
Thailand 28%
Peru 28%
Uruguay 27%
Liechtenstein 25%
Bulgaria 23%
Shanghai-China 21%
Malaysia 21%
Argentina 20%
Kazakhstan 19%
Macao-China 19%
Hungary 18%
United Arab Emirates  17%
Canada 17%
Chile 17%
Hong Kong-China 16%
Czech Republic 15%
Serbia 15%
Latvia 15%
Lithuania 14%
Jordan 14%
Australia 14%
Italy 14%
Greece 13%
New Zealand 12%
Korea 12%
Austria 12%
Portugal 12%
Spain 12%
France 12%
United States 11%
Chinese Taipei  11%
Poland 11%
Luxembourg 11%
Montenegro 10%
Israel 9%
Denmark 9%
Japan 9%
Ireland 9%
Slovak Republic 9%
Tunisia 9%
Switzerland 9%
Norway 8%
Estonia 8%
Russian Federation 8%
Iceland 7%
Sweden 7%
United Kingdom 7%
Slovenia 6%
Qatar 6%
Croatia 6%
Germany 5%
Singapore 5%
Belgium 5%
Finland 4%
Romania 4%
Cyprus 3%
Netherlands -1%
In general, Third World countries were bad at getting good coverage, suggesting that the First World v. Third World gap is even larger than the test scores imply.

Top scorer Shanghai missed 21%, so we should take its flashy scores with a few grains of salt.

America was at 11% missing, down from 18% missing in 2009, which may account for the slight decline in U.S. scores?

Consistent high-flier Finland had only 4% missing, so they aren’t cheating on this measure more than the competition is.

A major question is how random were the missing test-takers. If the missing were purely random, then no harm no foul. But of course, many of the missing are dropouts, or in special day classes, or in juvy hall, or whatever.

This may help excuse slightly Argentina’s horrible scores. The Argentineans misplaced only 20% of their 15-year-olds compared to the 37% of Mexicans who went missing.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
Via Staffan’s Personality Blog, here’s an article from a Swedish (ahem, sore loser, ahem) newspaper accusing PISA of using fabricated data from Slovenia, Italy, and the United Arab Emirates. The charges don’t involve students, but high school principals. The principals were supposed to fill in a 184 question survey for the Nosey Parkers at PISA, but there is evidence that dozens of principals just cut and pasted somebody else’s answers, which wouldn’t be hugely surprising with a survey that is 184 questions long.

A general problem with comparing results of countries in international tests are differing levels of motivation. It’s remarkable how plausible the PISA results are in general considering how much this factor is likely to vary from place to place and time to time.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
Commenter Power Child notes:

“We’ll fix it in post” are known to production guys as the five most expensive words in filmmaking.

“We’ll fix it in post” is also the reasoning behind an awful lot of government spending on education, welfare, medicine, prisons, and many other Gaps caused by lack of care upfront in the production of residents of America.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
The PISA test was given to large samples sizes in three American states. From the federal National Center for Educational Statistics:

PISA 2012
Race / Ethnicity Mean Math Science Reading
White 538 530 545 540
Black 467 458 466 476
Hispanic 460 446 460 475
Asian 578 569 580 584
Multiracial NA NA NA NA
White 542 534 547 546
Black 434 421 433 447
Hispanic 456 442 463 463
Asian 548 534 553 558
Multiracial 512 496 520 521
White 512 499 520 518
Black 429 413 425 449
Hispanic 474 458 475 489
Multiracial 486 467 500 492
White-Black Gaps
Massachusetts 72 72 79 64
Connecticut 109 113 114 99
Florida 83 86 95 69
White-Hispanic Gaps
Massachusetts 78 84 85 65
Connecticut 86 92 84 83
Florida 38 41 45 29

The standard deviation is supposed to be 100, so you can just put a decimal place in front of those gap numbers to convert them into rough z scores.

We can see patterns here that shouldn’t be unexpected. Massachusetts, home to the education-industrial complex since 1636, has smart whites. Connecticut, home to the hedge fund industry, has smart whites.

Florida, not so much. Still, this would be a good time for an old anecdote about how Florida isn’t wall-to-wall Parrot Heads. I had a girlfriend in college who went to the public high school in Cocoa Beach, FL (the town that was the setting for the 1960s sit-com I Dream of Jeannie). She told me she scored 1580 on the SAT (M+V, old-style). I exclaimed:

“You must have had the highest score in your high school!”

“Oh, no, I was fourth-highest.”

“Fourth? Who were the other three? The children of rocket scientists?” (In my defense, this was a relatively new witticism in 1979.)


Massachusetts has pretty smart blacks, going back to Phillis Wheatley and W.E. Du Bois. Connecticut and Florida, not so much.

Florida has pretty smart Hispanics, although the wealthy Cubans and other rich Latin Americans are getting diluted more and more.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
From the “Country Note” for the United States from PISA:

Students in the United States have particular weaknesses in performing mathematics tasks with higher cognitive demands, such as taking real-world situations, translating them into mathematical terms, and interpreting mathematical aspects in real-world problems. An alignment study between the Common Core State Standards for Mathematics and PISA suggests that a successful implementation of the Common Core Standards would yield significant performance gains also in PISA.

The key phrase there is “a successful implementation.”

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
How can PISA claim to fairly test in 65 countries in dozens of languages?

My vague hunch is that modern Item Response Theory testing, of which the PISA test’s Rasch Model is an example, allows testers to say, much like movie directors of sloppy productions: “We’ll fix it in Post.” 

You tell me that during the big, expensive action scene I just shot, the leading man’s fly was open and in the distant background two homeless guys got into a highly distracting shoving match? And you want to know whether we should do another take, even though we’d have to pay overtime to 125 people? 

“Eh, we’ll fix it in Post.”

Modern filmmakers have a lot of digital tricks up their sleeves for rescuing scenes, just as modern psychometricians have a lot of computing power available to rescue tests they’ve already given.

For example, how can the PISA people be sure ahead of time that their Portuguese translations are just as accurate as their Spanish translations? 

Well, that’s expensive to do and raises security problems. But, when they see the results come in, they can notice that, say, smart kids in both Brazil and Portugal who scored high overall, did no better on Question 11 than kids who don’t score well on the other questions, which suggests the translation of Question 11 might be ambiguous. Oh, yeah, there are, now that we think about it, two legitimately right answers to Question 11 in the Portuguese translation. So we’ll drop #11 from the scoring in those two countries. But, in the Spanish-speaking countries, this anomaly doesn’t show up in the results, so maybe we’ll count Question 11 for those countries.

This kind of post-hoc flexibility allows PISA to wring a lot out of their data. On the other hand, it’s also a little scary. 

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS
From Globes, an Israeli business publication:

The PISA exam shows substantial gaps between Hebrew and Arabic-speaking pupils. In the math exam, Hebrew speakers achieved a score of 489 points, while Arabic speakers achieved a score of 388 points. Arabic speakers scored 98 points less than Hebrew speakers in the science exam.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
🔊 Listen RSS


This graph displays the mean of the Math, Science, and Reading test scores from the OECD’s 2012 Programme for International Student Assessment. American scores are red, white countries are blue, East Asians countries are yellow, Muslim countries are green, and Latin American countries are brown.So, Asian Americans outscored all large Asian countries (with the exception of three rich cities); white Americans outperformed most, but not all, traditionally white countries; and Latino Americans did better than all Latin American countries. African Americans almost certainly scored higher than any black majority country would have performed.

Bear in mind that many countries did not take part in PISA, such as India, which dropped out after a trial run in two states produced average scores below any seen on this chart. For a broader sampling of Third World scores, see the 2011 TIMSS Math and Science scores.

The reality is that there is not much difference in PISA or TIMSS scores within major racial blocs of countries. The Northeast Asians all tend to score well, the European and white Anglosphere countries tend to score fairly well, the Latin American countries tend to score fair to middling, and on down from there. The rank order of continents is very much like the rank order of racial/ethnic groups on NAEP or SAT or CST tests. Newcomers to the topic like Amanda Ripley, author of The Smartest Kids in the World, get excited about minor differences in PISA scores within continents, but those often are statistical noise.

For more on how to think about PISA scores, see here. And all my postings on PISA are here.
(Republished from iSteve by permission of author or representative)
🔊 Listen RSS
From my new column at Taki’s Magazine:

PISA, Piece by Piece 

by Steve Sailer  

With the release of new PISA test scores for 65 countries’ 15-year-olds this week, it’s worth taking a look at TIME reporter Amanda Ripley’s latest book The Smartest Kids in the World: And How They Got that Way

Ripley came up with the clever idea of following three American high schoolers as exchange students in Finland, South Korea, and Poland. She chose Finland and South Korea because they are perennial PISA powerhouses, while Poland has improved its ranking significantly in this century. 

Her sample size of three American kids abroad is hardly foolproof, and yet it’s a start. Everybody has opinions on schooling, but few people have firsthand experience with different countries’ school systems because it’s immensely time-consuming to sit in on classrooms long enough that the teacher runs out of her dog-and-pony shows for visitors and finally gets down to normal business. 

Having only recently become interested in the topic of education, Ripley is a true believer in PISA scores. 

Should you be? In truth, nobody seems to really know how much to trust PISA and its ace salesman Andreas Schleicher. … The sheer logistical challenge of what PISA attempts to do should raise common-sense questions about how perfectly 65 countries can be compared. Translation of tests, selection of representative samples, and prevention of local authorities putting their thumbs on the scale are challenges so daunting to get exactly equal around the world that most observers just seem to hope for the best and trust that Schleicher has somehow devised a globally level playing field.

Please read the whole thing there.

(Republished from iSteve by permission of author or representative)
• Tags: PISA 
Steve Sailer
About Steve Sailer

Steve Sailer is a journalist, movie critic for Taki's Magazine, columnist, and founder of the Human Biodiversity discussion group for top scientists and public intellectuals.

The evidence is clear — but often ignored
The unspoken statistical reality of urban crime over the last quarter century.
The “war hero” candidate buried information about POWs left behind in Vietnam.
The major media overlooked Communist spies and Madoff’s fraud. What are they missing today?
What Was John McCain's True Wartime Record in Vietnam?