The Unz Review - Mobile
A Collection of Interesting, Important, and Controversial Perspectives Largely Excluded from the American Mainstream Media
 Russian Reaction BlogTeasers
TIMSS vs. PISA Performance
Search Text Case Sensitive  Exact Words  Include Comments

The commenter “m” did some calculations to work out the relative performance of different countries in PISA vs. TIMSS, and in Math vs. Science.

pca-timss-pisa

m writes:

Dimension 1 is overall performance across all 4 (PISA Math, PISA Science, TIMMS Math, TIMMS science). Everything goes up with this dimension. Highest performers: Singapore, Chinese Taipei, Japan. Weakest performers: Turkey, UAE, Malta.

Dimension 2 separates stronger performers on TIMMS vs PISA: strongest performers on TIMMS relative to PISA are in order: Turkey, Korea, Russia, Hungary, United Arab Emirates, while strongest performers on PISA relative to TIMMS are: New Zealand, Canada, Australia, Norway, Italy. The five most balanced countries in the tradeoff are roughly: Slovenia, England, Hong Kong, Japan, USA.

Overperformance in TIMSS relative to PISA can arguably be used as a proxy for schooling quality, since it’s more dependent on academic/curricular skills than on raw intelligence. I am not surprised by the good figures for Korea, Russia, and to a lesser extent, the rest of East Asia and the post-Communist world. However, the UAE and Turkey are surprising.

Dimension 3 separates out Science nations vs Math nations: Most heavily Science vs Math: Slovenia, England, USA, New Zealand, Turkey and most heavily Math vs Science: Hong Kong, Malta, Korea, Italy, Norway.

As much as including TIMMS might be a worse proxy of “IQ” than just PISA, I have included in the above graphic a measure of using the PC1 overall performance score to convert to IQ, based on the assumption that England is 100 and Japan 104.3 as in your PISA conversion. There’s a bit of swing, not too much, compared to PISA alone.

m then extended his analysis to encompass Reading, which is unsurprisingly “less correlated with the other measures”:

pca-reading

as well as to the PIAAC Survey of Adult Skills:

pca-piaac

Comment:

Singapore’s the biggest relative loser when the skills measure is rolled in as well, with the least advantage on PIAAC skills relative to TIMMS / PISA. Most other countries gain compared to the other PCA, as they are more advantaged relative to England and the East Asians on young people’s life skills than they are on young people’s education measures.

 
• Category: Race/Ethnicity • Tags: PISA, Psychometrics, Statistics 
Email This Page to Someone

 Remember My Information



=>
Commenters to Ignore...to FollowEndorsed Only
    []
  1. [Overperformance in TIMSS relative to PISA can arguably be used as a proxy for schooling quality, since it’s more dependent on academic/curricular skills than on raw intelligence]

    Surely that effect, if it exists, would be confounded at least in the case of the TIMSS 4th grade tests, since they should be considerably younger than the PISA students and have had less time to benefit by schooling.

    Read More
    ReplyAgree/Disagree/Etc.
    AgreeDisagreeLOLTroll
    These buttons register your public Agreement, Disagreement, Troll, or LOL with the selected comment. They are ONLY available to recent, frequent commenters who have saved their Name+Email using the 'Remember My Information' checkbox, and may also ONLY be used once per hour.
    Sharing Comment via Twitter
    //www.unz.com/akarlin/timss-vs-pisa-performance/#comment-1690197
    More... This Commenter Display All Comments
  2. I was distracted by the thread about PISA hype. Rather than just hype, the PISA scores seem to be very significantly correlated with how the multinational corporate executives view the countries. The World Economic Forum had conducted the so called “Executive Opinion Survey” WEFEOS with rating from 0 to 100, high values being good. Regression with the PISA 2015 scores,

    WEFEOS = +0.30018*Sci -77.3491; n=58; Rsq=0.4935; p=7.938e-10

    WEFEOS = +0.267138*Math -60.6147; n=58; Rsq=0.4621; p=4.418e-09

    WEFEOS = +0.283822*Read -69.0029; n=58; Rsq=0.4601; p=4.901e-09

    As it stand there are already few countries with % at proficiency levels 5 and 6 greater than 20%. Any more abstract tests might fail to measure anything meaningful for many countries. There is already significant size country withdrawing from the PISA survey.

    Read More
  3. Roger Schank, 6 December: Getting people to think for themselves, becoming capable of employment, able to live happy lives, would seem to be better goals to me, but instead we have PISA.

    https://educationoutrage.blogspot.com/2016/12/last-week-i-spoke-at-online-education.html

    His rant: as with Trump, you should take him seriously but not always literally.

    Regarding the value of testing, he has company. Various educationists have been pointing out that grading students has absolutely no basis in science. I’ve heard them saying that on radio. If you’re interested in references, ask Alfie Kohn: ….. While I understand the social mobility argument — a justification similar to what was offered for using standardized test scores as a criterion for university admission in the U.S. a century ago — the disadvantages [of exams] are now widely understood to swamp any benefits.

    - he wrote to me when I showed him some data from a course in China’s history.
    I asked him about how come the Chinese state was often at the same time both exam-centered and very, very capable (at least according to sinologists). He replied he doesn’t know.

    Of course, one has to keep in mind that institutions and institutionalised social patterns – like tests – tend to have fairly short shelf lives.

    Read More
  4. “high levels of equity” between rich and poor pupils very important!

    Please also see:

    “”Even in rural areas and in disadvantaged environments, you see a remarkable performance.”
    In particular, he said the test results showed the “resilience” of pupils to succeed despite tough backgrounds – and the “high levels of equity” between rich and poor pupils.”

    http://www.bbc.co.uk/news/business-17585201

    China: The world’s cleverest country?

    Read More
  5. Just came across an article about Ukraine’s TIMMS score:

    http://voxukraine.org/2016/01/28/evaluating-ukraine-the-pisa-ranking-en/

    Ukraine’s predicted PISA scores would be 478.6 in science (same as Iceland) and 460 in math (between Israel’s 466 and Greece’s 453).

    The average of Ukraine’s predicted science and math PISA scores, 469, would be between the reading + science + math PISA averages of Slovakia (471) and Greece (466). In contrast, Russia’s readng + science + math average is 481, Serbia’s 447, Romania’s 440. Poland’s is 521.

    Ukraine will be taking part in PISA in 2018.

    Read More
    • Replies: @Anatoly Karlin

    Here we evaluate the goal of ‘being in the top 50 of the PISA ranking’.
     
    I.e., doing marginally better than Turkey, the UAE, and perhaps even Romania. Lame.

    Not surprisingly, countries’ results in PISA closely track countries’ results in TIMMS. Below we present the results of a regression analysis of PISA results on TIMMS results for the 28 countries which were participating both in the 2012 wave of PISA and the 2011 wave of TIMMS. Given Ukraine, participated in the 2011 wave of TIMMS, the estimated regression equation allows us to predict the score Ukraine would have had if Ukraine had participated in 2012 in PISA, and hence estimate the rank Ukraine would have had at that moment.
     
    So here's the problem: One would assume that, based on the common legacy of its education system that, as with Russia and the other ex-Soviet regions, Ukraine would do relatively much better on TIMSS (PS. LOL, I always forget whether its two Ms or two Ss as well) than on PISA. So the regression analysis would probably be more accurate if done wrt to just the ex-Soviet countries.

    In TIMSS 2011, Ukraine did an average (across Math/Science) of about 50 points (~7.5 IQ points) worse than Russia.

    So I think expecting ~470 is quite optimistic, though still in line with my oft-made prediction that Ukraine is probably somewhere between Russia and the Balkans.

    I suspect Ukraine will be more like a Romania/Bulgaria-like 450, with historical Novorossiya at a Stavropol/Rostov/Bryansk like 460, the far West at a Balkans-like level of not far above 400, and Kiev 1/2-2/3 S.D. above the country average, like Moscow, with perhaps 520.

    But we'll see. I am certainly looking forwards to December 2019. Let's put it this way. Ukraine gets >460, you win; <460, I win? :)
  6. @AP
    Just came across an article about Ukraine's TIMMS score:

    http://voxukraine.org/2016/01/28/evaluating-ukraine-the-pisa-ranking-en/

    Ukraine's predicted PISA scores would be 478.6 in science (same as Iceland) and 460 in math (between Israel's 466 and Greece's 453).

    The average of Ukraine's predicted science and math PISA scores, 469, would be between the reading + science + math PISA averages of Slovakia (471) and Greece (466). In contrast, Russia's readng + science + math average is 481, Serbia's 447, Romania's 440. Poland's is 521.

    Ukraine will be taking part in PISA in 2018.

    Here we evaluate the goal of ‘being in the top 50 of the PISA ranking’.

    I.e., doing marginally better than Turkey, the UAE, and perhaps even Romania. Lame.

    Not surprisingly, countries’ results in PISA closely track countries’ results in TIMMS. Below we present the results of a regression analysis of PISA results on TIMMS results for the 28 countries which were participating both in the 2012 wave of PISA and the 2011 wave of TIMMS. Given Ukraine, participated in the 2011 wave of TIMMS, the estimated regression equation allows us to predict the score Ukraine would have had if Ukraine had participated in 2012 in PISA, and hence estimate the rank Ukraine would have had at that moment.

    So here’s the problem: One would assume that, based on the common legacy of its education system that, as with Russia and the other ex-Soviet regions, Ukraine would do relatively much better on TIMSS (PS. LOL, I always forget whether its two Ms or two Ss as well) than on PISA. So the regression analysis would probably be more accurate if done wrt to just the ex-Soviet countries.

    In TIMSS 2011, Ukraine did an average (across Math/Science) of about 50 points (~7.5 IQ points) worse than Russia.

    So I think expecting ~470 is quite optimistic, though still in line with my oft-made prediction that Ukraine is probably somewhere between Russia and the Balkans.

    I suspect Ukraine will be more like a Romania/Bulgaria-like 450, with historical Novorossiya at a Stavropol/Rostov/Bryansk like 460, the far West at a Balkans-like level of not far above 400, and Kiev 1/2-2/3 S.D. above the country average, like Moscow, with perhaps 520.

    But we’ll see. I am certainly looking forwards to December 2019. Let’s put it this way. Ukraine gets >460, you win; <460, I win? :)

    Read More
    • Replies: @AP

    I suspect Ukraine will be more like a Romania/Bulgaria-like 450, with historical Novorossiya at a Stavropol/Rostov/Bryansk like 460, the far West at a Balkans-like level of not far above 400, and Kiev 1/2-2/3 S.D. above the country average, like Moscow, with perhaps 520.
     
    Chernivtsi and Zakapatska oblast might be at a Balkans level but Lviv and Ivano-Frankivsk (Galicia) would probably be closer to Kiev. It'll be interesting. Poor performance relative to IQ corroborates post-independence poor governance.
  7. @Anatoly Karlin

    Here we evaluate the goal of ‘being in the top 50 of the PISA ranking’.
     
    I.e., doing marginally better than Turkey, the UAE, and perhaps even Romania. Lame.

    Not surprisingly, countries’ results in PISA closely track countries’ results in TIMMS. Below we present the results of a regression analysis of PISA results on TIMMS results for the 28 countries which were participating both in the 2012 wave of PISA and the 2011 wave of TIMMS. Given Ukraine, participated in the 2011 wave of TIMMS, the estimated regression equation allows us to predict the score Ukraine would have had if Ukraine had participated in 2012 in PISA, and hence estimate the rank Ukraine would have had at that moment.
     
    So here's the problem: One would assume that, based on the common legacy of its education system that, as with Russia and the other ex-Soviet regions, Ukraine would do relatively much better on TIMSS (PS. LOL, I always forget whether its two Ms or two Ss as well) than on PISA. So the regression analysis would probably be more accurate if done wrt to just the ex-Soviet countries.

    In TIMSS 2011, Ukraine did an average (across Math/Science) of about 50 points (~7.5 IQ points) worse than Russia.

    So I think expecting ~470 is quite optimistic, though still in line with my oft-made prediction that Ukraine is probably somewhere between Russia and the Balkans.

    I suspect Ukraine will be more like a Romania/Bulgaria-like 450, with historical Novorossiya at a Stavropol/Rostov/Bryansk like 460, the far West at a Balkans-like level of not far above 400, and Kiev 1/2-2/3 S.D. above the country average, like Moscow, with perhaps 520.

    But we'll see. I am certainly looking forwards to December 2019. Let's put it this way. Ukraine gets >460, you win; <460, I win? :)

    I suspect Ukraine will be more like a Romania/Bulgaria-like 450, with historical Novorossiya at a Stavropol/Rostov/Bryansk like 460, the far West at a Balkans-like level of not far above 400, and Kiev 1/2-2/3 S.D. above the country average, like Moscow, with perhaps 520.

    Chernivtsi and Zakapatska oblast might be at a Balkans level but Lviv and Ivano-Frankivsk (Galicia) would probably be closer to Kiev. It’ll be interesting. Poor performance relative to IQ corroborates post-independence poor governance.

    Read More

Comments are closed.

PastClassics
What Was John McCain's True Wartime Record in Vietnam?
The “war hero” candidate buried information about POWs left behind in Vietnam.
The evidence is clear — but often ignored
What the facts tell us about a taboo subject
While other top brass played press agents for the administration’s war, William Odom told the truth about Iraq—though few listened.