For years, Audacious Epigone and myself have been pointing out that Texas public school kids do surprisingly well on the federal NAEP exam within each ethnic group. Now, the NYT finally figures that out, too:
Surprise: Florida and Texas Excel in Math and Reading Scores
OCT. 26, 2015
When the Education Department releases its biennial scorecard of reading and math scores for all 50 states this week, Florida and Texas are likely to look pretty mediocre. In 2013, the last time that scores were released, Florida ranked 30th on the tests, which are given to fourth and eighth graders, and Texas ranked 32nd.
But these raw scores, which receive widespread attention, almost certainly present a misleading picture — and one that gives short shrift to both Florida and Texas. In truth, schools in both states appear to be well above average at teaching their students math and reading. Florida and Texas look worse than they deserve to because they’re educating a more disadvantaged group of students than most states are.
A report released Monday by the Urban Institute has adjusted the raw scores for each state to account for student demographics, including poverty, race, native language and the share of students in special education. The central idea behind the adjustments is that not all students arrive at school equally prepared, and states should not be judged as if students did.
“Making these demographic adjustments,” said Matthew Chingos, a senior fellow at the Urban Institute and the report’s author, “gives us a much better picture of how students are doing.”
With the adjustments, Texas jumps all the way to third in the 2013 state ranking, and Florida to fourth. Massachusetts, which also ranks first with unadjusted scores, remains in the top spot; although the state is relatively affluent, its students perform even better than its demographics would predict. New Jersey ranks second.
Other states with a less extreme version of the Florida and Texas story — that is, their schools are performing better than is often understood — include Arkansas, Georgia, Nevada and New York.
The new results will no doubt offer fodder for the continuing debate over education. Florida and Texas are mostly Republican-run states, where teacher unions are relatively weak and policy makers have tried to introduce more competition and accountability. At the same time, some states with a strong union presence, including New Jersey and New York, also perform well.
The results do seem to offer another vote of confidence for rigorous, common standards — an idea that took off with the Common Core, but has since come under harsh political attack. Massachusetts helped pioneer the idea of such standards in 1993, with ambitious goals, clear assessments and increased school funding.
States with less impressive results in the Urban Institute analysis, where favorable demographics are disguising mediocre performance, include Connecticut, Nebraska, Wyoming, Montana and Iowa. And while New Hampshire, Vermont, Minnesota and Washington are still above average, their scores are not as impressive as the unadjusted numbers suggest.
Many of these states are affluent or predominantly white — if not both. The new analysis suggests that many of their school systems have better reputations than they deserve. They enroll a lot of students who come to school well prepared and thus excel on tests. But the schools themselves are not doing as good a job as their test scores suggest.
This won’t come as a surprise to long-time readers of the Steveosphere.
But, while it’s journalistic custom to refer to the NAEP as “the gold standard” of testing, how much can we really trust the NAEP for making these kind of subtle state by state comparisons?
Specifically, the NAEP are low stakes tests to the kids, and in some states, the adults administering the NAEP treat them as low stakes for them too. On the other hand, there is some evidence that Texas administrators cares about their schools scoring better on the NAEP. For example, Texas excuses 10% of its sample of 4th graders from taking the NAEP while California only excuses 3% of its sample.
In contrast, the SAT and ACT college admissions tests taken by juniors and seniors in high school are clearly high stakes tests on which students have an incentive to try hard.
But comparing SAT and ACT average scores are tricky because in most states not everybody takes even one of the tests because they aren’t interested in applying to a competitive college.
And there are regional differences in whether a state is traditionally an ACT state (e.g, Iowa) or an SAT state (e.g., New Jersey) that influence average scores. For example, a few decades ago, Iowa usually led the county in average SAT scores because the only Iowa students who took the SAT were brainiacs interested in applying to the exclusive coastal colleges.
On the other hand, the regional differences are blurring as, especially, the ACT aggressively pushed into SAT states. Now it’s becoming common for ambitious students to take both tests to see which one they do better upon.
In general, the smarter people are, the more likely they are to take a college entrance exam. So, the lower the percentage of kids taking an exam in a state, the more inflated that state’s average score tends to be relative to the whole population of kids in the state, which is the figure I’d like to roughly estimate.
So, I’m going to present the 2014 SAT and ACT numbers for the two biggest states, Texas and California, both average scores and percent of the cohort taking each exam.
Unfortunately, I don’t remember who provided me with these numbers of SAT and ACT scores from 2006 through 2014, both set on the old-fashioned SAT 400 to 1600 scale (i.e., leaving out the doomed Writing subtest; the ACT is a 3 part exam with a maximum score of 36 but the ACT people publish tables for how to convert ACT scores to SAT scores). So, I don’t know if these data are reliable. But they don’t seem too implausible either. Also, I found the 2010 Census data for 13 year olds by race in each state as a proxy for 17 year olds in 2014. (For some reason, I couldn’t find Asians by age in Texas in 2010, so I’ll just stick to the Big Three racial/ethnic groups.)
Let’s start off by looking at the white scores. In California, the total number of SAT tests said to be taken by whites in 2014 was equal to 47% of the number of white 13-year-olds in 2010 on the Census. In Texas, the percentage of SAT takers among whites was 49%, so we can compare the average SAT scores pretty directly, with just a reminder that this comparison is slightly biased in favor of California: California white kids score 1099, which is 36 points higher than Texas’s average white SAT score. The standard deviation on the 400 to 1600 scale was supposed to be 200 (although it’s gotten larger over time), so that would suggest California kids score about 0.18 standard deviations higher on the SAT, which is not a large gap, but not vanishingly small either.
On the ACT, California white high school students average 1144 and Texas white kids 1078, for a 64 point or .32 s.d. gap. But only 21% of white kids in California take the ACT, suggesting it’s kind of a boutique test in California for strivers. In contrast, 32% of Texas whites take the test, suggesting there the ACT in Texas falls in between a boutique test and a meat and potatoes test. So, it’s hard to compare the ACT scores for whites directly.
But my general impression is that whites in California, at least among the college curious, score a little better on college admissions tests than whites in Texas.
(One methodological quibble to keep in mind is that I don’t know how the data treats an individual student retaking the same test in one year. Do they enter all the scores or just the highest? And is the likelihood of retaking the same test greater in one state or the other?)
The difference is pretty small, but it’s in the opposite direction of the difference reported by the NAEP.
Among blacks, the California advantage appears to be quite similar to what’s seen among whites: small but not insignificant.
On the other hand, among Hispanics, California’s advantages in test scores are smaller than among among whites and blacks, and Texas Hispanics are somewhat more likely to take both the SAT and ACT. I’m not at all confident that California Hispanics would do better overall than Texas Hispanics if everybody in both states took a college admissions test.
So, my best guess would be: modest advantages for the white and black populations of Californians over white and black Texans, respectively, but Hispanics in Texas overall are no lower scoring and might actually be a few points higher.
By the way, Texas Asians score 16 points higher on the SAT and 17 points higher on the ACT than California Asians. I would include them in the table if I could find the 2010 Census figures for the number of 13-year-old Asians in Texas.