Finding the Pony in the ACT

WOW -- Look at what a good PR effort can produce! Here's how the this year's release of ACT scores was headlined this morning:

Tribune:

"Illinois ACT Scores Up Across Board"

Sun-Times:

"City, State ACT Scores jump, Still Lag"

Daily Herald:

"A Tough ACT to Follow / Test Scores Rise Across State ..."

Where the "good" news came from

Yesterday, the ISBE, our state's education bureaucracy, sent out a carefully worded press release selecting specific numbers from the ACT results, which they got to see in advance of the official release. As all good bureaucrats do, the news was spun to highlight the best stories possible.

Here are the key "good" stories that the ISBE managed to extricate from the ACT:

• Illinois' "composite" score was higher than the previous year, going from 20.3 to 20.5
• Illinois' Reading score is 20.6 (which is up from 20.3 the previous year)

Now, the REAL story

But today the results are available to everyone, and the story about Illinois is less impressive than those headlines suggest.

It turns out that the ACT scores are up nationally, not just in Illinois.

The new composite ACT score is 21.1, up from 20.9, an "increase" of 0.2. So, Illinois' "increase" by the same 0.2 (from 20.3 to 20.5) turns out to be merely keeping pace with the national score!

So, can Illinois at least claim that it's improving along with the rest of the nation?

Well, we'd like to think that the whole country is improving. But we have to wonder: has the country improved, or has the test gotten a little easier? A rising tide raises all boats. If you see boats going up, that tells you more about the tide than about boats.

I looked at results for the 50 states plus the District of Columbia. It turns out that 45 -- 45!-- of them had ACT composite scores that were either improved (37) or unchanged (8). Only 6 states had drops!

Wow -- that means that just about EVERY state had something to crow about!

What about specific tests?

Illinois educrats will (no doubt) be cheering about results in each of the areas of the ACT. Illinois scores are up in English, Math and Reading. Only in Science were results flat.

But again, this "rising tide" MUST be compared to the national changes:

	Composite	English	Math	Reading	Science
National	+0.2	+0.2	+0.1	+0.1	0
Illinois	+0.2	+0.3	+0.1	+0.3	0

The modest Illinois math "improvement" turns out to be the same as the national change.

Ah, but what about those English and Reading scores, that are better than the national changes? For that, we really need to look at longer trends.

In Reading, the Illinois "improvement" this year (+0.3) is largely a recovery from a drop (-0.2) the year before. The English score is a more consistent increase. Here's the trend:

	Reading		English
Report Year	Illinois	National	Illinois	National
2002	20.3	21.1	19.4	20.2
2003	20.4	21.2	19.6	20.3
2004	20.5	21.3	19.7	20.4
2005	20.3	21.3	19.9	20.4
2006	20.6	21.4	20.2	20.6

So, the trends are up, and Illinois is at its best score in the five-year series. But once again, notice the "tide" effect: the national scores are up as well. Is the nation as a whole improving? Or is the ACT getting easier?

Finding Good News

The ACT results are a playground for eager public relations people looking for "good" news to crow about. With four subject areas plus a composite score, and then a "rising tide" of scores nationally in the ACT, wow, just about any state can find something good to announce!

I explored this a little further ...

As reported above, a mere 6 states had declines in their composite scores, so everyone else gets to announce "Overall gains" or at least "Keeping pace".

When we break it out by subject, the spinnable good news (largely thanks to the national score increases) multiplies rapidly. We find:

Test component	States that get to announce "improvements" or "keeping pace"
Composite	45
English	43
Math	46
Reading	44
Science	41

In another way of looking at the data, I found that 44 states would be able to announce "increases" in one or more of the test results! Only two states (Alaska and Rhode Island) would be unable to announce even a "keeping pace" in at least one of the results.

Finding the Pony in the Bad News:
The Woebegone Effect

A good PR effort doesn't stop even when it's hard to find the good news.

I wondered about that in looking at the results for our neighbors in Indiana. Poor Indiana! Down in English (despite national increases), flat in Reading and in the Composite (despite national increases), and math and science track at the national rate. Yikes, what can the Hoosier educrats crow about?

Well, I searched, and found that Indianapolis did indeed pull a rabbit from an empty-looking hat:

"Indiana ACT Scores Top National Average for 16th Year"

(See their PR release here.)

Yup, when all else fails, shout, "We're better than average!"

So, who IS better than average in the ACT? In investigating this, I found that Lake Woebegone is thriving at the ACT! There are 32 states that have composites better than the national average. Moreover, 36 states get to claim they are at the national average or better in Reading!

All the Details

I have included all of the details behind these conclusions in an Excel workbook available for download at this link.