Peter over at Burnt Orange Nation (later republished at EDSBS) beat us to the punch on an analysis of the accuracy of the AP Pre-Season Poll we were planning to do. [shaking fist] He did raise some good questions that we feel we can address with our analysis. This will be the first part in a three part series that examines the accuracy of the pre-season AP poll. In the first part, we'll take a look at the history of the preseason poll. When has it succeeded? When has it failed miserably? Then we'll try to put our combined six degrees in aerospace engineering to good use and quantify just how successful the AP writers are in their attempt to predict the future.

In the second part, we'll seek to provide some further insight into the quality of the AP poll by comparing its success with that of a random number generator (yes we are dorks). In our final part, we'll examine the potential dangers of issuing pre-season polls, specifically, does the preseason poll influence later polls and ultimately the national champion.

Before we begin, many thanks to SoonerStats.com, from whom all the data used in this analysis was taken.

Nobody moved paper like newsboys

To start, we need some perspective. The Associated Press first published a pre-season poll before the 1950 season. The opinions of the poll then were much the same as those floating around the blogosphere now. Take this article from the Washington Post:

"Who? Me?"

That was the general incredulous, if not downright alarmed, reaction of college coaches whose teams were chosen in the top 10 in the Associated Press pre-season football poll.

The Comments ranged from Michgan's Bennie Oosterbaan's "no comment" to a doleful "we'll be lucky to win half our games" from Tennessee's Bob Neyland, but all left the impression that the coaches felt they were being put in a very unwelcome spot.

"AP Poll Give Lucky (?) Coaches Heebie Jeebies." Washington Post 21 Sep 1950: p 17.

Before 1950, the AP would wait until most teams had played a couple of games and the writers actually had some performance on which to base their votes before issuing their first poll. Even in 1950, the AP didn't issue it's first in-season poll until some teams had played as many as three games. It wasn't until 1954 that the writers published a poll for every week of the college football season.

It is commonly assumed that the AP began publishing pre-season polls to increase readership for their affiliate newspapers. A preseason poll not only helps jump start the media machine that has always surrounded college football, but also gives columnists an excuse to fill countless inches of print on why a team should or should not be ranked where they are.

The circumstances surrounding the inaugural pre-season poll give no reason to suspect otherwise. In this light, it should come as no surprise that the first team ranked number one in the AP pre-season poll was Notre Dame. Always the media darling, the poll's debut and Notre Dame's number one ranking seem too much of a coincidence not to be related.

Entering the 1950 season, Notre Dame was coming off a 10-0 national championship season, and its third national title in four years under head coach Frank Leahy. The team had not lost a game since 1945.

The Irish however were to fall victim to attrition and would finish the season 4-4-1 and out of the rankings. The first AP pre-season poll was less than prophetic. The voters would redeem themselves when they correctly predicted the national champion the next two years with Tennessee and Michigan State. For the record, despite winning the AP national title four times in those 56 years, in the five years when Notre Dame has been ranked #1 in the pre-season poll, not once did they win the national championship.

Eventual Champs who were Unranked at Season Start

Year	Team
1958	LSU
1960	Minnesota
1962	Southern Cal
1980	Georgia
1981	Clemson
1983	Miami (FL)
1984	BYU

Correctly Predicted National Champions

Year	Team
1951	Tennessee
1952	Michigan St.
1956	Oklahoma
1974	Oklahoma
1975	Oklahoma
1978	Alabama
1995	Oklahoma
1993	Florida St.
1999	Florida St.
2004	Southern Cal

She blinded me with science

In all, the AP writers have correctly ranked the eventual national champion as the pre-season number one 10 times in 56 years (17.9%), a pretty impressive track record considering 119 teams play D1-A football; however, the voters are completely wrong nearly as often. In seven of the 56 seasons (12.5%), the eventual national champion wasn't even ranked in the initial pre-season polls.

It should be noted that from 1961 to 1967, only ten teams were ranked in the pre-season poll. During these years, only once was the eventual national champion unranked in the pre-season poll. Not counting these years, six times out of 49 seasons were champions unranked, a nearly identical 12.2%.

Pre-season Ranking of the Eventual National Champion

Ranking	Number	Percentage
No. 1	10	17.9%
2-5	16	28.6%
6-10	15	26.8%
10-25	8	14.3%
Not Ranked	7	12.5%

But the poll is about more than just predicting a national champion. The poll is supposed to reflect what 10, 20, or 25 teams should prove themselves to be the best in the country in the upcoming season. In this regard, the poll fairs much worse. More often than not, 40% of the teams ranked in the pre-season poll will find themselves unranked at the end of the season. In the modern poll, that would be 10 out of 25 teams. While the pollsters have gotten a little better over the years improving from 45% of teams being dropped in the 1950s to only 33% in the 1990s, they have been very consistent in their lousy picking. For example in the 2000s, nearly 47% of teams ranked in the pre-season poll have been absent from the final polls.

Probability of a Team Finishing the Season Unranked

Ranking	Probability
1-5	15%
6-10	32%
11-15	51%
16-20	49%
21-25	63%

Of interest is not only how many teams are missing from the rankings at the end of the season, but where in the rankings these choke artists were originally ranked. Not surprisingly, the lower a team ranks in the pre-season ranking, the more likely they are to be missing from the rankings at the end of the season. While there is only a 15% chance that fans of a team ranked in the Top 5 will commit seppuku, for teams ranked outside of the Top 10, there is a better than 50% chance their name will not appear in the final season poll.

Care to go deeper?

While one way of quantifying the accuracy of pre-season polls is to examine how may teams from the pre-season are still in the final poll, another measure is to examine how far the remaining teams are from their original rankings.

Perhaps the best known means of finding the differences in two sets of data (at least to anyone who has used Excel) is the standard deviation. Without getting too technical, the standard deviation really is meant to apply to random scatter in data, which we sincerely hope college football ranking are not, or are they . . .

Instead, we'll determine the differences between the pre-season and final rankings by examining the average difference between a team's pre-season and final ranking in the AP poll. We took the absolute difference, so direction, whether a team was ranked higher or lower, doesn't matter. Teams that were no longer ranked at the end of the season were assigned a value of one greater than the last ranked team, the most conservative ranking for those teams. For a typical season, any given team in the pre-season ranking would finish on average 6.3 places away from their preseason rank. The most accurate year with an average difference of 4.4 was 1974 when Oklahoma was pre-season and season-ending #1. The least accurate year was ten years later in 1984 when BYU came out of nowhere to win the national championship.

Percentage of Correctly Ranked Teams

Ranking	Percentage
1-5	10.7%
6-10	3.2%
11-15	3.3%
16-20	3.2%
21-25	3.6%
Total	5.1%

So what exactly do these numbers mean? Well think of it in this context. Most years of the polls existence, 20 teams were ranked. An error in ranking of 6.3 is a none-to-insignificant one-third of the poll. Further, the team ranked 11th in the pre-season can at worst be 10 places away in the final poll. In this light 6.3 is not looking all that impressive. Finally, since this is an average, every time a team is ranked correctly, another team is off by 12.6 places.

Delving a little deeper, the pollsters have correctly ranked teams in the pre-season poll 5.1% of the time. Teams in the Top 5 are ranked the same in the pre-season and final polls an uncharacteristically high 10.7% of the time. All other teams are correct only 3.3% of the time. This is the clearest indication of the relative accuracy of the highest ranked teams, but the unreliability of the rest of the poll.

Do they get anything right?

Well, despite the problems mentioned above, we still find it impressive that the AP Poll correctly predicted 10 national champions. Given all that can transpire over the course of a college football season, predicting the winner is an impressive feat. Further, among teams ranked in the Top 5, 57% of the time, those teams finish within five spots of their original rank.

However, by publishing a full poll, the voter are claiming to do more than merely predict the national champion or even pick three of the top five teams. They instead claim to judge exactly where the top 25 teams in the country stand before a ball is even snapped. Given just how often these predictions turn out to be totally wrong, publishing a poll in the first place seems pointless.

But wrong isn't necessarily bad, is it? Well, it can be. Given the recent trend of maintaining your place in line as long as you win, the pre-season rankings can have a tremendous impact on the final rankings. Just ask Auburn. We'll examine this aspect of the pre-season polls in more detail in the third part of our series.

Next though, we'll see if a random number generator can do just as well as the combined wisdom of 65 of the most prominent members of the media.