In the first part of our three part series examining the historical performance of the AP Pre-Season Poll, we examined quantitatively the accuracy of the pre-season poll when compared to the season-ending poll.

We used primarily two measures of success (or failure, as the case might be) -- the typical number of pre-season ranked teams dropped from the final poll and the mean absolute difference between the individual teams' position in the polls. From a typical AP Pre-Season Poll, 40% of the teams will have fallen out of the rankings by the final poll. The average team in a typical AP Pre-Season Poll will be ranked 6.3 places from its original ranking at the end of the year.

While these numbers provide some insight into the failing of pre-season polls, it is difficult to wrap one's head around some of the abstractions that are inherent to any statistic. So to try to provide some perspective for these numbers, we decided to compare the AP Pre-Season Poll with two decidedly less contemplative rankings. The first is simply the previous season-ending AP Poll. While the previous season poll can provide a general indication of the ability of the next year's team, surely these polls can not fare as well as a new poll where voters have insight into personnel changes and different schedules. The second "poll" also draws from the previous year's final poll, but we used a computer to shake up the poll and assign the teams using a random number generator.

Best and Worst

While teams ranked in a typical final poll are on average 6.3 places from their starting position, the best in worst polls in these categories were 4.4 places in 1974 and 10.1 in 1984, respectively. The attrition numbers are not extrema, but the 1974 is a below average 30% and the 1984 is a typical 40%. These polls in their entirety are posted below.

Least Accurate Pre-Season AP Poll

1984 - Mean Error 10.1
Pre-Season	Rank		Final
Auburn	1	1	BYU
Nebraska	2	2	Washington
Pittsburgh	3	3	Florida
Clemson	4	4	Nebraska
UCLA	5	5	Boston College
Texas	6	6	Oklahoma
Ohio St.	7	7	Oklahoma St.
Notre Dame	8	8	SMU
Alabama	9	9	UCLA
Miami (FL)	10	10	Southern Cal
Penn St.	11	11	South Carolina
Iowa	12	12	Maryland
Arizona St.	13	13	Ohio St.
Michigan	14	14	Auburn
SMU	15	15	LSU
Oklahoma	16	16	Iowa
Florida	17	17	Florida St.
Washington	18	18	Miami (FL)
Boston College	19	19	Kentucky
Florida St.	20	20	Virginia

Most Accurate Pre-Season AP Poll

1974 - Mean Error 4.4
Pre-Season	Rank		Final
Oklahoma	1	1	Oklahoma
Ohio St.	2	2	Southern Cal
Notre Dame	3	3	Michigan
Alabama	4	4	Ohio St.
Southern Cal	5	5	Alabama
Michigan	6	6	Notre Dame
Nebraska	7	7	Penn St.
Penn St.	8	8	Auburn
LSU	9	9	Nebraska
Texas	10	10	Miami (OH)
Houston	11	11	NC State
UCLA	12	12	Michigan St.
Pittsburgh	13	13	Maryland
Maryland	14	14	Baylor
Arizona St.	15	15	Florida
Tennessee	16	16	Texas A&M
Arizona	17	17	Mississippi St.
NC State	18		Texas
Arkansas	19	19	Houston
Texas A&M	20	20	Tennessee

With just a quick glance at the 1974 Poll, it is clear why this poll was so successful in predicting the seasons end. First, the poll correctly predicted the national champion Oklahoma, although this was the only team with the same rank in both polls; however, of the other teams in the Top 9, only one would not finish the season also ranking in the Top 9. While the positions would change, none would be ranked further than three positions away at season's end. Further, all the teams in the bottom half of the poll not dropped from the poll were in the poll's bottom half in the end. All this leads to very little difference for most teams and therefore a low average -- a pretty successful poll.

At the opposite end of spectrum, just how complete a failure the 1984 Poll was is evident at a glance. To start, the pre-season number one Auburn dropped to number 14. The eventual national champion BYU was unranked at season start. Further inspection reveals that only two teams in the entire poll were ranked within three spots of their starting position. Half the teams ranked in the Top 10 were dropped from the poll. Of the remaining teams, they were more likely found in the bottom half of the poll than the top. Also problematic, the teams ranked at the very bottom of the pre-season poll ended up ranked near the top of the final poll with #15 SMU, #16 Oklahoma, #17 Florida, #18 Washington, and #19 Boston College in near reverse order.

Using these two polls as a reference, we can now explore our other contrived pre-season polls.

Oldies but Goodies

When filling out a pre-season poll, the logical starting point would seem to be last year's poll. See who was good last year and then ponder: Who had late season momentum? How many returning starters do the top teams have? What schedule difference will give teams an edge this year?

Well, what if we forgot about those questions and just filled out our ballot exactly as last season ended. The verdict -- not too shabby.

Pre-Season AP Poll vs. Previous Year's Final AP Poll

Pre-Season AP	Category	Last Year's
10	Correct N.C.	7
7	Unranked N.C.	12
40%	Teams Dropped	44%
10.1	Max. Error	9.4
6.3	Mean Error	6.8
4.4	Min. Error	4.8

In the 56 seasons the AP has had a pre-season poll, there have been seven repeat champions, so this method has correctly predicted the national champion seven times. Twelve times the eventual national champion would not have been ranked in these "pre-season" polls. Compared to 10 correct and 7 total misses for the actual AP Pre-Season Polls, this method has not proved itself better yet. One area of improvement, while 29% percent of the time the eventual national champion comes from positions 2 through 5 in the actual pre-season poll, using last year's final poll, the success rate in this range jumps to nearly 40%.

Looking at our first quantitative measure, typically 44% of the teams ranked in the previous year's final poll are not ranked in the next year's final poll. This is only 4% greater than the actual AP Pre-Season Poll, approximately one team in a 25 team poll -- worse, but not prohibitively so.

In our final statistical category, mean position error, utilizing the previous season's final poll results in a typical mean error of 6.8 places, approximately 0.5 places worse than the actual pre-season poll. The best and worst years are presented below for comparison.

The most successful year with this method is once again 1974. One notes again that all but one of the Top 9 teams at the end of the 1973 season finish the 1974 season again in the Top 9, an indication perhaps that the success of the 1974 AP Pre-Season Poll was not due to tremendous insight on the part of the voters, but rather two successive years of little change in the quality of the best teams. The least successful year for this method was 2002. The poll achieved this poor performance despite both Washington St. and Virginia Tech remaining in exactly the same place between the two years. The downfall was that a whopping 60% of the teams from the 2001 Final Poll were gone by season's end in 2002.

Least Accurate Previous Season Poll as Pre-Season

2002 - Mean Error 9.4
2001 Final	Rank		2002 Final
Miami (FL)	1	1	Ohio St.
Oregon	2	2	Miami (FL)
Florida	3	3	Georgia
Tennessee	4	4	Southern Cal
Texas	5	5	Oklahoma
Oklahoma	6	6	Texas
LSU	7	7	Kansas St.
Nebraska	8	8	Iowa
Colorado	9	9	Michigan
Washington St.	10	10	Washington St.
Maryland	11	11	Alabama
Illinois	12	12	NC State
South Carolina	13	13	Maryland
Syracuse	14	14	Auburn
Florida State	15	15	Boise St.
Stanford	16	16	Penn St.
Louisville	17	17	Notre Dame
Virginia Tech	18	18	Virginia Tech
Washington	19	19	Pittsburgh
Michigan	20	20	Colorado
Boston College	21	21	Florida St.
Georgia	22	22	Virginia
Toledo	23	23	TCU
Georgia Tech	24	24	Marshall
BYU	25	25	West Virginia

Most Accurate Previous Season Poll as Pre-Season

1974 - Mean Error 4.8
1973 Final	Rank		1974 Final
Notre Dame	1	1	Oklahoma
Ohio St.	2	2	Southern Cal
Oklahoma	3	3	Michigan
Alabama	4	4	Ohio St.
Penn St.	5	5	Alabama
Michigan	6	6	Notre Dame
Nebraska	7	7	Penn St.
Southern Cal	8	8	Auburn
Arizona State	9	9	Nebraska
Houston	9	10	Miami (OH)
Texas Tech	11	11	NC State
UCLA	12	12	Michigan St.
LSU	13	13	Maryland
Texas	14	14	Baylor
Miami (OH)	15	15	Florida
NC State	16	16	Texas A&M
Missouri	17	17	Mississippi State
Kansas	18		Texas
Tennessee	19	19	Houston
Maryland	20	20	Tennessee
Tulane

Has anyone seen the movie Tron?

Next we decided to put to use one of the few resource our new blog has at its deposal -- the markmaybewrong super computer. We decided to use the previous season's final poll as our starting point. Using those teams, we reordered the list randomly, but such that no team was ranked further than five places from their position in the original poll. Why five places? It seemed reasonable. You'll just have to trust us that it wasn't based on some extensive analysis to get the best result. This is just for fun.

We did this for all 56 years of the AP Pre-Season Poll. For years where the number of teams changed from the pre-season to final poll, we only examined the number of positions in the lesser of the two polls.

Pre-Season AP Poll vs. Computer-Generated Poll

Pre-Season AP	Category	CPU Poll
10	Correct N.C.	5
7	Unranked N.C.	12
40%	Teams Dropped	44%
10.1	Max. Error	9.8
6.3	Mean Error	5.8
4.4	Min. Error	3.6

Now we only did this once for each year. We could have run this 100 times and gotten different results each time. Again, you'll have to trust us that this was the first and only time we did this, and we didn't wait until we hit gold or anything. Why this clarification?

Well against all logic, our computer's picks out performed the AP pollsters. Now to be fair, it is only in one metric. The computer didn't do a good job of picking the national champion, finding success only five times. Since 30 times the national champion has been ranked in the Top 6 of the previous season's poll, getting 1 out of 6 right is the exact number predicted by a probability analysis, but no where near as good as the ten correctly predicted by the AP voters.

Likewise, when it comes to number of national champions unranked in the poll or the total number of teams in the pre-season poll dropped from the final poll, the computer also falls short. Because the teams ranked are the same as those in the final poll from the previous year (just in a different order), these metrics are identical to the above analaysis -- 12 unranked national champions and 44% of teams dropped.

But in our final metric, the average difference in ranking between the pre-season and final polls, our computer-generated poll really sticks it to the AP pollsters. While teams in the actual AP Pre-Season Poll are on average ranked 6.3 places away from their original position in the final poll, the computer generated poll sees an error of only 5.8. In fact, in 37 of the 56 years (66%), the computer generated poll outperformed the human voters.

Now again, this is one run that could simply be a case of a little dumb luck, but don't you really expect human voters to outperform a random number generator all the time? Well, everywhere outside of a roulette table at least.

Least Accurate Computer-Generated Pre-Season Poll

2002 - Mean Error 9.8
CPU Pre-Season	Rank		2002 AP Final
Oklahoma	1	1	Ohio St.
Florida	2	2	Miami (FL)
Tennessee	3	3	Georgia
Miami (FL)	4	4	Southern Cal
Texas	5	5	Oklahoma
Oregon	6	6	Texas
Nebraska	7	7	Kansas St.
South Carolina	8	8	Iowa
Syracuse	9	9	Michigan
LSU	10	10	Washington St.
Washington St.	11	11	Alabama
Louisville	12	12	NC State
Colorado	13	13	Maryland
Maryland	14	14	Auburn
Florida State	15	15	Boise St.
Illinois	16	16	Penn St.
Michigan	17	17	Notre Dame
Toledo	18	18	Virginia Tech
Stanford	19	19	Pittsburgh
BYU	20	20	Colorado
Virginia Tech	21	21	Florida St.
Washington	22	22	Virginia
Georgia Tech	23	23	TCU
Boston College	24	24	Marshall
Georgia	25	25	West Virginia

Most Accurate Computer-Generated Pre-Season Poll

1964 - Mean Error 3.6
CPU Pre-Season	Rank		1969 AP Final
Notre Dame	1	1	Texas
Texas	2	2	Penn St.
Ohio St.	3	3	Southern Cal
Missouri	4	4	Ohio St.
Southern Cal	5	5	Notre Dame
Penn St.	6	6	Missouri
Michigan	7	7	Arkansas
Arkansas	8	8	Mississippi
Kansas	9	9	Michigan
Tennessee	10	10	LSU
Georgia	11	11	Nebraska
SMU	12	12	Houston
Purdue	13	13	UCLA
Alabama	14	14	Florida
Oklahoma	15	15	Tennessee
LSU	16	16	Colorado
Oregon St.	17	17	West Virginia
Ohio	18	18	Purdue
Houston	19	19	Stanford
Auburn	20	20	Auburn

Final Thought

So what has this analysis shown us? Well, hopefully you have a better grasp for the quantities we tried to use to analyze the performance of the AP voters. What we have shown is that the pollsters do outperform other techniques when it comes to picking the national champion, but when it comes to analyzing the poll as a whole, it is really no better than a random collection of good teams from last year. Even in the AP pollsters best year, their pre-season poll was pretty much the previous year's final poll. Not very impressive.

So hopefully when the AP Pre-Season Poll comes out, you'll treat all but the highest ranked teams with a grain of salt. The important questions is will the voters? Recent experience has shown that the voters can be very loyal to their pre-season preconceptions with potentially disastrous results. In the final part of our series, we'll look for hard evidence of these empirical notions.