Drafting QBs 2

My last post looked at how likely NFL quarterbacks are to become Pro Bowl selectees based on their draft order. This post will directly analyze QB performance by draft round and order based on passing stats. Ultimately, the passing stats will be converted into average expected team wins through linear regression. The result reveals how many more wins per year a 1st round pick could be expected to deliver than a 2nd round pick, or the 1st QB taken over the 2nd.

How QB Performance Was Measured

Passing performance is measured by career average yards per attempt and interception rate. These stats are combined into a single measure, commonly referred to as Adjusted Yards Per Attempt (AdjYPA). AdjYPA basically turns every interception into a -45 yard pass play. This is a commonly accepted equivalence based on research going back to a 1988 book called The Hidden Game of Football. It's also somewhat intuitive--An interception basically precludes the possibility of a punt, which is typically about 45 yds before the return. Additionally, 10 yards of passing is added for every touchdown thrown. This adjustment is intended to compensate for the truncation of the field and the added difficulty in the compressed field near the goal line.

The Data

Data were obtained from Pro-Football-Reference.com. Quarterbacks drafted from 1980 through 2000 in the first 7 rounds were studied. Players from round 8 or later were excluded because the current draft is limited to 7 rounds, and the point of this analysis is prescriptive. Supplemental picks were not included because the focus here is on draft day itself.

Because the passing game in the NFL has evolved over the years (it's become steadily easier), an adjustment was made for year. Average AdjYPA has increased signficantly between the draft classes of 1980 and 2000. The 1980-1983 draft classes averaged 5.6 AdjYds per attempt compared to an average of 6.0 AdjYds per attempt for the 1998-2000 draft classes. The increase overall in the NFL was a fairly steady 0.13 yds per year. The midpoint of each QB's career was used to adjust all QB's stats as if they played in 2004.

Draftees Without Many Attempts

One big problem about this kind of draft analysis is how to score drafted QBs who never played. If they are assigned zero yards per attempt, an unrealistically low record, it would severely weigh down the averages of all but the very top picks. If they are excluded from the data, then the true expected value of QBs drafted in later rounds would be severely biased upward. QBs who played but did not have many pass attempts are also problematic. They can have erratic stats, some having an insanely high 15 yards per attempt or as low as -20 yds per attempt.

My solution to the problem of QBs with too few attempts was to assign them the stats of the 5th percentile qualifying QB. I chose 200 attempts as the qualifying level because it's where the the stats settle down to reasonably steady and apparently representative levels. The 5th percentile makes sense because it would be unfair to say every QB that didn't get their shot would be as bad the very worst to play. Although many of them would undoubtedly be pretty bad, some of them were simply bottlenecked behind slightly better QBs.

Thus we now have the Wuerffle Line, football's very own version of the Mendoza Line. The 5th percentile falls between Danny Wuerffle and Akili Smith. Even though I think Danny is a good guy, the "Smith-Line" just doesn't resonate.

Now we have a good measure of QB performance that considers passing efficiency, interceptions, and career era, but we're left with an abstract, indirect stat--AdjYPA. The bottom line for every NFL team is wins and not passing stats, so I converted the AdjYPA stat into expected team wins per season.

I ran a regression of all teams from the 2002-2006 seasons that weighs offensive and defensive passing stats, running efficiency, turnover rates, and penalty rates. The model was very similar to the one I ran here. AdjYPA was highly significant, the residuals were randomly distributed, and the model's overall r-squared was 0.72. By holding equal all the factors other than the passing stats, we can see how much passing contributes to team wins. For every 1 yd increase in AdjYPA by a QB, a team can expect to win an additional 1.4 wins each season.

A reference point is necessary when comparing QB to QB, so I borrowed an idea from baseball sabermetrics--the replacement player. But since there is no replacement QB across 21 years of draft picks, I used the 5th percentile again as the reference point. The number of wins added above the 5th percentile was calculated for each QB draft pick. Instead of "Wins Above Replacement," I termed this stat "Wins Above Basement (WAB)." The reference point itself is arbitrary, but it's needed to compare QBs from one round to another.

The first graph below illustrate how many Wins Above Basement a QB could be expected to have based on draft round. The average first round pick earns about 0.3 wins per year more than the average second round pick.
As you can see, the relationship between round and expected QB performance is very linear. But as the next graph shows, not all first round QBs are created equal. The next graph shows wins added by QB draft order. For example, the first QB taken in each draft earns about 2.4 wins per year (more than the worst QBs), and the second QB taken earns about 2.0 wins per year.
Again, we see a very linear relationship. But there may be something more. There are large drop offs in performance from the 1st QB taken to the 2nd, and from the 2nd to the 3rd. Then from there until the 9th or 10th QB taken, it's pretty random. It appears that if your team doesn't get one of the first two QB picks, it might as well take a chance on a later pick. Chances fall off quickly after the first two QBs that a team will find a franchise player.

Overconfidence

Teams often jockey their draft position, even by just a few spots, to ensure they can pick a particular player. Part of the reason they do this may lie in their overconfidence in their ability to identify the better player, a point made in the Massey-Thaler paper. Sometimes players taken later in the draft turn out to be superior to the players taken early.

The table below lists the probability that each QB taken will end up better than the next QB taken in the draft (based on AdjYPA). For example, the 1st QB taken in the draft has turned out to be better than the second QB taken 81% of the time, and the 2nd QB taken has been better than the 3rd 38% of the time. (Note that some of the lower probabilities are due to the high number of consecutive QBs in the later rounds who did not have enough qualifying attempts. They are considers "ties.")

 QB Pick Pr(Better) 1 0.81 2 0.38 3 0.52 4 0.38 5 0.50 6 0.28 7 0.31 8 0.54

It appears that after the first QB taken, there isn't much certainty among GMs in predicting who will turn out to be the better passer. Teams are jockeying draft position, and paying a price to do so, for very small marginal probability of picking the better player. Perhaps there are times when the quality level drops off sharply between a QB and the subsequent one on the draft board, and trading up makes some sense. But this table should give GMs pause when considering whether to trade away next year's second round pick to move up 7 picks in the first round.

The data really tells two stories. Overall, when taking all 21 years of picks into account, the higher picks tend to have significantly better performance compared to subsequent picks. But when you look at the data in any one year, the differences aren't so clear, especially after the second QB taken.

Years as Primary Starter

Another way to grade draft picks, aside from Pro Bowl appearances or actual performance stats, is their years as their team's primary starter. The table below breaks out the average number of years a QB draft pick will serve as their team's starter by draft round.

 Round Yrs as Starter 1 5.6 2 4.0 3 1.8 4 1.0 5 1.7 6 1.6 7 0.2

Here is the same data, this time broken out by draft order.

 QB Pick Yrs as Starter 1 6.4 2 3.9 3 2.3 4 2.6 5 2.0 6 1.2 7 1.4 8 1.8 9 1.2 10 0.2 11 0.1 12 0.0 13 0.5 14 0.0

Years as starter may be a flawed comparison, however. Opportunity is everything, especially for QBs in the NFL. Top picks would certainly benefit from disproportionate opportunities. The years as starter figures for 1st round picks may be inflated because there are nearly no zeros. Teams might also stick with poorly performing QBs who were top picks longer than they deserved because of sunk costs. I think the better measure is what players do once they get their chance on the field. (Now that I think about it a little more, a better idea may have been to compare "years as starter beyond the first year.")

2 Responses to “Drafting QBs 2”

1. JTapp says:

There was an article a couple years ago in the Wall Street Journal where the author regressed NFL QB ratings on college performance.

The author (don't recall who) found that draft order was insignificant, and players who saw a final-year "bump" in their college QB performance tended to do worse in the NFL. Players who performed at high levels for multiple years in college were better on average.

He used it to predict that Jay Cutler and Matt Leinart would perform better than Vince Young in their careers.

2. Anonymous says:

I've always had the motto of "beware of the blow-up year" in the NFL draft. I never quantified it, but it seemed obvious to me. Thanks for letting me know someone did research it.