Last Word on the Draft and Wins

Recap

Through a series of regression models, I tested the hypothesis that more and higher draft picks translated into wins on the field. It became clear that neither conventional draft points nor Massey-Thaler surplus values were significant predictors of team wins the following season when previous year wins were taken into account.

One important point: It goes without saying that what really matters is the players who are picked. The most over-cited example contrasts Tom Brady, picked in the 6th round, and Ryan Leaf, picked #2 overall. The same principle holds throughout the draft. What is truly important is how good the chosen players are, and not how high the pick was. However, on average, players picked higher definitively outperform players picked lower. Over time the significance of any one player outperforming or underperforming his place in the draft should average out. Theoretically, in a large enough sample, draft picks (not the players) should correlate with team talent levels and therefore wins too.

Are draft pick values non-significant because they don't matter, or because we don't have enough data? Consider M-T values in 2005 as a case study. The coefficient for M-T value without accounting for previous year's wins is 0.86 wins per million $ of surplus value. The coefficient drops to 0.24 when accounting for previous year's wins, but we cannot have much confidence in that number because it is not significant (p=0.42).

Example 1

In 2005, the team with the most M-T value was TEN with $5.3 million in surplus. The team with the least M-T value was NYG with $2.2 million. (The average was $3.7 million.) That's a difference of $3.1 million between the top and bottom teams.

Using the 0.24 wins/$value coefficent, we get:
3.1 x 0.24 = 0.75 wins.

Between the top team and bottom team in terms of M-T draft value, we could estimate a difference of 0.75 wins. That's a very rough estimate because of the lack of statistical significance, but it is significant in NFL terms. 0.75 games would be a lot in a 16 game season.

Example 2

Let's do the same drill with conventional draft points in the same year, 2005. Without considering the previous year's record, the coefficient for draft points was 0.0022 wins/draft pt. The coefficient drops to 0.00040 when accounting for previous year's wins, but it is not signficant (p=0.36).

In 2005, the team with the most draft points was SF with 4049 and the team with the least was BUF with 603 for a difference of 3446. The average that year was 1898 points.

Using the 0.00040 wins/pt coefficient, we get:
0.00040 x 3446 = 1.37 wins.

1.37 wins is very significant in NFL terms in a 16-game season, especially considering that the difference between 9-7 and 7-9 sometimes means going to the playoffs or firing the head coach. We just can't have much confidence that the real number is 1.37 without more data.

Conclusion

This leads me to suspect that conventional draft points, M-T values, or other measures of draft "capital" are probably important to a team's fortunes. I simply don't have a database large enough (yet) to determine a significant coefficient. But in the case studies above give us an idea of the likely potential magnitude of the effects. Admittedly, it is possible that the effects are zero or near-zero (or even negative). There is a third possibility--that draft picks do matter, but neither conventional draft points nor M-T surplus are good measures of their value. Another system of values, one that is calibrated to the effect the picks have on wins, might prove to be the best method of valuing draft picks.

The effect of the previous year's wins is powerful. The constant churn of poor teams improving and good teams getting worse dominates the effect of either draft variable. Ignoring draft variables, teams with 3 wins tend to improve to 7 wins the following year. Teams with 12 wins one year tend to stumble to 9 wins the next. I suspect this is partially why it is difficult to distill the true effect on wins of draft picks, assuming there is one.

The reasons for the churn may include strength of schedule, salary cap considerations, and the draft. But luck and other mathematical reasons may also be at work. I plan to address this in a later post.

Do Draft Picks Equate to Wins? 3

Does the Massey-Thaler (M-T) surplus value of picks translate into wins? I repeated the analysis I did with conventional draft points.

First, I performed a simple single-variable regression with "DeltaWins" (the change in team wins from the previous year) as the dependent variable and M-T surplus value as the independent variable. M-T surplus was significant with an r-squared of 0.03. The coefficient was 0.86, which means that the difference between the #1 team and the #32 team in the draft equates to 0.34 wins.

This is far less than the r-squared for draft surplus (0.26). But keep in mind, M-T surplus is not as directly related to previous year's record as draft points. We already know that a team's previous year's record is dominantly predictive of the following year's win total, regardless of draft points.

To account for the other factors that produce the tendency for poor teams to improve and good teams to get worse, the previous year's wins was added to the model. M-T Value is no longer significant (p=0.42) and last year's wins (LASTWINS) dominates. Adjusted r-squared = 0.38, which was the same as for conventional points.

I ran several more regressions and got very similar results to the conventional draft point models. Adding various combinations of M-T values, previous year values, and cumulative values did not improve the r-squared found by using last year's wins alone, which is about 0.37. Below is the table of results. Asterisks denote significance and negative coefficents are in red.

If draft points don't matter (compared to other factors), then M-T values matter even less.

Do Draft Picks Equate to Wins? 2

Teams that have losing records tend to improve and teams that have winning records tend to get worse. There may be several reasons including the draft, but scheduling and other effects may also be factors. This isn't any revelation, except that it severely complicates the question of whether draft picks can be converted into wins.

My first regression was a simple single-variable model that estimated net wins based on a teams conventional draft points in the immediately preceeding draft. This time, the model will include the teams' previous year's record to account for the other factors that explain the tendency for teams to regress to the mean--better teams get worse, bad teams get better.

Because draft points are mostly a function of a team's previous record, I expected a large amount of overlap in their effect on future wins. To really isolate and understand each variable's effect on wins, it would take some creative "differential" regression--we need to run several different regressions and compare the results.

Including both draft points and last year's win totals in the model reveals the following. Last year's record is highly significant and the coefficient is negative (as expected--more wins last year means less this year). Draft points, however, is not significant (p=0.36) but still postitive. The adjusted r-squared is 0.38.


Regressing last year's wins onto the following year's wins (without any draft variables) results in a virtually identical r-squared of 0.38. This means that draft points do not contribute any significant predictive power to the model. The other factors that explain why teams regress to the mean dominate the effect of the draft. This is strong evidence that draft picks don't matter, at least when compared to other explanations for improvements in team wins.

I conducted numerous other regressions. For example, I used draft points from 1 and 2 years preceeding the current year, theorizing that draft picks require a year or two to mature before impacting a team's record. I also used the cumulative draft points from the preceeding years. Curiously, draft points from 1 year ago was significant at the p=0.10 level, but its coefficient was negative. In other words, teams with high picks the year preceeding the season in question tended to get worse.

The table below summarizes many of the regression models I ran. Each row is one model. Significance is denoted by asterisks (p=0.10, 0.050, and 0.01 levels). Negative coefficients are in red.



No matter how it's sliced, previous year wins dominates the model.

Do Draft Picks Equate to Wins? 1

My theory was that yes, more and higher draft picks typically mean increased success to their team. Keep in mind I am talking about the draft picks in the abstract--not necessarily the players themselves but simply the selection positions. (Credit goes to Doug Drinen who originally attempted to use regression to test the connection between draft points and wins.)

To test the theory, I created a variable called DELTAWINS which is change in total wins from year to year for each team from 2002-06. This was my dependent variable. For my independent variables I used conventional draft pick point values. In other words, I used each team's total draft points from the immediately preceeding draft to see if it significantly correlated with an improvement in each team's season win total.

Simply put, higher draft picks--and more of them--should lead to more wins the following year. If so, the regression coefficient for conventional draft pick point values would be expected to be positive and significant. Actually, the coefficient is 0.0022 and is strongly significant, with an r-squared of 0.26. (The constant for the equation is -4.2.)

The team with the #1 pick in each round, the team with the worst record, has a total of 3957 points. It would be expected to improve by:

-4.2 + 0.0022 x 3957 = 4.6 wins.

The team with the #32 pick in each round, the Super Bowl champion, has a total of 1020 points. It would be expected to get worse by:

-4.2 + 0.0022 x 1020 = -1.9 wins.

However, the regression has problems. First, the distribution of draft points is not normal--a bell curve. (Someone who points this out is not normal either, I admit.) It is a gamma distribution. One of the prerequisites for linear regression is that the variables's distributions are approximately normal.

For now, however, we'll ignore that problem because there is a bigger one. NFL teams with losing records tend to improve and ones with winning records tend to get worse. The draft may be one reason, as my hypothesis states, but there are others such as "strength of schedule" or salary cap boom/bust cycles--more on the other possible reasons later. For now, we'll accept that the model needs to account for a teams previous W-L record.

Robust M-T Draft Team Comparisons

Previously, I ranked teams in terms of how well they converted conventional draft points into surplus performance value as calculated by the Massey-Thaler (M-T) study. Teams were ranked by draft points, then ranked by M-T value. The difference between each team's rankings indicates how well each team played the Massey-Thaler game.

Comparing ordinal rankings is not the most sophisticated analysis. Statisically speaking, continuous variables are more robust than ordinal variables for analyzing correlations.

So to see if a team, or teams, stood out at maximizing M-T surplus value, I regressed conventional draft points onto M-T value. In other words, I performed a simple regression with draft points as the independent variable and M-T value as the dependent variable. The sample includes the '02-'06 drafts.

The "residual values" of the regression are the differences between the predicted team M-T values and actual M-T values. If teams did no trading, and made no effort to increase M-T value, then the residuals should be zero, or close to zero. If a team made an effort to maximize M-T value then the residual for that team would be positive. Negative residuals indicate a team has "squandered" value in M-T terms.

Below is the table of residuals. For comparison, the table includes the difference in conventional points ranking and M-T value ranking I previously used to compare teams. Also listed are the team wins for the sample time period '02-'06.

The Titans and Rams are at the top of the list, while the Redskins again are at the bottom. There seems to be a drop-off from Tennessee and St. Louis to the rest of the pack. Perhaps they are teams that have struggled with salary cap issues during the time period, then were forced to makee draft pick trades to minimize the cap hit of their picks.

The average number of wins for the period is 40. The big question is 'how do team wins correlate with M-T value over the same time period?' The correlation coefficient is 0.44 (r-squared = 0.19). About 20% of the variance in wins can be accounted for by Massey-Thaler surplus value.

This is very remarkable. Conventional draft points should correlate very negatively with team wins. The more a team loses, the higher the picks it gets. In fact, the correlation between team wins and conventional draft points is -0.65.

M-T values are not as directly related to a team's previous record as conventional draft points. However, the losing teams with the higher picks still start with more M-T value than the winning teams with later picks. Therefore, if there is no "M-T phenomenon," then we'd expect the correlation between M-T "effort" and wins to be weaker than between conventional draft points and wins, but also decidedly negative.

Instead, we find that the correlation is solidly and significantly postive. (The significance threshold is 0.15.)

Overpaid Free Agent Positions?

In the last couple posts, the relative draft value of each position was calculated by summing the conventional draft points for each position. The same can be done with Massey-Thaler draft surplus value for each position. Here is the result:

RB, WR, TE, and CB top the list. But what does that actually mean? It's not as clear as draft point value. The point value from the conventional draft chart represents the draft "capital" spent to select a player. But the M-T surplus value represents the typical performance of the pick above his contract cost. The value of the pick's performance is based on the cost of a free agent with equivalent performance.

So what the table above actually shows is how each draft position overperforms compared to free agents at the same position. So the positions at the top of the list are the positions at which free agents are the most overpaid relative to draft picks.

Take RB for example, the position at the very top of the list. The data basically says that teams are probably better served by employing drafted RBs rather than free agent RBs. One example that comes to mind is the recent experience of the Colts. Marshall Faulk was an excellent player throughout his initial draft contract but the Colts did not resign him. They drafted Egderrin James who performed practically at the same level, and for half the price. When James' contract came up, the Colts allowed him to leave and replaced him with another draft pick--Joseph Addai, who was more than talented enough to help the team win a championship.

The same might be said for WRs, CBs, and TEs.

"Day 1" Draft Values by Position

In the last post, I calculated the value that NFL teams place on each position during the draft. The analysis accounted for the number of players required at each position in terms of players on the field (1 QB, 2 WRs, etc.) and in terms of roster composition (3 QBs, 6 WRs). I considered all 7 rounds of the draft from '99-'06. But "impact" players, those who are most likely to become starters and help win games, typically come from the first 3 rounds, rare exceptions excluded. So I performed the same analysis, but this time only considered players taken on "Day 1" of the draft.

The first table ranks each position by 1st-day draft points per player on the field. For comparison, I also list the relative ranking of each position per player on the field from all 7 rounds.

The second table ranks each position by 1st-day draft points per roster composition. Again, I also list the relative ranking per roster composition from all 7 rounds.


There is no difference in the relative rankings of the positions between day 1 and day 2. That should have been obvious to me from the outset because the value of picks drops off so sharply from the first round.

How NFL Teams Value Positions

Using the conventional draft value point chart, we can evaluate the relative weight that teams place on each position. By summing the draft points "spent" on players at certain positions we can rank the relative importance of each position in the draft. Here is a list of each position and how many total draft points have been spent on them based on the '99-'06 drafts for all 7 rounds.

It's an interesting list because high-profile positions such as QB or RB are not at the top of the list as we'd expect. Many other analysts make a chart like this each year, but there is a flaw with ranking positions this way.

A far better measure of the relative value placed on each position in the draft would be one that takes into account the number of players needed at each position. We can allow for the difference in the number of players needed in two ways. First, we can divide total points by the number of players of each position typically on a roster. Alternatively, we can divide by the number of players of each position typically on the field.

Here is the table of position draft pick value per player on the field:

Here is the table of position draft pick value per player on a typical roster:


Expectedly, "singular" positions such as QB and RB move up the lists because there are fewer of them required and "multiple" positions such as WR or CB move down the list. A lot of conclusions can be drawn from these tables, but what surprises me most is that RBs are valued as highly as QBs. Conventional wisdom says that solid QBs are so much harder to find than solid RBs, but it appears teams choose to spend roughly the same amount of draft points on those positions.

Perhaps the "typical" roster has more than 3 RBs or less than 3 QBs, but I'd suspect the true average number of roster spots isn't that far off. Plus, we get the same result when accounting for players on the field, and it's been a long time since we've seen 2 RBs on the field simultaneously.

"Day 1" Massey-Thaler Comparisons

Teams could theoretically rack up "Massey-Thaler" surplus value with lots of bottom-round picks, but teams can only carry so many players. Rare exceptions aside, players from the top rounds are the ones that make a difference in the fortunes of a team. For every Tom Brady (6th round) there are several Mannings, Rivers, Palmers, Roethlisbergers, etc.

Let's compare total conventional draft values, M-T values, and their differences for each team for the first 3 rounds only, the "Day 1" picks. The rankings are listed below:

There are some exceptions, but we see generally successful teams at the top of the list and generally unsuccessful teams at the bottom. It's a rough analysis, but it leads me to believe there is some level of correlation between maneuvering for M-T surplus value and team success on the field. Keep in mind, bottom-feeder teams with the top picks have more M-T surplus value than winning teams with the later picks. Therefore, if there truly is a significant correlation between wins and M-T value, the direction of causation would be from M-T value to wins, not vice-versa.

Playing the Massey-Thaler Game

Are any teams actually aware of the phenomenon proposed in the Massey-Thaler Study? Perhaps one or more NFL teams has been conscious of the true costs of various draft picks and have maneuvered to maximize the value of their picks with respect to both performance AND cost.

To investigate I ranked each team's M-T surplus values and each team's conventional draft values for the last 8 drafts. To see if any team or teams stand out as "M-T aware," I subtracted the M-T ranking from the conventional ranking. The resulting number indicates how well a team has converted conventional value into M-T surplus value. For example, if a team was #32 in conventional value but was #1 in M-T value, the resulting difference of 31 would indicate that team is successfully trading picks to convert low conventional vlue into high M-T value. Low (negative) numbers would indicate the opposite, that a team is "wasting" its conventional draft position without earning much M-T surplus value. Here is the list:

One team does stand out at the top. Tennessee is certainly an outlier at the top of the list with a difference of 20. Admittedly, this is not a sophisticated statistical test. Ordinal rankings are rarely as powerful as proportional continuous values, but they are still very useful.

Taking a look at the bottom of the list we find teams that have generally enjoyed the luxury of top draft picks over the last several years, but, with rare exceptions, have not been able to convert those picks into wins. At the very least, we can say they have underperformed their expectations, such as in the case of Washington. Perhaps Houston can be excused due to its expansion status.

At the top of the list are teams such as Tennessee, New England, St. Louis, and Indianapolis. These are teams that we can generally agree have achieved consistent regular season success over the past 8 years, even considering recent rebuilding efforts in Tennessee and St. Louis. But one corollary of the M-T concept is that "rebuilding" should not be required if a team's salary cap is managed well.

Conventional Pick Value Rankings

In comparison to the Massey-Thaler values discussed in the previous post, NFL teams generally use a much simpler method to value draft picks. Every pick in the draft is awarded a point value relative to the 1st overall pick, which is given an arbitrary value of 3000 points. I'll refer to these as "conventional" draft values. Some teams' draft charts might vary slightly but not significantly, as shown by the M-T study. They plotted every pick for pick trade and they universally conformed to the chart cited 2 posts ago. While the chart may not reflect the true relative worth of each pick, it does reflect how the NFL teams value them.

Using the same database of draft picks from 1999 through 2006, we can tally the total "conventional" points associated with each team. Teams with high point totals are teams which had higher value picks, more picks, or both than most other teams. Here is the list:


The usual suspects top the chart. Arizona, Cincinnati, and Detroit have had the highest convential-value picks over the past 8 drafts. Cleveland could forgiven because it was an expansion team beginning in '99. Houston might have the highest per year value, but it didn't come into existence until '02.

At the bottom of the list I'd expect New England, Tennessee, Philadelphia, or St. Louis--teams that have been somewhat dominant over the past 8 yrs. But instead we find Tampa Bay and Miami--solid teams, but not quite as dominant. Just as in the M-T value rankings, we see that trades can significantly alter the values associated with each team. In the next post, we'll compare each team's M-T values and conventional values to see if any of them might be deliberately maneuvering to maximize M-T value.

Massey-Thaler Draft Values

As discussed in the two previous posts, the Massey-Thaler study quantified the surplus value of draft picks by estimating the excess performance of a pick above his contract cost. The excess performance is measured in dollars by calculating the price of equivalent performance by a free agent.

One of the first things I wanted to investigate was how each team compared in terms of "Massey-Thaler" values (M-T values) over the past several years. Does a team (or teams) stand out as one that capitalizes on this phenomenon? Does it correlate with performance on the field?

Using data from the published study and a database of draft picks from 1999-2006, I assigned each pick its M-T value. I then summed each team's total M-T values and ranked them. Here is the list:


What stands out at first is which teams are at the top of the list and which are at the bottom. Tennessee is first, which makes sense because it's had a particularly strong regular season record during the 99-06 period which puts them near the bottom of the 1st round where M-T values are highest. St. Louis, which is third, has also had strong records. San Francisco, however, has not, and neither have several of the other teams at the top of the list. This indicates there is something at work beyond where teams' first round picks land based on regular season records.

The Redskins place last, consistent with their reputation of poor draft and salary cap management. In fact, they are significantly worse than the next worse team, Miami. The difference between the top team, Tennessee, and the bottom, Washington, is about $12 million in "surplus value" over the past 8 years.

How significant is this spread? If teams never traded picks, the team with the top pick in the draft would 'naturally' have a $3.7 million surplus value and the team with the 32nd pick each round would have a $3.4 million surplus value. Over an 8-year period, a team that had the highest 'natural' surplus every single year would accumulate only a $3.2 million advantage over the team with the lowest 'natural' surplus. A $12 million spread means something is at work well beyond simply where teams find themselves in the draft order each year.

Are some teams actively trading picks to maximize M-T surplus value? Without ever reading the M-T study or even conciously trying, teams might be forced into doing so because of various salary cap constraints. In following posts, I'll examine this further.

Draft Pick Values

Below is a conventional draft pick value chart for the first round. Not all teams use the identical chart, but this is an example of a very typical one. In fact, the Massey-Thaler study plots actual draft day trades (pick for pick, no actual players involved in the trade), and the trades match the curve of this chart nearly exactly.

# PTS
----------
1 3,000
2 2,600
3 2,200
4 1,800
5 1,700
6 1,600
7 1,500
8 1,400
9 1,350
10 1,300
11 1,250
12 1,200
13 1,150
14 1,100
15 1,050
16 1,000
17 950
18 900
19 875
20 850
21 800
22 780
23 760
24 740
25 720
26 700
27 680
28 660
29 640
30 620
31 600
32 590

According to the chart, the #1 pick is worth three times the #16 pick, or the #15, #16, and #17 picks combined. At least that's how the NFL teams value the picks. As an approximate rule of thumb, the following year's value is worth one round less than an equivalent pick this year. In other words, a second round pick this year is worth a first round pick next year.

This chart gives us some insight into the thinking of GMs. Whether or not the chart is accurate, i.e. a #1 is actually worth 3 times a #16, is almost irrelevent. Teams believe it and act accordingly.

By tallying the conventional draft points by position, we can see how the teams value the importance and scarcity of skilled players at those positions. Although conventional wisdom is that good RBs can be found anywhere in the draft, are teams' actual picks consistent with this view? Just how prized are QBs relative to other positions? We can't say how truly important one position is relative to the others, but we can determine how the GMs value them.

Draft Analysis

With the recent completion of the NFL Draft, I thought it would be interesting to look into the stats available for the draft and see what kinds of trends or surprises can be found. A recent study on the draft by Massey and Thaler was very interesting. They found evidence that the salary cap has changed the value of draft picks.

The study covered several points. First, they found that top draft choices are overvalued based on the value teams put on draft picks according the the trade chart widely used in the NFL. Player performance falls off much slower than the value teams place on the picks. They also considered the steep contract amounts top picks receive and found that teams can maximize their draft values by trading top picks for multiple picks at the bottom of the 1st round and throughout the 2nd round where the cost and risk of the players are much lower.

I intend to address their findings, but first I want to take a step back and look at some simpler things we can learn from the draft.

I created a database of draft picks from 1999-2006. According to the conventional NFL draft value chart, each pick is assigned a relative value from 1 down to "Mr. Irrelevant." In addition, the database includes the surplus value calculated by the Massey-Thaler study defined by the average marginal performance of each pick above their contract price measured in dollars. Performance value is estimated by starts, pro-bowl appearances, games played, roster appearance, and lastly comparison to each player's salary after 5 yrs, when the player would by then have earned free-agent status. The database can be downloaded here.

Menu backgrounds