clock menu more-arrow no yes

Filed under:

Selected criticisms of Football Outsiders Almanac 2009

New, 1 comment

Make no mistake about it: Football Outsiders Almanac 2009 (by Aaron Schatz, et al.) is ostensibly a publication geared towards NFL fans. The "Pro Football Prospectus" name may be gone, and the first vestiges of college football analysis are in place, but the scope of this publication is, for all intents and purposes, geared towards readers interested in the pro game. That's not a problem for me, or an obstacle for reviewing it here. I have a great deal of interest in the NFL, and many of my posts tend to overlap between professional football and the college game.

That being said, I wanted to note several tidbits from FOA 2009 that either struck me as notable, or in some way intersect the Rutgers football program. This is by no means an in-depth analysis; if you want the full book, go to their website, and pluck down $12 for the digital download (it's a little more if you want a printed copy). I also have quite a bit more to say about related topics in statistical sports analysis (and my overall thoughts on the almanac) below, if you somehow have the patience to get through all of this. Let's delve on in.

The Rutgers football team does receive a fairly optimistic projection for 2009. Their two primary evaluation stats are Brian Fremeau's FEI, based on drive efficiency, and Bill Connelly play-based S&P+ rankings. Rutgers fares well in both, although slightly better (top 25 as opposed to top 30) by FEI. The team's offense is actually projected to be pretty good, and the strength of schedule comes in at a surprising #78 (eat it Cuse fans, with your #54 projected SOS). The college section is not nearly as comprehensive as the pro coverage, and was not intended as anything other than a bonus addendum with room for future growth and expansion. The tricky thing with this kind of analysis is that college football offers a much larger sample size than the NFL, but there's also far more variance owing to the wider talent and scheme disparities.

Additionally, Connelly also penned an essay appearing near the end of the book focusing on the effect of recruiting rankings. Maybe this is just nitpicking about an issue that I'm familiar with, but if there was ever an article practically begging, screaming for a regression analysis, it was this one. That methodology is FO's bread and butter. Compare the rankings to career starts, All-America teams, draft position, team W/L record, I don't care. SOMETHING.

Correlation doesn't prove causation, but it can't hurt. This just struck a nerve, because I have been waiting a while for someone to do a definitive regression analysis of recruiting rankings, and FOA 2009 would have been seemingly the perfect vehicle for that. I think, in the end, I'm going to just have to dig out my old SPSS notes and relearn basic statistics if I ever want to see something along these lines implemented, if even on a small and limited scale. The project admittedly would be very time and labor intensive.

The FOA 2009 book is a good resource for keeping track of the various changes from around the league during the offseason. For instance, there was a bit about how new Indianapolis DC Larry Coyer isn't going to run strictly a Tampa-2 scheme anymore, which eschews blitzing. Coyer favors smaller personnel too, but likes to bring more pressure at times, which I think might be to Eric Foster's benefit. if you're wondering how Brian Leonard will fare in Cincinnati this year, it could be a case of out of the frying pan (St. Louis), into the fire, if Cincinnati's offensive line doesn't improve. FOA 2009 projects that they will, but that's a cinch considering the depths that they're digging out from under.

"Cincy’s offensive line was the worst single unit in the NFL in 2008. Dead last in Adjusted Line Yards. Last in Stuff Percentage, and close to the bottom in blown blocks, runs over 25 yards, and power situations. That, and the franchise quarterback was a sitting duck even before he was knocked out for the season. Line coach Paul Alexander somehow kept his job, and now has some tools to rebuild. Rookie linemen have had outsized success in recent seasons, so Andre Smith should hopefully contribute right away, especially in the run game, provided he doesn’t wind up in the wrong city on game days. A blocking sled would do a better job than Eric Ghiaciuc, so even though center is a difficult position for rookies to master, Johnathan Luigs has to be considered an upgrade. Second-year men Anthony Collins and Nate Livings were solid when inserted into the lineup late last year–the best running games the Bengals had came with those two in the game. Andrew Whitworth is dependable at either guard or tackle, and Bobbie Williams continues to be very durable–a rare trait among Bengals lately."

One comment that I didn't understand: the Tampa chapter correctly notes that the Bucs are moving their offensive line to a zone blocking scheme. However, it later on the author claims that Jeremy Zuttah's skillset is more suited for guard. That's not true; it would be in a man blocking scheme, where the chief pulling guard is typically either really big and strong, or very athletic. Zeus has the athleticism to play either tackle spot in a zone scheme. He moved inside as pro owing to his lack of sufficient height to handle playing tackle outside in man.

The last Scarlet Knight that I want to focus on from the book is rookie receiver Kenny Britt. I understand that a lot of NFL fans don't follow the college game, so that's why rookie projections in the book will depend on second hand scouting reports, but what I take issue with is that a lot of those reports in general seem to be way off the mark when it came to Britt. As a player, legitimate knocks on him from a scouting perspective where that his catching technique wasn't great, he didn't have great acceleration, and didn't look entirely smooth when changing direction. That's fair; no prospect is perfect.

What's not is to merely Britt as raw. In fact, after Hakeem Nicks, Britt is as polished as any rookie WR (not counting special teams contributions). He already is a good route runner, with experience in a pro style offense. He's big, plays a physical style, and has a large wingspan to boot. Britt is already strong enough (with 23 Combine reps on the bench press, the second most of any WR), to laugh at press coverage, and knows how to beat the press.

"From my freshman through my junior year, every day at practice we had press coverage," Britt said. "We really stressed getting off the jam. His defense is mostly press coverage, and in our league we didn't get much press coverage. We didn't get pressed in games at all."
...
"Especially if you're the first look, the quarterback is coming off of you fast," Britt said. "If you can't get off the line, you're not getting the ball, you're not getting your stats and you're not helping your team."

The Tennessee coaching staff echoed those points.

"One of the hardest things for wide receivers is to get off the bump and run, and I thought Kenny was one of the top one or two looking at all those wide receivers of getting off bump and run," Heimerdinger said.

Added Fisher, "He’s strong and explosive, and he’s an out of frame catcher. For a tall guy, he can adjust to the low ball. He’s one of those guys that if you put the ball up, he can catch it."

Nor will he be a possession WR in the pros. I don't know how Britt's blurb/fantasy projection at the end of the book came to that conclusion; either based off his college stats, some off the mark scouting report, or accounting for the fact that Tennessee signed a free agent named Nate Washington whose game solely consists of running Post and Fly routes. They actually project Britt to get a lot of playing time and raw yardage, but very little in the way of big plays or yards per catch. It will depend on getting into camp on time, and staying healthy, but I think Britt plays earlier than expected, and develops into a vertical threat downfield. He's equally adept at going across the middle, or working the sidelines.

Two other notes when it comes to projecting Britt's performance: last year's prospectus introduced a metric (Pitzer, Rob. "Wide Receivers: Size Matters".) that's more along the lines of a traditional scouting analysis. The author plotted receivers' height and BMI on a grid, and noted that there tended to be four clusters of successful receivers. That is, they each tended to meet a certain preset profile: Tall (ex: Terrell Owens), Thick (ex: Hines Ward), Slight (ex: Torry Holt), or Short (Greg Jennings). Build-wise, the closest comparison to Britt was TO. For reference, being listed at 6'1, 184 lbs, Tiquan Underwood's BMI is slimmer than any successful receiver included in the feature.

This year's almanac introduced a statistical predictor more along the lines of the Lewin forecast for QBs, or Bill Barnwell's Speed Score (the idea being, adjust 40 times for weight) metric for evaluating RB performance at the NFL's Scouting Combine. It's called Playmaker Score; the idea being that it's not necessarily foolproof for evaluating potential stars, but does a good job weeding out obvious pretenders. The formula is yards per catch times touchdowns per game. If you're a Raider or Viking fan, you might be a little perturbed by their first round receiver picks being pegged as megabusts (for the record, I am a Harvin hater, and possibly the only person outside of The Black Hole that liked the DHB pick).

Britt falls in with Hakeem Nicks and Jeremy Maclin as players with relatively mediocre projections. Britt comes in a hair behind Nicks, but if you assume that he would have put up big numbers if not suspended against Morgan State last season, he would probably be slightly ahead of Nicks. As a Michael Crabtree skeptic, I think his draft status, and lofty Playmaker Score projection, will serve as useful test cases for ultimately settling the whole spread/scouting debate in the years to come.

The numbers for the Shotgun formation's overall level of effectiveness are jarring, and right in line with the prevalence and success of the spread in the college game (you can also look at each team's chapter and see how often they used certain personnel groups, like say 3-receiver sets). For the regular readers of this blog, I'd just like to clarify that my concerns with the spread more concern its usage on the college level, and its distorting effects on evaluating and projecting amateur players to the professional ranks. I think that some of the success on both levels stems from novelty, but there is something to be said on a scouting level too. These formations and schemes may give quarterbacks more time to throw, a better view of the field, or similar advantages, and might be purely superior solely on a field-neutral, individualized play by play, X's and O's level.

Lastly, I do not dispute Bill Barnwell's claim that the Giants' defensively line wore down significantly during the season. I still believe that he is significantly underselling the impact that losing Plaxico Burress had on New York's Super Bowl hopes last season. Yes, the Giants played a far tougher schedule near the end of the season. My contention isn't exactly falsifiable, given my belief that Domenik Hixon is poised to take the next step into a legitimate, starting-caliber #1 receiver this season. My contention though (based on admittedly a very small sample size), is that the team's passing game was far less effective without Burress.

I recognize the role that bias and intuition plays in analyzing questions like these. Oftentimes, if statistics lead me to an unintuitive conclusion, I often find that my set assumptions were wrong, and toss them aside. Occasionally though, there is value in stubbornness and a resistance to persuasion. I'm not dead set on these beliefs - but if I repeatedly keep considering a point, and still am unwilling to accept it; there's just a certain point where I will not at all. It's difficult to balance being willing to second guess your bliefs, and resistant to potential groupthink (there's a good quote about Matt Millen that I wanted to cite here, but couldn't find). Ultimately, you do have to trust your instincts and interpretations, as long as they have a track record of working reasonably well over time. It would not be the first time that they were wrong; or right, for that matter.


I have a love/hate relationship with advanced football analysis.

Don't get me wrong; I'm all for quantifying all unknown variables in sports, shattering unwarranted myths and cliches in the process. The game of baseball is so profoundly boring that statisticians can definitively pronounce the best strategy for scoring the most offensive runs over the long term. That's what will happen when a each "play" is individual and discreet, and it truly does pay off to be as selfish as possible. Some fans are nostalgic for earlier eras; my interest stems solely from the desire to quantify and demystify every minute aspect of the game. Otherwise, it's a slow, languid to the point of exhaustion game that bores me to tears, and offers no inherent appeal on its own merits.

Football and the other team sports are a bit of a different beast. For one thing, the baseball eggheads have a big head start. Legends Branch Rickey and Earl Weaver both valued on base percentage and not wasting outs. Meaning that, the statistical revolution in baseball was less of a sudden usurpation, and more along the lines of a restoring of the once and future royal line.1 What the so-called "traditionalist" know-nothings don't exactly advertise is that they cite statistics just as often as the statheads; they just cite the wrong ones, like runs batted in and saves.2

Even though the old curmudgeons won't admit it, they have lost.3 This stuff is mainstream now; Baseball Prospectus, Football Outsiders, and John Hollinger are featured on ESPN.com. Michael Lewis writes in the New York Times (not to mention, best selling paperbacks). The Times, the Washington Post, the Wall Street Journal, SI, they are permeating everywhere. Oakland never won a World Series, but Boston won two by adopting their strategies (with the benefit of a much higher payroll). Before George Steinbrenner bloated up the Yankees again at the start of this decade, Joe Torre won by importing pitch-selective veterans like Wade Boggs and Paul O'Neill. Although, the great pitching didn't hurt.

Gil Brandt of the Dallas Cowboys was an early innovator on the gridiron, using computer analysis for a leg up in the NFL Draft. As per FO's website, two watershed early works in the field, in spirit if not practice, were:

The two best books about football, both of which are unfortunately out of print, are The Hidden Game of Football by John Thorn, Pete Palmer, and Bob Carroll -- which inspired many of the early stats on this website -- and The New Thinking Man's Guide to Pro Football by Paul Zimmerman. The first book explores the statistical side of the game, the second book explores strategy and play on the field

Since its founding in 2003, Football Outsiders has helped to popularize statistical football analysis, maintaining a popular website, contributing to mainstream media outlets like espn.com, and publishing a series of preseason annuals.

Owing to their general hit-or-miss nature, I've never quite understood the appeal of the modern preseason annual. With his combination of groundbreaking insights, and biting prose4, I can get the appeal of a sharp, singular voice like Bill James, even though his heyday was way before my time. His earlier work still holds up well today. Take Baseball Prospectus for example, it has arguably been a victim of its own success, bleeding quality staffers over time, and inspiring a new round of formidable competitors. I thought that their best contributor by far was Nate Silver. Now that his interests have shifted elsewhere, I'm not nearly as interested in their work as I have been in the past.

In F.O.A. 2009, each team's chapter contains the same level of statistical insight. You'll learn about each team's formation tendencies, last season's strengths and weaknesses, and how those factors could influence the team in the coming year. Each group of text's largest component is a team-specific essay. In these, outsider Mike Tanier so profoundly towers over his colleagues that they inevitably suffer by comparison. I'm a die-hard New York Giants fan, but consider the Eagles-devotee's weekly Walkthrough columns on footballoutsiders.com an absolute must-read. The other contributors' essays vary the spectrum in quality. They are, by and large, more than acceptable, but not the absolute treat that would be a publication solely penned by Tanier.

It’s best to see advanced football statistical analysis at this point as a continual work in progress. FO and its ilk kind of vacillate on this point at times, but would ultimately agree with some variation of that claim. Its strengths lie in an ability to adjust for context; did team X play a particularly strong or weak schedule? Did team Y recover more fumbles than what they could have reasonably been expected to on average? Aaron Schatz says as much in the book's introduction.

If you want to see which teams are good and which are bad, which strategies work and which do not, you frst need to filter out that context. Down and distance, field position, the current score, time left on the clock, the quality of the opponent — all of these elements infuence the objective of the play and/or its outcome.

Indeed, my biggest qualms with the field are usually when its authors make statements that are far too definitive. Brusque comments often will invite hostility, but at least in the case of baseball, the utterances are warranted and deserved.

Perhaps in recognition of present limitations, groups like FO have met far less initial hostility from their baseball counterparts. Schatz has even collaborated with Detroit head coach (and Georgetown economics grad) Jim Schwartz on research projects. It might owe to the nature of the NFL; franchises are always on their toes, and looking for the latest competitive advantage (c.f. Bill Belichick, and his in-house skunk works). Teams' proprietary research may have more of a scouting element, but it undoubtedly exists, and the most successful franchises hoard their secrets.

Case in point for me, as a devoted fan of the New York Giants, would be the 2007 NFL season. In the run up to preseason, traditional media outlets were prognosticating gloom and doom for Big Blue. Their locker room was supposedly in turmoil. I was anticipating Football Outsider's preseason predictions as if I was crawling to an oasis in the middle of a desert, projecting all of my rational and irrational hopes and fears. Truly, they, as masters of logic and statistics, would right all of the offseason's journalistic wrongs, recognizing that my Giants were besieged by injuries in 2006, and were poised to return to the top of the NFC East. FO's statistical projection was for a middle of the road season (with corresponding room for improvement or cratering). However, to my dismay, the Outsiders were content to parrot conventional wisdom. Spurred by a pessimist Giants fan on staff, one preseason over/unders column even predicted that the Giants would finish with the worst record in football.

Flash forward a couple months. The Giants made the playoffs as a wild card, and tantalized New York and the country by standing toe to toe with the vaunted New England Patriots in their regular season finale, as the latter flirted with undefeated immortality. Most media analysis had the Giants as heavy underdogs throughout the playoffs, and FO was no exception. I didn't necessarily have a problem with that. What critics of statistical analysis fail to grasp is that the idea behind running a regression analysis is a functional, outcome-neutral process. Did Billy Beane and Paul DePodesta wake up one day, and suddenly decide that walks were the key to victory? No; they commissioned an analysis, with the intent to determine which statistics had the highest correlation with winning. If it turned out that sacrifice squeezes were the key to victory (or say, kneeling to win), that would be Oakland's meal ticket.

Similarly, Football Outsiders derives its core claims ("fumble luck", "curse of 370", etc...) from analyzing seasons of statistical data. All the conventional, scouting-based evidence in the world, and not just years of statistical trends, pointed towards the Patriots clobbering the Giants in Glendale. However, given that the NFL season consists of a very small sample size, statistical noise can and will occur, obscuring some truths in the process.

Most of the time, the team that appears to playing over its head, is, and will customarily fall down to earth at some point. Every so often, a statistical anomaly defies reasonable expectations. The Giants weren't playing over their head; the playoffs and the following season evidenced that they were a legitimately talented team, which finally had started living up to their potential. It may have been a mistake for FO (and to be fair, almost everyone else to boot) to not qualify their predictions more strongly. However, despite, well, being wrong in that particular instance, FO was behaving correctly. They followed a reliable process that has a better chance to produce meaningful future predictions, instead of junking everything in the presence of one naked exception to general inferences proven effective over time.

Another reasonable complaint is the sentiment behind Lito Sheppard's pointed critique of the YPA metric. It's important not to confuse these sorts of claims with those of the afore-mentioned sports traditionalists.5 Rather; the complaint is that, while the statistical project is admirable, it's not yet to a point of sufficient comprehensiveness and definitive authority. I want things to get more complex and esoteric. Get out of the mainstream, and onto the newsgroups. That new ball tracking technology? Awesome, and kind of aiming towards the same goal as the Outsiders' game charting project. Developments like say, the Lewin forecast, are incremental steps forward. I remain ever frustrated by these, owing to their lack of immediate causal, explanatory power, and won't be satisfied until they slay far more yet-unknown variables.

My suspicion is that most readers haven't quite taken my brazen leap into the abyss yet. For the novice and curious reader, the Football Outsiders Almanac 2009 serves as a more than good introductory primer into advanced statistical analysis of the National Football League. Even readers familiar with aspects of their work will likely learn a thing or two along the way. Once again, there's a digital download available for $12 on footballoutsiders.com, which additionally serves as a useful starting point for further insights on these topics. Another good place to visit on the net is the NYT's Fifth Down blog, which regularly features more content of this ilk, written by FO contributors, along with other, similarly minded analysts.


1 Making Billy Beane the Robin Hood of this analogy.

2 A profoundly absurd example, given that the statistic is only forty years old, but now manages to profoundly influence baseball strategy despite its utter lack of meaningfulness.

3 In fact, you'd think that these old timers would be all for recent, largely successful attempts by statisticians to better quantify defense and baserunning. Out were the Oakland A's and their army of roided-out freaks. In are the Rays, Rangers, and Mariners, with their emphasis on pitching, athleticism, and playing quality defense. Unfortunately, they still believe that the errors and fielding percentage were handed down on Mt. Sinai, and stubbornly insist on resisting all efforts to the contrary, despite their dwindling numbers.

4 The latter is what elevated James above, say, a Pete Palmer, who was arguably more influential in the sabermetrics community. You knew with one of James's publications, everything would be at the same level of quality, or at least within one standard deviation. His newer statistical work is somewhat lacking in comparison, being relatively behind the times as far as the internet age (a golden time for sabermetrics) goes. However, his tongue remains as sharp as ever.

5 I.e., the loose belief that statistics cannot quantify "heart", "grit", "leadership", and similar nonsense that serves to mask the terrors of intuition and bias. Don't confuse a perceived present failure in adequately quantifying certain factors with a permanent future inability to do so. Certainly do not mistakenly attribute the effects of luck and other incidental factors as essential by-products of mystical, immaterial internal qualitative properties of players and teams, imagined out of thin air as if they were a new variant of phlogiston, subjectively applied based on no set and static criteria whatsoever.