FEI 2019: New and Improved
22 Aug 2019, 07:00pm
by Brian Fremeau
This year, I’m introducing a new version of the Fremeau Efficiency Index (FEI) college football ratings, which were first conceived of in 2002 and developed in the years since as an effort to better understand the game and better evaluate successful teams and units. This latest iteration of the FEI ratings system is the outcome of a significant offseason project I undertook oriented around two key goals. The first goal was to produce a more accessible and intuitive format for the ratings, and the second was to make the system more predictive.
One risk with producing and publishing a college football rating system is to draw more attention to the ranking of teams than to the rating of teams. It’s natural to fixate on arguments like “Team A is five ranking positions better than Team B,” as if that number of ranking positions is more valuable data than the delta and meaning of the respective ratings of the two teams. But the ranking, of course, is just the order in which the ratings are sorted and doesn’t carry any inherent meaning of its own. That problem was aggravated by the uncertain meaning of my old FEI ratings format. What, for instance, did Alabama’s 0.326 FEI rating mean? How do we compare that number with Clemson’s 0.311 rating? Did the 0.015 delta between those ratings mean something specific? Was it referring to a percentage, or something else? Could it be easily interpreted and understood?
I tested a potential new version of the FEI ratings output last year with something I called Adjusted Possession Advantage (APA), but I hadn’t had the opportunity then to run APA for past seasons and scrutinize its utility as a predictive or retrodictive measure. I did like the format, however — a better way to cleanly represent a team’s per-possession advantage over another team rather than the nebulous old FEI format. So I committed to restructuring the output of the new FEI ratings to adopt this format. The stat pages for overall FEI ratings, offensive FEI ratings, defensive FEI ratings, and special teams FEI ratings since 2007 have all been updated on this site and share the same format, now more clearly defined as a per-possession representation of team and unit strength. I have also published at-a-glance team history pages of all of the FEI ratings since 2007 on my own site.
The formatting issue was only part of the problem, of course. Was FEI optimized to predict, and if so, was it any good at doing so? To be honest, I had never given this question its due until this past offseason. I have been utilizing FEI as a forecasting tool for many years and had made several efforts to produce better predictions for seasons and for individual games. But I needed to take a much more thorough approach and figure out what FEI is and what it can and should do. I was particularly focused on challenging my own assumptions. For instance, FEI had for many years weighted certain results more than others. Games between teams of similar strength, for instance, were given more weight in the formula than games in which one of the opponents was far outmatched. Did the weights need to be tweaked? Were the weights needed at all? Was the weighting methodology a feature or a bug? Further, how would an optimized FEI rating perform better than one that is not optimized?
I decided first to focus my work on creating a revised rating model that produced more accurate individual game projections. That is, how well did a given week’s FEI rating forecast the outcome of the next week’s games? Since FEI ratings require possession data inputs, I chose to focus my attention specifically on the predictive quality of FEI ratings to forecast games in the second half of seasons, from Week 7 through the end of the postseason. And to measure accuracy, I focused on measuring the delta between game scoring margin predictions and the actual game scoring margin results, then calculating the mean error of the predictions. The new FEI ratings were ultimately the most predictive, producing the lowest mean error. They were also the most simplified in terms of their calculation. None of the game weights I tested proved to be better than removing the weights altogether.
I then investigated the season predictive power of the ratings. Were weighted five-year ratings histories still solid baseline predictors of next-year success? Did transitional data inputs like returning production and recruiting ratings make those predictions even more accurate? Yes, and the new FEI ratings outperformed the old ratings every step of the way.
It’s important to point out that better predictive ratings and a better output format did not necessarily result in “more palatable” or “better eye test” ratings. Alabama still ranks ahead of Clemson in 2018, and in fact, several of the national championship teams since 2007 are not ranked No. 1 in that year’s new FEI ratings. This is definitely a feature, not a bug. FEI isn’t designed to name a champion, it’s designed to measure per-possession efficiency and to apply that measurement to make better predictions of future success. Dismissing a rating system that “looks wrong” is not the right approach, something I had to come to grips with myself. Especially given evidence of the new system’s effectiveness to do what it is designed to do and make better predictions, ratings that “look wrong” are an invitation to dig a little deeper and uncover an insight about what they actually do reveal.
With that introduction, I present the 2019 Projected FEI Ratings. Applied over the last three seasons, this preseason projection methodology resulted in successfully forecasting the winner of 69.2 percent of all regular season games. More importantly, and since new possession data will be recorded and will better inform the projection model throughout the year, the preseason Projected FEI ratings have a 76.8 percent win percentage for Weeks 0-2 when no such in-season data is available. (Betting lines successfully projected the winner of only 75.3 percent of games in Weeks 0-2 over the same span). For those early week projections, the mean error (average difference between projected margin and actual margin) was 11.6 points per game, a projection improvement of more than 1.5 points per game over the old FEI system.
Projected FEI ratings (PFEI) represent the per-possession scoring advantage a team would be expected to have on a neutral field over an average opponent. Projected losses (PL) represents the average number of losses expected based on the team’s PFEI rating and that of its opponents produced from individual game win likelihoods. Projected OFEI Offense ratings (POFEI) and Projected DFEI Defense ratings (PDFEI) represent per-possession advantages for each unit.
|36||North Carolina State||.19||4.5||.47||21||-.01||54|
|58||San Diego State||.05||4.0||-.34||107||.37||23|
|122||San Jose State||-.63||8.1||-.76||125||-.52||109|
|126||New Mexico State||-.75||8.2||-.61||119||-.81||119|
There isn’t much variance between these projections and the national consensus at the very top, and that’s not at all surprising. Alabama and Clemson have met in the College Football Playoff four straight seasons (and have won 11 of the last 12 CFP games played) and are best positioned to be in the running again for playoff bids this fall. Right behind them are programs that have the talent, recent history, and pedigree to be in the mix as well: Georgia, Ohio State, Oklahoma, and LSU. But after that, there are some notable differences between FEI and the national consensus.
No. 50 Texas Longhorns
Texas racked up a couple of impressive victories in 2018 (they went 1-1 against Oklahoma and beat Georgia) and has earned attention as a potential playoff contender as a result. Among other accolades, the Associated Press writers voted the Longhorns No. 10 in their preseason poll. So what is FEI seeing that is dragging them all the way down to No. 50? Their projection input data simply isn’t very impressive, and that’s all this system has to work with. Texas ranked No. 28 in last season’s final FEI ratings, coupling those great wins with a number of very pedestrian results against weak opponents. They also have very little production returning this fall in comparison to other possible contending programs. Even if FEI is blind to Texas’ 2019 upside potential, it would take a major leap in efficiency on both sides of the ball to comfortably forecast the Longhorns winning double-digit regular season games.
No. 27 Oregon Ducks
The Ducks are also a dark-horse playoff contender in the eyes of some prognosticators (No. 11 in the Associated Press poll), but FEI doesn’t see it. They have solid returning production numbers, but not a very strong recent history. The Ducks have only one win against an FEI top-30 opponent in the last three seasons combined, and nine of their 18 losses in the same span have come against teams ranked outside of the top 30. With games away from Autzen Stadium against Auburn and Stanford in September, they’ll have opportunities to impress, but will have to prove it before FEI will recognize that potential.
No. 22 Texas A&M Aggies
The Aggies don’t jump out as terribly overrated or underrated, but are worth discussing due to their unusually challenging schedule. They’ll face No. 1 Alabama, No. 2 Clemson, No. 3 Georgia, and three other SEC West teams ranked in the FEI top 10. If Texas A&M were projected to be an elite team in terms of possession efficiency, they would still struggle to navigate that kind of slate without suffering at least two losses. Since FEI projects them to be merely good and not elite, they are staring down five or six projected losses, with worst-case scenarios looming if they suffer any injury setbacks. No team in the FEI projected top 30 has anywhere near as many landmines to avoid.
No. 7 Mississippi State Bulldogs
No. 8 Auburn Tigers
Speaking of those SEC West teams, FEI is demonstrably higher on these programs in comparison to the national consensus, certainly as it relates to their ranking if not to their respective forecasted records. The Bulldogs project to have the nation’s third-best defense this fall according to FEI and still project to lose four times. Auburn projects to have a top-20 offense and defense (one of only eight teams with such projected balance) but has 4.5 projected losses due to their brutal schedule. The rankings may appear to be lofty for Mississippi State and Auburn, but the loss projections are right in line with national betting win totals.
No. 11 Appalachian State Mountaineers
There hasn’t been much buzz this offseason with regard to Group of Five teams that might make noise, in part because the volume of noise those non-power programs can muster isn’t likely to move the needle with the playoff committee. Appalachian State has solid possession efficiency inputs and will couple that with a very manageable schedule. They may not face a credible threat to derail an undefeated season until Halloween, so a 10-2 regular season (or better) projects to be more likely than not. They won’t rack up any wins that would turn the committee’s head to break into the final four, but they are best positioned according to FEI to grab a major bowl bid just the same.
It should be repeated that FEI isn’t designed to be infallible. It very obviously has blind spots to transitional intangibles that can’t easily be measured. And though it is optimized to predict the bulk of FBS games as accurately as possible, it should be fully expected to miss wildly on several teams. My gut says the No. 50 Texas projection is a prime candidate for a wild swing and miss, but it isn’t prompting me to redesign the system around that outlier. I’m excited to see whether FEI is mostly right or dead wrong on this and the other 129 FBS teams. Either way, I’m encouraged to acknowledge FEI’s blind spots while still celebrating its effective qualities. I hope you’ll share the same perspective and follow along throughout the year to examine and challenge the weekly ratings along with me here at Football Outsiders.
If this is what PFEI is all about – “a per-possession representation of team and unit strength,” as you wrote – then is it oversimplistic to take Team A’s PFEI, subtract the same of Team B’s, then multiply by the expected number of possessions to determine the difference in strengths? Consider that for Clemson-Georgia, the difference in PFEIs is 0.15; over a 20-possession game, that’s 3.0 in Clemson’s favor. But, 3.0 *what*, exactly? If it’s points, that looks about right. But please tell me if I’ve got it wrong.
You are exactly correct. Take the difference between PFEI ratings for each team and multiply it by an expected number of game possessions to arrive at a projected score. Note that since 2007, FBS games have averaged 26.6 total game possessions, 13.3 total posessions per team per game: https://www.bcftoys.com/notes/
Really terrific stuff, for an incredibly difficult task. The reality is that metrics are always going to have a hard time capturing, to cite the most obvious example, what it means when an 18 year old qb makes huge strides in his first 3 months of playing football games against other adults. The important thing is to measure what can be well measured, as well as possible, before factoring what can’t be well measured. Or something like that. In any case, to have a model which gets more than a 1% edge over the betting lines, for the first two weeks of the season, is impressive.
The college team I watch the most, Minnesota, had an intangible event last season, the week 8 firing of the defensive coordinator, followed by the promotion of the defensive assistant who eventually became the permanent replacement, which triggered an almost unbelievable immediate turnaround in defensive efficiency. They went from a defense which couldn’t even get lined up correctly to one that dominated opposing offenses, in the span of about two weeks. I don’t know if I’ve ever seen anything quite like it.
In any case, it is interesting to me to see FEI like the 2019 team much more than the conventional rankers. I wonder if that is due to the huge late season improvement last year, that the conventional rankers are ignoring, or more due to last year’s team being the youngest in the country, with nearly everybody back, or Minnesota’s rise, over the past couple seasons, in the recruiting rankings. Or all three factors? Or something else? Or are the conventional rankers such prisoners to the somewhat distant past that they would take multiple years to catch the movement of a program from the 50ish rank to the 30ish rank, whereas FEI may recognize it a year or two earlier? It’ll be interesting to see how the model performs for a mid tier program.