June 30, 2017

QUAR: The CFL’s new quarterback ranking method

THE CANADIAN PRESS

The first part of this series covered a brief review of the historical development of our game statistics which in their foundations date primarily to 1950. That was the year out west that the old Western Interprovincial Football Union (WIFU, and today’s CFL West Division) first compiled and published detailed season team and player totals for the basics of our game. That compilation first began in the East in 1954 and essentially that is what we still do today adding new categories and ways of evaluating game data to the benefit of our clubs, fans and the game itself.

Of particular importance in our three-down brand are passing numbers since they accounted for around 70 per cent of our offensive play calls in 2016 and have grown significantly in recent years. As a rule, we track those frequency numbers closely as they have changed dramatically since the “rushing” roots of our early days. Here’s how …

1. PASS TO RUN RATIO

Below are tables that provide a summary of the CFL’s total rushing and pass attempts along with our Pass-Run Ratio over time – a crucial measure and one that significantly affects game flow, officiating and Rulebook discussions. The first provides the Ratio based solely on statistical attempts, the second key CFL metric factors in Sacks Allowed, effectively pass calls prevented by defences by sacking the Quarterback before a throw can be made. This second ‘Play Call Ratio’ metric is actually even higher for passing scenarios when you consider that more than 60% of QB rushing attempts are actually scrambles out of the pocket after an initial drop back to pass call:

  CFL PASS-RUN RATIO *   CFL PLAY CALL RATIO **   CFL Pass   Compl
YEAR Rush % Pass %   Rush % Pass+ %   Efficiency   Pct
2016 2,851 33 5,826 67   2,851 31 6,204 69   98.0   68.3%
2015 3,007 35 5,460 65   3,007 34 5,827 66   92.3   66.0%
2014 3,212 38 5,188 62   3,212 36 5,627 64   85.3   62.0%
                   
2010s 20,180 36 35,757 64   20,180 34 38,343 66   91.0   63.8%
2000s 30,835 38 50,782 62   30,835 36 54,051 64   85.7   59.4%
1990s 34,133 37 58,469 63   34,133 35 62,386 65   79.2   55.4%
1980s 31,579 38 51,072 62   31,579 36 55,708 64   73.1   54.1%
                   
1970s 38,239 50 38,395 50             67.8   53.5%
1960s 41,068 57 30,690 43             69.1   52.1%
1950s 35,029 60 23,624 40             69.4   51.7%

*  Based on pure Rushing and Pass attempts only
** Based on play calls: Rush attempts vs Pass attempts & Sacks allowed (combined)

These data show that the trend toward passing accelerated out of the 1960s when 57 per cent of all play calls were runs, to just 36-37 per cent on average from 1981 through the early 2000s. Since 2014, an even more dramatic shift has occurred and this is very evident since 69 per cent of all attempts (statistically) in 2016 were pass calls. Clearly QBs are more valuable than ever now.

2. PASS EFFICIENCY RATING

A look at the league-wide Pass Efficiency Rating (PER), a fairly simple evaluation measure adopted by the CFL in 1996 (but  dating to 1973 and applicable to any year), shows historically that as more pass attempts were being made, passing results were not necessarily better in response to this change of offensive strategy.  However … since a low CFL-wide PER of just 66.9 in 1988, the increasing PER trend has remained and no more so than in 2016 to an all-time high of 98.0.

Why?  One of the many reasons lies in the measure itself which emphasizes completion percentage which has increased in EVERY decade since data was first compiled. Coaches across all eras work hard to achieve this kind of improved offensive success and this effect can often be measured statistically. The CFL-wide completion percentage of 68.3 in 2016 was in fact by far the highest ever single-season result in our history. Out of that analysis comes the fact that the CFL Pass Efficiency leader in 2014 (Bo Levi Mitchell at 98.0), produced exactly the same rating as only the average result in 2016.

These trends are the main among many reasons that it is now the right time to re-evaluate the way that the CFL measures and reports team and player passing performance in light of this historic shift. The other more important motivation to innovate and move beyond simple efficiency is the oversimplified calculation itself of just four basic indices based solely on attempts and on no other performance criteria at all:

  • Passing yards per attempt Completions per attempt
  • Interceptions per attempt TD passes per attempt

The Pass Efficiency Rating is further complicated by a mysterious maximum value of 158.3 among other mathematical considerations and the individual season results are hard to compare as the game changes over time.

3. MOVING FROM A PASSER RATING TO A COMPREHENSIVE QUARTERBACK RATING – QUAR

In order to make this fundamental evaluation shift, several advance research steps and establishment of important principles was mandatory in theory and practice.  In brief, here are the important considerations the CFL has made in developing what we have dubbed ‘QUAR’, a new and comprehensive methodology for evaluating CFL Quarterback Performance based on:

  • The fundamental step of integrating traditional Individual passing statistics with Team-based Offensive results.
  • Doing relevant calculations on a PER POSSESSION basis for each Quarterback enabling direct comparison.
  • Setting excellent (maximum) or poor (minimum) performance standards based on ACTUAL CFL-wide offensive statistics tested across a large sample size of game data.
  • Calculating QUAR using discrete ‘performance scales’ based on these actual CFL results and standards.
  • Retaining the original Pass Efficiency Rating but limiting that calculation to 40% of the new overall QUAR Rating (since basic passing data on a per attempt basis still has considerable performance value) and adding in more rating criteria.
  • New Criteria: Eight metrics which measure direct team offensive results while that QB was on the field.
  • The Process: Reflect and weight proportionately critical aspects of QB play using purely objective game statistics.
  • Essentials like team scoring, ball movement, win-loss results, conversions on 2nd down, first downs and more.
  • Integrating additional individual data: Tendencies to fumble, take Sacks or make positive contributions by QB rushing.
  • And finally, expressing it in an Index out of 100 for ease of comparison to an overall standard.

 

The goal at the end of the process is to get two things:

1) QUAR results on a scale of 0-100,

2) A ranked list of QBs that reflects overall individual-team performance

Methodology

After a full off-season of development and several prior years spent considering the objectives, the QUAR Index has been developed based on an in-depth study of CFL passing statistics covering the 2009 to 2016 seasons. This was completed with a substantial communication process with our clubs who were invited to comment/make suggestions from their perspective.

The research foundation of the study covered 603 games of 18,018 possessions and 93,561 plays over an eight-year period. Within that data population, there were 40,624 pass attempts to account for and test and hundreds of sets of season totals to compare and check for reasonableness. In short, this was anything but a quick study and consideration was given to what the end results looked like. For example, how does Bo Levi Mitchell’s incredible 2016 season compare to others in that or any year? The only data limitation (at this point) is the need to sub-divide passing and team offensive data by QB for each game – a significant task but one which we will work further back in time on for greater historical comparison.

QUAR Metrics

The QUAR Index is based on eight new metrics as follows and are shown with their associated weighting. One of QUAR’s features is that these weightings are adjustable to test for influence of one element over another on results. On average each club gets 15.0 possessions per game so multiplying by that factor gives a game equivalent total.

Metric  |   Weight    |   Description         |          Sample results or average

Offensive points 15%      Points per possession                   2 or 33 points on offence for a game

W-L % as Starter 10%      W-L percentage                           15-1-1, .912 win %

Net Offence             10%      Net yards per possession              0 or 420 yards for a game

First Downs   5%      First downs per possession            4 or 21 first downs for a game

2nd Down Conversions   5%      Percentage made                         48% (say 12 for 25)

Individual Fumbles   5%      Fumbles per possession                03 or about one every other game

QB Sacks Taken   5%      QB Sacks Taken per possession      0.15 or about 2.25 per game

QB Rushing yards   5%      Rushing yards per possession        2.00 or about 30 yards for a game

Pass Efficiency Rating 40%      Existing Pass Efficiency indices        Completions, Yards, Ints, TDs per attempt

QUAR  –  2016 Season & 2017 Week #1 Results

In our first week of action in 2017, the play call trends highlighted above carried right on with an even greater increase in the league-wide pass efficiency rate to 101.1 and tendency toward even fewer rushing plays than ever (28%).

  CFL PLAY CALL RATIO **   CFL Pass   Compl   Rush Pass
YEAR Rush % Pass+ %   Efficiency   Pct   Atts Atts
2017 130 28 327 72   101.1   68.9%   130 312
2016 2,851 31 6,204 69   98.0   68.3% 2,851 5,826
2015 3,007 34 5,827 66   92.3   66.0% 3,007 5,460

 

With that trend and the need to find a better way to rate QBs, let’s look at the first cut of results from a full season of data analysis for 2016 for Quarterbacks with at least 90 drives led (about six full games played). We calculate here our first QUAR results and then compare that data to what we just saw come out of 2017 Week #1 for our eight starters thus far:

(FOR UPDATED WEEKLY QUAR RANKINGS, CLICK HERE).

2016 DATA   Pass W-L No Off 1st Net 2D     Rush
CFL QB QUAR Effic Pct Poss Pts Dns Yds Pct QS Fum Yds
MITCHELL, Bo Levi 91.0 107.9 .912 239 539 377 6,809 50.1 18 4 60
REILLY, Mike 87.4 104.5 .529 237 470 424 7,033 53.6 31 6 406
HARRIS, Trevor 86.2 116.0 .450 131 236 198 3,990 50.7 23 6 126
JENNINGS, Jonathon 85.9 102.2 .667 239 479 390 6,912 52.4 30 6 363
COLLAROS, Zach 77.9 100.8 .300 115 240 189 3,336 49.5 13 4 39
NICHOLS, Matt 76.8 97.1 .769 184 367 279 4,755 44.8 21 2 90
RAY, Ricky 74.1 106.1 .333 113 192 186 2,823 48.9 25 1 53
BURRIS, Henry 73.3 98.0 .571 105 186 169 2,864 53.7 20 6 88
MASOLI, Jeremiah 69.0 93.6 .500 127 212 185 3,095 51.6 21 5 180
GLENN, Kevin 62.7 93.6 .333 125 188 185 3,037 49.0 21 3 57
DURANT, Darian 58.4 93.3 .267 195 248 276 4,629 45.6 42 9 308
CATO, Rakeem 55.8 100.7 .167 92 116 112 1,796 38.9 31 3 159
WILLY, Drew 50.4 92.2 .125 115 117 154 2,536 43.4 25 4 175

 

The QUAR leader for 2016 was Bo Levi Mitchell at 91.0 out of a possible 100 which clearly reflects the high quality season that he had for Calgary. It is notable that he did not lead the CFL in pass efficiency (107.9) trailing Trevor Harris, but what stands out is that all of his other attributes vaulted him to the top of the QUAR rating.

Take note also that the first six QBs in the ranked QUAR data … all led their teams to a playoff spot suggesting that they were the most effective QBs in our league across the 2016 season. And now for Week #1 results:

2017 WEEK #1   Pass W-L No Off 1st Net 2D     Rush
CFL QUAR DATA QUAR Effic Pct Poss Pts Dns Yds Pct QS Fum Yds
RAY, Ricky 90.8 126.7 1.000 13 30 23 515 48.0 3 0 4
REILLY, Mike 86.8 132.3 1.000 15 30 17 411 50.0 1 0 1
HARRIS, Trevor 82.5 103.9 .500 16 31 28 412 50.0 0 0 7
DURANT, Darian 79.0 108.7 1.000 13 17 16 331 42.9 0 0 9
MITCHELL, Bo Levi 75.1 102.4 .500 16 31 25 441 31.8 0 0 0
GLENN, Kevin 64.7 87.1 .000 14 16 25 395 47.8 1 0 16
JENNINGS, Jonathon 58.9 88.4 .000 15 27 27 355 41.7 5 1 32
COLLAROS, Zach 13.5 72.8 .000 12 9 12 221 23.8 5 2 0

 

Ricky Ray’s fabulous performance in Week #1 was the clear leader by this new rating method and yet again he did not lead in the old pass efficiency rating – that went to Mike Reilly. Each led their team to a victory last week though and this begs the question – why at 506 yards did Ray’s QUAR not approach 100? Well, he was sacked three times, did not run the ball much and Toronto’s 2nd down conversion rate of 48% was not excessive. That said, 90.8 is very, very good and the QUAR methodology makes being perfect a rare event as it should be. The old pass efficiency rating of 158.3 was exceeded many times in recent years so this simply sets a higher standard for excellence among our Quarterbacks.

Conclusions

To wrap up, the QUAR Index matches our fundamental philosophy in that it is a blend of innovation with the traditional Efficiency Rating and therefore seeks not eliminate it, but to enhance that original value with more elements into a fuller picture. To be sure, many more factors could included and these were debated and tested. Weightings could also easily be manipulated and like many ratings are somewhat arbitrary. But … they translate to the extent possible into a single and easily-derived number that can express comprehensively QB performance on a scale that ranges from outstanding to less so based in real data.

And lastly, the QUAR metric must be of value to our clubs and to our league overall in order to be an effective tool that serves to highlight the achievements of our players and put them into clear perspective. We have more ideas to implement in the days ahead including a greater emphasis on evaluation of defensive players.

Future articles will discuss QUAR a little more and also delve into the first results we have for our newly-minted “Passing by Range” statistics. Check out our Game Notes for updates on that new data analysis.