QUAR: The CFL’s new quarterback ranking method

The first part of this series covered a brief review of the historical development of our game statistics which in their foundations date primarily to 1950. That was the year out west that the old Western Interprovincial Football Union (WIFU, and today’s CFL West Division) first compiled and published detailed season team and player totals for the basics of our game. That compilation first began in the East in 1954 and essentially that is what we still do today adding new categories and ways of evaluating game data to the benefit of our clubs, fans and the game itself.

Of particular importance in our three-down brand are passing numbers since they accounted for around 70 per cent of our offensive play calls in 2016 and have grown significantly in recent years. As a rule, we track those frequency numbers closely as they have changed dramatically since the “rushing” roots of our early days. Here’s how …

1. PASS TO RUN RATIO

Below are tables that provide a summary of the CFL’s total rushing and pass attempts along with our Pass-Run Ratio over time – a crucial measure and one that significantly affects game flow, officiating and Rulebook discussions. The first provides the Ratio based solely on statistical attempts, the second key CFL metric factors in Sacks Allowed, effectively pass calls prevented by defences by sacking the Quarterback before a throw can be made. This second ‘Play Call Ratio’ metric is actually even higher for passing scenarios when you consider that more than 60% of QB rushing attempts are actually scrambles out of the pocket after an initial drop back to pass call:

*  Based on pure Rushing and Pass attempts only
** Based on play calls: Rush attempts vs Pass attempts & Sacks allowed (combined)

These data show that the trend toward passing accelerated out of the 1960s when 57 per cent of all play calls were runs, to just 36-37 per cent on average from 1981 through the early 2000s. Since 2014, an even more dramatic shift has occurred and this is very evident since 69 per cent of all attempts (statistically) in 2016 were pass calls. Clearly QBs are more valuable than ever now.

2. PASS EFFICIENCY RATING

A look at the league-wide Pass Efficiency Rating (PER), a fairly simple evaluation measure adopted by the CFL in 1996 (but  dating to 1973 and applicable to any year), shows historically that as more pass attempts were being made, passing results were not necessarily better in response to this change of offensive strategy.  However … since a low CFL-wide PER of just 66.9 in 1988, the increasing PER trend has remained and no more so than in 2016 to an all-time high of 98.0.

Why?  One of the many reasons lies in the measure itself which emphasizes completion percentage which has increased in EVERY decade since data was first compiled. Coaches across all eras work hard to achieve this kind of improved offensive success and this effect can often be measured statistically. The CFL-wide completion percentage of 68.3 in 2016 was in fact by far the highest ever single-season result in our history. Out of that analysis comes the fact that the CFL Pass Efficiency leader in 2014 (Bo Levi Mitchell at 98.0), produced exactly the same rating as only the average result in 2016.

These trends are the main among many reasons that it is now the right time to re-evaluate the way that the CFL measures and reports team and player passing performance in light of this historic shift. The other more important motivation to innovate and move beyond simple efficiency is the oversimplified calculation itself of just four basic indices based solely on attempts and on no other performance criteria at all:

• Passing yards per attempt Completions per attempt
• Interceptions per attempt TD passes per attempt

The Pass Efficiency Rating is further complicated by a mysterious maximum value of 158.3 among other mathematical considerations and the individual season results are hard to compare as the game changes over time.

3. MOVING FROM A PASSER RATING TO A COMPREHENSIVE QUARTERBACK RATING – QUAR

In order to make this fundamental evaluation shift, several advance research steps and establishment of important principles was mandatory in theory and practice.  In brief, here are the important considerations the CFL has made in developing what we have dubbed ‘QUAR’, a new and comprehensive methodology for evaluating CFL Quarterback Performance based on:

• The fundamental step of integrating traditional Individual passing statistics with Team-based Offensive results.
• Doing relevant calculations on a PER POSSESSION basis for each Quarterback enabling direct comparison.
• Setting excellent (maximum) or poor (minimum) performance standards based on ACTUAL CFL-wide offensive statistics tested across a large sample size of game data.
• Calculating QUAR using discrete ‘performance scales’ based on these actual CFL results and standards.
• Retaining the original Pass Efficiency Rating but limiting that calculation to 40% of the new overall QUAR Rating (since basic passing data on a per attempt basis still has considerable performance value) and adding in more rating criteria.
• New Criteria: Eight metrics which measure direct team offensive results while that QB was on the field.
• The Process: Reflect and weight proportionately critical aspects of QB play using purely objective game statistics.
• Essentials like team scoring, ball movement, win-loss results, conversions on 2nd down, first downs and more.
• Integrating additional individual data: Tendencies to fumble, take Sacks or make positive contributions by QB rushing.
• And finally, expressing it in an Index out of 100 for ease of comparison to an overall standard.

The goal at the end of the process is to get two things:

1) QUAR results on a scale of 0-100,

2) A ranked list of QBs that reflects overall individual-team performance

Methodology

After a full off-season of development and several prior years spent considering the objectives, the QUAR Index has been developed based on an in-depth study of CFL passing statistics covering the 2009 to 2016 seasons. This was completed with a substantial communication process with our clubs who were invited to comment/make suggestions from their perspective.

The research foundation of the study covered 603 games of 18,018 possessions and 93,561 plays over an eight-year period. Within that data population, there were 40,624 pass attempts to account for and test and hundreds of sets of season totals to compare and check for reasonableness. In short, this was anything but a quick study and consideration was given to what the end results looked like. For example, how does Bo Levi Mitchell’s incredible 2016 season compare to others in that or any year? The only data limitation (at this point) is the need to sub-divide passing and team offensive data by QB for each game – a significant task but one which we will work further back in time on for greater historical comparison.

QUAR Metrics

The QUAR Index is based on eight new metrics as follows and are shown with their associated weighting. One of QUAR’s features is that these weightings are adjustable to test for influence of one element over another on results. On average each club gets 15.0 possessions per game so multiplying by that factor gives a game equivalent total.

Metric  |   Weight    |   Description         |          Sample results or average

Offensive points 15%      Points per possession                   2 or 33 points on offence for a game

W-L % as Starter 10%      W-L percentage                           15-1-1, .912 win %

Net Offence             10%      Net yards per possession              0 or 420 yards for a game

First Downs   5%      First downs per possession            4 or 21 first downs for a game

2nd Down Conversions   5%      Percentage made                         48% (say 12 for 25)

Individual Fumbles   5%      Fumbles per possession                03 or about one every other game

QB Sacks Taken   5%      QB Sacks Taken per possession      0.15 or about 2.25 per game

QB Rushing yards   5%      Rushing yards per possession        2.00 or about 30 yards for a game

Pass Efficiency Rating 40%      Existing Pass Efficiency indices        Completions, Yards, Ints, TDs per attempt

QUAR  –  2016 Season & 2017 Week #1 Results

In our first week of action in 2017, the play call trends highlighted above carried right on with an even greater increase in the league-wide pass efficiency rate to 101.1 and tendency toward even fewer rushing plays than ever (28%).

With that trend and the need to find a better way to rate QBs, let’s look at the first cut of results from a full season of data analysis for 2016 for Quarterbacks with at least 90 drives led (about six full games played). We calculate here our first QUAR results and then compare that data to what we just saw come out of 2017 Week #1 for our eight starters thus far:

(FOR UPDATED WEEKLY QUAR RANKINGS, CLICK HERE).

The QUAR leader for 2016 was Bo Levi Mitchell at 91.0 out of a possible 100 which clearly reflects the high quality season that he had for Calgary. It is notable that he did not lead the CFL in pass efficiency (107.9) trailing Trevor Harris, but what stands out is that all of his other attributes vaulted him to the top of the QUAR rating.

Take note also that the first six QBs in the ranked QUAR data … all led their teams to a playoff spot suggesting that they were the most effective QBs in our league across the 2016 season. And now for Week #1 results:

Ricky Ray’s fabulous performance in Week #1 was the clear leader by this new rating method and yet again he did not lead in the old pass efficiency rating – that went to Mike Reilly. Each led their team to a victory last week though and this begs the question – why at 506 yards did Ray’s QUAR not approach 100? Well, he was sacked three times, did not run the ball much and Toronto’s 2nd down conversion rate of 48% was not excessive. That said, 90.8 is very, very good and the QUAR methodology makes being perfect a rare event as it should be. The old pass efficiency rating of 158.3 was exceeded many times in recent years so this simply sets a higher standard for excellence among our Quarterbacks.

Conclusions

To wrap up, the QUAR Index matches our fundamental philosophy in that it is a blend of innovation with the traditional Efficiency Rating and therefore seeks not eliminate it, but to enhance that original value with more elements into a fuller picture. To be sure, many more factors could included and these were debated and tested. Weightings could also easily be manipulated and like many ratings are somewhat arbitrary. But … they translate to the extent possible into a single and easily-derived number that can express comprehensively QB performance on a scale that ranges from outstanding to less so based in real data.

And lastly, the QUAR metric must be of value to our clubs and to our league overall in order to be an effective tool that serves to highlight the achievements of our players and put them into clear perspective. We have more ideas to implement in the days ahead including a greater emphasis on evaluation of defensive players.

Future articles will discuss QUAR a little more and also delve into the first results we have for our newly-minted “Passing by Range” statistics. Check out our Game Notes for updates on that new data analysis.