A few weeks back, I went to the doctor for my yearly uh-oh appointment. My doctor asked me what tests I wanted done and I told him I wanted them all to be safe. Doc told me that there was a new test for Herpes but he doesn’t recommend people getting it. I incredulously asked why and he responded:
[caption id="attachment_966" align="alignleft" width="300" caption="My doctor was McDreamy cool"]
[/caption]
“Well, the test can only tell whether your immune system has ever seen herpes internally, either the garden variety that a majority of the population has or the kind that we joke about at parties. Your immune system could have seen it and fought it. You could be transmitting it to others but have no visible symptoms. It also has a 5% false positive rate. So if I tell you the test came back positive what does it do for you? You can’t treat the bad kind if you do have it, you could have the harmless strain, or you may not have either. AT SOME POINT YOU HAVE TO SAY FUCK IT AND LIVE YOUR LIFE.”
After an emotional hug with a guy I will most likely never visit again, I thought to myself that some of the NFL analytics commentary I’ve been reading lately is like this Herpes test. Invoking game theory, parrots of the analytics community point out that run/pass ratios may be overly run-oriented because yards per attempt averages are greater than yards per rush averages. The logic is that in equilibrium these numbers should converge. Merely pointing out this fact doesn’t specify where exactly offensive coordinators are making mistakes. This simple test of offensive play-calling has come back positive, but what does it really mean?
**Kwesi Adofo-Mensah is pursing a PhD in Economics at Stanford University and originally hails from Philadelphia, PA. Prior, Kwesi attended Princeton University where he was a walk-on point guard for the men’s basketball team. After graduating from Princeton with a degree in economics, Kwesi spent eight years working in New York in commodity trading.**
The Economic Chess Match of Play-Calling
“Offensive play-calling is about probability.”
Those 6 words are the oft-repeated offensive creed of one Ron Jaworski. It appears that play-calling is more of an economic or statistical exercise rather than a football one. Comparing run/pass means is looking at play-calling balance from 3000 feet, but we need to inspect the actual decision facing coordinators; the decision that Jaws so frequently references.
The chess match between coordinators play is one of the hidden treasures of football. Encyclopedic play sheets represent a best response plan to an extremely complex economic “game” as much as they do ignored loved ones. In the moment, how does a coordinator decide to run or pass?
The coordinator must first analyze down and distance, field position, score of game, game clock, their strengths and weaknesses vs. the defenses, previous play calling this game vs. previous play-calling tendencies in other games, etc. Conditional on all these factors, he must come up with a distribution for possible defensive plays they may face (in game theory, the possible defensive plays are known as a strategy). In non-geek speak, “coming up with a distribution” means an offensive coordinator must be able to say, “I think there is a 50% chance they will play a cover-2 and there is a 50% they will play press-man with an extra guy in the box.” Given this first set of beliefs, the play-caller must then form another set of beliefs about the distribution of possible outcomes for each type of offensive play versus each type of defensive play he believes is possible(head spinning yet?, I spent my entire first year in grad school like that) . The optimal play will be the play that provides the best expected outcome against all these beliefs and distributions. Basically, equilibrium requires that both parties in a game are choosing the strategy that is a best response to their beliefs.
When Yards per attempt over Yards per rush Goes Wrong
In the huddle, calling run or pass isn’t as simple as passing averages X and running averages Y so we should have a predetermined number of W pass plays and Z run plays. The decision is rife with complexity and made murkier by uncertainty and the variability in potential outcomes. In addition, any potential distribution of run or pass outcomes would be seriously correlated to how often each occurred in the past and the ability to surprise the defense. Oh and the whole 40 seconds to decide things.
Let’s look through a quick incredibly simplified example of the decision coordinators face. Let’s assume we have an offensive coordinator named Andy R. Andy is mulling over a 3rd and 4 at his own 25 yard line. Let’s also assume that Andy is only capable of calling one type of pass play or one type of run play. For simplicity, let’s say the coordinator is 100% certain the defense will play Cover-2. Given the coordinators beliefs he estimates on average that he will make almost 5 yards rushing (4.95) and about 7 yards passing (7.05). Should he run or pass?
You really can’t answer yet because I haven’t told you enough. We need to drill a little further into the play-callers beliefs. Let’s look at Andy’s belief in the potential outcomes of running or passing alongside their likelihood of occurring.
These numbers aren’t out of thin air. Think of the run distribution as one coming from a good offensive line that gets consistent push off the ball and doesn’t commit holding penalties with a running back that doesn’t frequently make guys miss when he gets to the second level. The pass distribution mirrors an offense that has a 2012 Eagles- like sack rate, a tackle like Demetrius Bell holding anything that moves (sacks and holding represent the -10), a formerly-elite QB turned erratic from more hits than a Beyoncé half-time show, and intermediate and deep play potential when they strike(The 2012 Eagles! …I’ll finish this after I stop crying). The important thing to notice here isn’t the actual numbers but where they are concentrated. The running outcomes are concentrated in the 3-5 yard range while the passing outcomes are more spread out. Another way of seeing this is through the standard deviation of running outcomes(4.03 yards) being much lower than the standard deviation of passing outcomes(12.35 yards)
Hypothetical Probability Density Function of the Run and Pass Plays
Again, I’ll ask the question. Do you run or pass? The expected outcome of a pass is certainly higher (7.05 vs. 4.95). With a few quick calculations though, I think we would all agree that it’s time to run. It being 3rd down and football generally requiring the ball to score (unless Peanut Tillman is your corner), one way to fine define “success” is to call the play that best gives him a chance of getting a first down. Adding up all the plays that net 4 yards or more, we can see that running results in a first down 67% of the time while passing only moves the chains 52% of the time. Good evidence but not enough to convince the jury just yet. Even though passing may move the chains less frequently, is there a chance it leads to more eventual scoring by getting further down the field (in expectation)?
To check this question, I’ll look to see the Expected EPA (Yes, I’m calculating an Expectation of Expectation and No I’m not sorry). EP (Expected Points) is a dope concept from AdvancedNFLStats.com which shows the average next score by all teams from a particular yard-line/down/distance combination. Positive point values mean the offense on average scores before the opposing team and negative means the opposite. For example, a team with a 1st and Goal on the 1 yard line has an EP of 5.96 meaning that teams score touchdowns with almost certainty. Positive Expected Points Added means just that; you added to likelihood of your team scoring next. Expected EPA will show whether advancing further down field in expectation outweighs the lower odds of getting a first down.
Using this metric, let’s look at our two possible plays again:
Here, running results in an Expected EPA of .5845 while passing has an Expected EPA of .4886. Clearly the importance of maintaining possession outweighs the impact of advancing further in expectation. So by both measures, the offensive coordinator should call a run play here. As Jaws harps so often, the actual decision is about probabilities and distribution not just simply averages.
Flacco: Gridiron Game Theorist
It goes without saying that the example I created was a gross oversimplification in which I cherry-picked reasonable statistics to prove a point. In actuality, an offensive coordinator must formulate “beliefs” about a myriad of potential defensive schemes (although he can limit the possibilities by his own personnel packages and formation) and then decide a best response from his own dictionary-sized playbook. Ultimately, the QB may be the one to carry out this calculation at the line of scrimmage as he is told to audible to a different play if the defensive alignment dictates it. Case and point, we saw this very thing on one of the biggest plays in the biggest game of the football year. In a recent SI interview at his MVP after-party, Joe Flacco discussed what was going through his mind on that crucial 3rd and inches from the Baltimore:
“[Flacco] approached the line with four options: He could sneak; he could run Rice to the left; he could execute an option play; or he could throw to Boldin on the right side, about a 12-yard out…..[he went on to say] ‘Their formation took away the run, and it took away the option. I don’t like to sneak. I always think the quarterback sneak’s a crapshoot. So I only had one choice-throw to Anquan.’
[caption id="attachment_967" align="alignright" width="300" caption="No matter whatever hand-wavy math I lay down, Flacco had some donkey kong-sized ones to make that throw"]
[/caption]
If you’ll allow this South Jerseyan poetic license to translate another, I would say that Flacco came up with the following type of distribution conditional on the defense he believed they were in. In his mind, running the ball left with Rice or running the option had a 100% chance of gaining less than 0 yards. The crapshoot QB-sneak had a 33% chance of losing a yard, 33% chance of no gain, and a 33% chance of gaining a yard. Finally, the fade route distanced itself as the best option. Due to its short drop-back for the QB, a negative outcome is extremely unlikely(I’ll assume 0%). The outcome of this play would be decided by the ability of Flacco to make a catchable throw into single coverage to one of the most physical receivers in the NFL and the ability of Boldin to hold onto the ball with hands surpassed by no one save Lincoln Hawk. Even if Joe Cool 2.0 determined that all this summed up to a 34% chance of completing a 12 yard pass (far lower than his completion percentage overall), throwing the fade provided the best likelihood of success. Someone in his party remarked that it was risky but in Flacco’s mind, he was doing the safest thing.
More Herpes Talk
Diagnosing run/pass inefficiencies by looking at averages is akin to the seemingly ineffective current test for Herpes. Just like my wonky doctor said, a positive test could mean that someone actually has the disease but doesn’t know how to treat it. This is akin to the difference in yards per attempt for passing and rushing possibly indicating that NFL coordinators are clinging to a sub-optimal ratio of passes based on an archaic belief in “balance” but we haven’t necessarily identified situationally where they are erring. It could be that coordinators have not updated their beliefs as to the improving ‘risk-profile’ of passing with the popularity of short-passing, rules that penalize defenders for so much as breathing on receivers(until the Super Bowl), or no-huddle offenses limiting the ability of defenses to substitute thus severely shrinking the defensive strategy profile. It could also be that coordinators are scurred (exhibiting risk aversion) as they think a one-dimensional pass attack would open them up to more mentions on “Come-on Man”. The “pass-happy” crowd could certainly be right but we need to frame future analysis around the actual decisions faced by coordinators, the decisions highlighted in the example above.
A positive test could also mean that you have been exposed to the less harmful form of the disease or the difference in means could indicate that the coordinators believe there is some inherent value to the pass/run ratios currently employed that an economic model can’t necessarily quantify. We’ve all heard the theory that running gives lineman a chance to attack defensive lineman and the task of blocking in the passing game is made impossible with defenders pinning their ears back against a one dimensional offense. Coordinators could be slightly overcompensating to appear “balanced” but that could be reasonable due to the costs of failing (like turning your franchise QB into a crash-test dummy).
Finally, a positive test result could be a false positive just like the difference in means may simply be rational due to the potential higher variance in passing outcomes. Stocks outperform bonds on average but that doesn’t make bond investors foolish because stocks are inherently riskier.
Which of these is right? I have no clue. If I had to guess, I’d say coordinators haven’t updated beliefs concerning the risk of passing with favorable rules changes and wide-spread use of the screen game. Either way, I’m sure one of the football quants is currently in a monogamous relationship with Excel while he figures it out. The best answer I’ve seen is in analyzing EPA/type of play as Brian Burke of AdvancedNFLStats.com does here. I believe this works a proxy for how effective a team is mixing their play –calling. Until more refined analysis like this is done though, I’m guessing offensive-coordinators should simply heed my doctor’s advice and skip the Herpes test.
Back to the New England Patriots Newsfeed