I’ve recently seen some fans use statistics to evaluate a play calling in a way that seems inappropriate to me. Here are three different ways I’ve seen statistics used to evaluate play calling:
1. Counting the number of times a playcaller ran or passed the ball on at certain downs and distances (e.g., on 3rd and 4, the play caller called a run play 68% of the time, etc.).
2. Examine the sequence of runs and passes to evaluate predictability. For example, one person criticized a play caller for utilizing the run-run-pass sequence way too often.
3. Identifying conversion 3rd down conversion rates for both passing and running at different distances–e.g., 45% success running and 55% success passing from 3rd and 4–and using this to make play calling decisions.
Do you guys think this is sound and appropriate way to evaluate a play caller–e.g., determining if they’re too predictable? What are sound and unsound ways of using these type of statistics to evaluate play calling?
I would say those stats give you tendencies. More runs than passes, the OC isn’t a West Coast guy. The stats wouldn’t necessarily tell you how effective a play caller the OC is, but I would guess you would rather have an OC who runs at least 50% of the time no?
Bottom line though, any person who only looks at stats or only looks at certain stats will never have the entire picture.
I have a question to your question. Do you think how predictable the play-calling is determines how effective or ineffective an offense is? Option teams could run the same handful of plays 90% of the time, but that don’t mean they are not effective. The Eagles love to run the pass-run-option (PRO) a lot this year (not a crazy percentage of time would be my guess), but their play-calling could be called predictable. It’s just hard to stop the play, maybe because of the different options.
I think identifying tendencies of a team (or players and coaches) are critical, but the best way to identify this, in my opinion, is to watch film. You’re not only getting what a player caller will do during certain downs and distances (and specific game situations), but you’ll also see what kind of formations and plays they run against certain types of defenses and personnel. All of that information is really important.
I’ll address option plays in a bit, but let me ask you, if you take out those options plays how would you answer this?
With option plays, by nature, they’re unpredictable, right? That’s how they work. The defense really doesn’t know precisely what’s going to happen until the very moment the offensive player with the ball makes a post-snap decision. At the same time, the possibilities are more limited then what a play caller can do from play to play, especially if they can also substitute players. Option plays might be harder to disguise, but I’m not sure about that, too.
Here’s another question that you and I have discussed over the years: Why aren’t option plays more successful or at least used more often in the NFL? I think there are several factors:
1. My guess is that assignments and discipline are the keys to stopping option offenses. With the right positions and assignments, a defense can really limit the effectiveness of option offenses. This is even more true if the defense is disciplined and possess sufficient athleticism. In college, the offense is viable because a) defenses have weak links that can be exploited; b) defensive coordinators might not have time or knowledge to prepare their teams. Finally, I notice a pattern that these type of plays/offenses can have some early success in the NFL, like the wildcat or read-option. Why? My guess is that NFL coaches haven’t figured out the position/schemes/assignments to defend these plays, yet. But it’s really only a matter of time. The read-option can still be effective, but far less than it’s initial introduction to the NFL. A good test of this theory might be Deshaun Watson and the Texan offense. I noticed at least one wrinkle in their mis-direction offense–formations and plays that reminded me of Paul Johnson’s triple-option offense. If my theory is correct, the offense will be far less explosive/effective in the next two years. (Another example: Chiefs offense this year. All their misdirection seemed to give defense a lot of problems, but by a certain point, the defenses seemed to catch up.)
I thought of another situation relating mostly to going for it on 4th down. Let’s say we use field position, score, and time remaining in the game, based on similar in the past, we can calculate odds of winning the game if you go for it. Let’s suppose the odds are 80%. My sense is that the analytics guys believe you should go for it almost all the time in this situation. The thinking here has never seemed sound to me.
Here is one reason that recently came to mind, and curious to know what you guys think.
Let’s say the probability is based on a 1,000 of these situations. And let’s say we then break up all these situations by games–that is, we separate when the situation occurred in the regular season or the playoffs, and we even separate the different playoff games–e.g., wild-card, divisional round, conference championship, and Super Bowl. Wouldn’t most of the situations occur during regular season games–primarily because there are more regular season games, and therefore greater chance for these situations to occur? (Or, is it possible that some conditions in the playoffs increase the likelihood of more 4th down situations? Maybe if you have more evenly matched teams, you’ll have more 4th down situations?…)
Let’s assume this is true. Playoff games are more valuable and important than regular season games, and each subsequent game in the playoffs is more valuable than the other–with the Super Bowl being the most valuable. (Or is that not right? In some sense, one could argue that every playoff game is equally important. Or, if the Super Bowl is the most important, you need to win all the playoff games preceding it to get to the Super Bowl.) What I’m getting at is that the football games are not equally valuable and one is the most valuable. You can win every game, but if you lose the Super Bowl, it will be a big disappointment. I think most people would rather have the 2007 Giants season than the 2007 Patriots season. (I know I would–without a doubt.) If this is true, then all 4th down decisions at the same field position, time remaining, and score are not all equal.
To put it another way. Football is very different from black-jack. The former has so many more variables for every given play. The games are different as I just discussed. If one believes that a team should go for it almost 100% of the time if the odds of winning are 80% or higher, isn’t that ignoring differences in the value of games? If you could played a 1,000 Super Bowls, over and over, it would make sense to play the odds, but obviously that’s not how things work.
(There are a lot of other variables that I haven’t mentioned–e.g., how dominant your team is on offense, the quality of the opponent, the nature of the game so far; physical and mental condition of your team and the opponent’s; tendencies of the opposing coach; field conditions, etc. It’s possible to factor in these variables, look at previous situations, but I would think the sample size would shrink significantly.)
By the way, to bolster my point about the way many variables should be accounted for and are not accounted for in analytics, the following anecdote of Mike Zimmer and LB, Scott Fujita, illustrates the reason play calls can’t just rely on probabilities:
I believe Zimmer’s ability to predicting the opposing offense goes way beyond the sequence of runs and passes (e.g., run-run-pass), or probabilities on a given down, distance, and score. He knows or has a good feel for how the OC is thinking–how an OC will attack specific defenses, how they’ll use early moves to set up later ones. I don’t someone who just knew analytics could do this. There’s other information that is crucial. And I think this “other” type of information would also be crucial in making 4th down decisions.
In terms of going for on fourth down, are you just against using statistics as rational to go for it on fourth down or are you against going for it on fourth down in general? I’m guessing that NFL teams go for it on fourth down three times are much now than they did like ten years ago. And yet NFL teams probably go for it on fourth down way less than statistics would say to go for it. So my point is yes, I’m sure stats can be “blamed” for this new trend (although like the word “trends” would indicate, part of it is just that a “trend”), yet because teams are still not going for it as much as stats say they should, teams are not solely using stats to determine whether or not they should be going for it on fourth down.
I believe teams should use stats, and the real issue is blending analytics and more qualitative information (e.g., football knowledge, psychology, etc.) I’m against the notion that analytics is far more important than the qualitative information, and that the former makes decisions relatively easy. I think it’s harder and more complex than they think.
Also, I don’t think analytics should be blamed for teams going for it more often. I actually think they should be credited for that. I agree teams are going for it way more than when we were growing up, and I feel like that’s a good thing. (Would you agree with that?)
But I feel like analytics advocates think that decisions should basically align with the probabilities, and they’ll criticize teams that don’t do this. I don’t agree with this. To really criticize a team, I think you’d have to examine analytics and the qualitative information in specific situations. The analytics guys I know don’t really do that or even speak as if this is important.