Statistics Are Dumb
Jonathan Willis
November 16 2011 08:18PM
Yes, I wrote that title. Not only did I write it, but I mean it.
“Dumb” basically means lacking intelligence. Most NHL statistics, frankly, require no intelligence. Let’s look at the complicated math involved in the basic statistics on NHL.com’s summary page.
- Goals: Watching and counting
- Assists: Watching and counting
- Points: Adding goals and assists (or, alternately, watching and counting)
- Plus/Minus: Watching and counting
- Penalty minutes: Watching and counting
- Power play goals: Watching and counting
- Shorthanded goals: Watching and counting
- Game-winning goals: Basic addition, watching and counting
- Overtime goals: Watching and counting
- Shots: Watching and counting
- Shooting percentage: Basic division, watching and counting
- Time on ice: Watching and counting
- Shifts per game: Watching and counting
- Face-off percentage: Basic division, watching and counting
Basically, if you’re capable of turning on your TV and counting things, you can create almost any NHL statistic from scratch. If you’re capable of doing that and then later using the division key on a calculator or computer, you can create any NHL statistic from scratch. I’ve listed a bunch above, but they’re all basically the same – goalie stats involve counting shots, goals and minutes played, real-time statistics all consist entirely of counting, and so on.
What about all those fancy advanced statistics that get thrown around? Scoring chance percentage, Fenwick, Corsi, EVPTS/60 – those are more complicated, right?
No.
Scoring chances involve somebody watching the game and counting. Scoring chance percentage simply involves taking the number of good scoring chances, and dividing them by the total number of scoring chances. In other words, if you know how to count and can press a division key on a calculator, you can have a firm grasp of this “advanced” statistic.
What about Fenwick? Well, you take those shots and missed shots that somebody counted up, and then you add them together – just like plus/minus. Corsi is the same thing, except that it includes blocked shots as well.
Points per 60 minutes of even-strength ice-time (or EVPTS/60) is almost as simple – one takes all the points a player scored at even-strength, and divides them by ice-time at even-strength to create a scoring rate. It is, once again, counting and pressing the divide key on a calculator. Pretty much as simple as can be.
But let’s go back to scoring chances. In an article yesterday, I did something audacious – I added up scoring chances for and against for Oilers’ defensemen. In the comments section, Robin Brownlee jokingly advised one commenter (i.e. not me) to do the following:
Your only option is to watch the games and draw your own conclusions.
Personally, I think that’s a great idea for everyone. It’s a little obvious, perhaps, but still a great idea.
It is, after all, what I do. I look for specific things – which players play the best opponents, what part of the ice players start their shifts in, how often players helps their team create a scoring chance, and how often players make mistakes that lead to chances against. As a rule, I try and get a gut feel for the game based on those things (others too, of course – which players take bad penalties, who wins faceoffs, etc.). Rather than watch the game multiple times and count those things up, I rely on others to do it – the NHL keeps track of a lot of these things (as mentioned above, by watching the game and counting) and people like Dennis King and Gabriel Desjardins catch the rest. I find that a firm number (i.e. Eric Belanger won 7 of 10 faceoffs) is better than my gut feeling (Eric Belanger wins a lot if faceoffs), so usually I’ll use the firm number instead of simply repeating my gut feeling. It’s the same thing with scoring chances – I know that Cam Barker’s getting heavily out-chanced by his opposition, but rather than say something like “man, that Cam Barker looks really bad” I’ll look up Dennis’ work and say “Cam Barker has been on the ice for 35 chances for and 49 against, which is one of the worst totals on the Oilers!” Afterward, rather than add “and he looks bad even though he’s got an easier job than other defensemen” I might use a number – like how many times he’s started shifts in the offensive zone, or how often he’s played the other team’s top line.
Of course, when I say “Barker has been on the ice for 35 chances for and 49 against” rather than “man, Cam Barker looks really bad,” someone comes along to tell me I should “watch the games.” I laugh, because it’s funny.
Jonathan. I rely on you for my advanced stats, because while I can watch, count, AND use a slide rule, I'm usually bombed by the middle of the second period.
Thanks for explaining the "advanced stats" (Fenwick, Corsi et al)
~I feel it's time the Oilers should trade Shawn Horcoff.~
@Robin Brownlee
It wasn't a shot at you, Robin. I'll make that clearer in the post.
And just in case anyone out there misinterprets the comment:
I was not and am not offended by Robin's comment. I simply found it interesting that he (in jest) contrasted statistics and watching the game, because to me you can't have the first without the second.
11 posts in and nearly all by the big kids at the table. Where's Gregor and Wanye? Don't they want to get in on this...
not to get too meta-discursive here or anything... but... I take it for granted that with the exception of a few extremists most people around here enjoy the diversity of analysis from the contributors and appreciate all ON voices.
one of the best qualities of this site is its ability to blend the high and the low; the serious and the silly - often in the same article!
for what its worth, it seems to me that Brownlee's taking the piss out of us luddite commenters for not being able to grasp math as much if not more than he is you adv. stats guys. I certainly take no offence. I find the running narrative hilarious and so does my Walkman
Willis is the most patient person alive.
I like to watch.
Chance the gardener
@Romulus' Apotheosis
A classic. It dates me. Back to the topic. I really just enjoy watching the games. Analysis is for when the team is not performing particularly well. Unfortunately, I have spent too much time analyzing and not enough cheering for the past few years.
Haha.. delineates:)
http://www.youtube.com/watch?v=Dr_3S4GcZNo
Nobody seems to be struggling for goaltending right now. If Khabibulin remains in that .930 range at the deadline there would be some interest i'm sure. If he keeps this up for most of the year the Oilers will need him most....being close or in the playoffs and all.
Work Khabby like a rented mule this season or ride him till he breaks down. He plays till he loses or whenever there are back to back games. Give him what he wants, he's only played 70 gms the last 2 seasons combined. To heck with waiting for Dubnyk to get his ship together. Everytime i see him starting in goal i say to myself, well, there goes another two points.
One of the things I like best about oilersnation is the combination of stats and inside info you don't really get anywhere else. Keep up the good work boys I come here multiple times a day !
HERE HERE!
~Insert Sam Gagner should be traded quote here~
@Michael
I suspect most people who quote that phrase don't know its history. Wikipedia has a fairly strong write-up on the subject.
It was originally in reference to three types of witnesses at a trial: liars, damned liars, and experts, with the meaning being that an expert witness can use the persuasive power of statistics to bolster an otherwise weak case.
It's certainly true that statistics - particularly when carefully selected by those with knowledge and given to those without - can be used in that way.
For instance - I might say "Cam Barker (or Theo Peckham, or Jeff Petry) leads the Oilers with a plus-3 rating; he's obviously their best defenseman!" and if you knew nothing about hockey, or hadn't seen the Oilers play, you might accept me at my word because of my use of a simple statistic. Of course, I'd be lying - by eye, or by comprehensive statistics, Barker's clearly been one of or the worst Oilers defenseman.
What I'm here discussing are comprehensive statistics - in other words, getting as much of the picture as we can by using stats that directly relate to things we see when we watch the game. In other words, the exact opposite of the quote you just used.
I'm glad you mentioned it, though - too many people use that quote as an excuse to ignore statistics altogether, when really the point being made is the need to examine them thoroughly to check for validity.
Unless, of course, you think originally that the quote meant one should ignore all scientific experts, which would obviously be crazy.
In short: Skepticism is healthy, ignorance is not.
For context:
"It's a real shame about Barker... getting that draft pedigree for so cheap could have turned out awesome. I was willing to withhold serious judgment until the break... but now that he is out... I'm looking forward to not thinking about him for a month. we'll see if he finds some enthusiasm coming off the injury. What is really bizarre though and must give the stats folk headaches is the +/- situation... Barker, Peckham and Petry (the three D Willis correctly isolates as the weakest on the team - although Petry obviously is high on this particular list) somehow are tied for first on the team at +3!! That makes no sense to me... Didn't we just have an article about how shot quality doesn't matter or something? If shots/scoring chances are the crucial marker underlying success... how come the players with the worst shots/chances for/against have the highest +/-??? doesn't that imply shot quality has an effect? is there some other missing link that can explain this inversion?? I'm so confused... Brownlee, crank up the Betamax... we've got some numbers to crunch!"
Complete quote in response was:
@Romulus' Apotheosis
"I'm out of fingers and toes and the Bronte 5000 is down for maintenance. You're on your own. Your only option is to watch the games and draw your own conclusions."
Actually, one of the reasons I enjoy Jonathan's work is that he does the stats work and expresses it in a manner that dummies like me can understand.
And I think Jonathan also gives fair view to what we see on the ice, too. A combination of the two has greatest value imo. Bill James once admitted Lou Brock was an astounding leadoff hitter while also being a poor candidate for the slot in the order. I think that's the kind of balance Jonathan brings to these discussions.
Keep up the great work, Jonathan.
@Jonathan Willis
What I find interesting is leaving out the first part of the quote, which was offered in response to somebody asking me to "crank up the Betamax." That seems like a curious choice, if one is interested in context.
I don't consider watching the games and keeping tabs of some advanced stats for a broader picture as an either/or situation.
@ Robin Brownlee:
I included the link, and figured most readers would be familiar with the discussion yesterday. Still, your inclusion of the original comment and response in its entirety should drive home the fact that it was a joking exchange.
@Lowetide
Thank you.
But I would never have started looking at hockey statistics (or blogs, for that matter) if I hadn't come across your work. I suspect a lot of fans can say the same thing.
C'mon Willis....just update the pool.
Those are statistics :)
@ DSF:
Yeah, it kind of slipped through the cracks this past weekend. But we'll all get to see your name way up high on the list this Sunday :)
@Jonathan Willis
Careful L.T.
While I am sure Willis is being somewhat sincere, if he asks you to start his car for him, run for the hills. After first pi$$ing off Spec and then Brownlee I would be more than a little cautious. Those guys don't get mad ...
how long do we stay in here?
http://www.youtube.com/watch?v=hUsiUCzPyoA
How does a list that delineates those who know what they're talking about "slip through the cracks"?
Just asking.
Well said Willis.
Good defence of what you do.
Makes sense to me.
WILLIS, YOU SHOULD REALLY WATCH AN OILER GAME BEFORE OPENING YOUR MOUTH!!!!!!!!!!!!!
Ive read your opinions on Dubnyk but I gotta ask.
Do you think the Oilers are getting any decent offers for Khabbi at the moment?
True, the stats in sports do involve a lot of counting. I'm taking a stats course right now at UofC. Probability is what is more interesting and way more prevalent.
What is the probability a team can come back from and 0-3 deficit? (in a game or series).
What is the probability the Flames can get back to .500? :)
What is the probability the Flames will make the playoffs? :)
How is QualComp calculated? Is it counting as well?
@ RKD:
QualComp's a little more complicated, to be sure. Personally, I use it as a shorthand, but much prefer to look up games and see who is actually playing against which lines.
QualComp takes the plus/minus of every player on the ice (adjusted for team) and essentially multiplies it by ice-time. Corsi QualComp does the same thing, but instead uses shots, missed shots, and blocked shots - the idea being that because there are many more shots than goals, the bigger sample gives a more accurate reading.
It's not a perfect system, but after ~30-40 games it closely resembles the match-ups I've seen.
Naturally, a better way to evaluate Quality of Competition is to watch how coaches match lines, or to go back game to game and check out the head-to-head ice-time.
@The Farmer
Don't let me down here, ON.
DRUM ROLL PLEASE.................................
@Jonathan Willis
HERE HERE!!!
Multiple times a day?? [looks around at the audience] Well hopefully not at work...am I right????? [Waits for laughtrack], I was wondering why you seem to have a few sticky letters [fist pump for the punchline]
HEYOOOO!
But talking about the article, I don't have a problem with advanced stats if they are used but it is when they are used to argue against or for a player when if you watch you can clearly see a different game then the stats suggest (Not saying you do this Willis because I don't think you do). But for example advanced stats guy say Ryan Jones sucks and isn't a hockey player well 18 goals shows me he is, in my opinion a goal is a goal and they win hockey games.
And one reason I am against advanced stats is that they don't measure hockey IQ, or heart/emotion. And people can laugh all they want but hockey is played with emotion and it plays a huge role but there is no formula to go along with it.
It isn't so much that they're dumb. Statistics, when compared to players flying down the ice, firing crisp passes, going upstairs with a backhand…are just frickin' boring. Watch, and count if you must.
@Jonathan Willis
Hey Willis,
Do you stats Guru's do any regression analysis for past seasons (Say take the first 30 games of the 2010 season or all 82 for the 2009) on the advanced and basic stats to see if any of it can accuretly model performance with any certainty say an alpha over 80%?
Or are you saying this is more of a way to qualify how you felt you saw someone perform?
Plus minus would be real relevant if the player who was penalized received a minus five if they scored while he is in the box, and plus one (the rest goes to those on ice ) for a shorty.
Also...
The person drawing the penalty gets a plus five if their team scores.
Of Course... Plus five is too high, but there is a correct number. A moron taking bad penalties costs the team, and his actions are not accounted for under the current system.
Plus minus should be off the board when the goalie is pulled. It makes no sense to hammer Shaun Horcoff the last two years for his plus minus, when he was the only guy called on for offense in the last minute, when down a goal. (That was more the case two years ago.)
And, if it doesn't apply during PP, why then when there's a man advantage and no goalie?
iPad edit
Whoops I'd like to say I meant to do that ;)
@ Oilcan:
And you'll find a lot of people who agree with you.
Statistics, at their core, are an attempt to separate what really matters vs. what doesn't. Your own comment bears that out, actually - you view goals as a key measure, and a lot of people agree with you.
If one player scores 30 goals and another 15, even if the 15-goal scorer shows a lot of heart and emotion and the 30-goal scorer is coolly efficient, you're likely to place more value on the 30-goal scorer, right? That's because you know that goals matter a lot - probably more than a player's personality. For myself, I care a lot more about the result (30 goals) than about the heart and passion involved.
Basically, that's all that advanced statistics is - except that, because goals are relatively rare events (the best player in the league might hit 60, most players are down around 10 over a full season) I prefer scoring chances or various shot-based metrics. Because there are a lot more of them, we have a larger sample to work with. Additionally, they show us not just offense, but also defense, which is highly important but not really tracked statistically in the NHL (except by clumsy measures like plus/minus).
I'm not asking you in this article to change the way you view the game, or even to agree that I'm looking at the right things. I do hope that explaining how the numbers are connected to watching the game, and showing how they're grounded in reality, helps people just to understand a little better why folks like me use them. Sometimes, I'll throw a bunch of numbers out there (Player X has a -1.0 Rel. Corsi, 55.0% ZS and second-level QC while scoring 1.34 EVPTS/60) and it's easy to just blank on the jargon and dismiss them out of hand, when what I should be doing is explaining what those numbers mean and why I think they matter.
It'll still be up to each reader to decide if they agree with me, of course. But at least then we'll be speaking the same language.
@ Rob Gilgan:
Believe me, I get that. Lots of people just prefer to sit back and enjoy their hockey without thinking too much about it. Sometimes I liked to do it.
But I watched the game like that for more than a decade without really appreciating basic coaching strategies like line-matching, and I don't think I'm alone. When you hear a commentator on a nationally televised game say the coach is 'just rolling four lines' (I used to hear it a lot; it's rarer now but still happens) when in fact he's very carefully making sure certain skaters play certain opponents, you're hearing someone who's watching the game but is missing out on a huge part of the strategy involved.
I don't always sit and record these things, but I always watch for them - because if I don't I feel like I'm missing out on a big chunk of what's actually happening.
Again, that's my personal feeling and I don't expect everyone to agree with me; it's just where I'm coming from.
@ Clyde Frog:
You're setting the confidence level awfully high at 80%.
Take goalies, for example - even-strength save percentage is the most repeatable year-to-year statistic (more so than wins, more so than GAA, more so than shutouts, etc.) and the year to year repeatability is below 50% with a single year of data (add up years and it jumps, but for one year it's fairly low).
Skater and team statistics are much better - measures like Fenwick correlate strongly with goals for/against, goals/for against correlates strongly with wins - but at this point there's no silver bullet statistic.
Then again, by eye measures don't come close to hitting 80% either. I've spent some time looking at team predictions from the start of the year in various hockey magazines, and the best of the bunch routinely are in the 60% range (to clarify - that's each year, the leader will hit 60%. I haven't seen a magazine yet that can hit 60% every year) when it comes to predicting team finish, simply because there are so many variables to try and lock down.
Stats are just something Oilers bloggers made up to trick everyone into thinking Tom Gilbert isn't just a big softy whose manhood is obviously questionable. LOOK AT HIM NOT HIT ANYONE. HE IS A GIRL. JONES FOREVER!
Ok I think I see your point Mr. Willis. I have just a few minor questions about the process however.
Do you generally use a staff or a wand when conjuring this mystical data, or do you prefer an open hand?
What effect does the colour of one's wizard robes have on the collection and analysis of these statistics? I myself tend toward a simple nondescript hooded black robe, but I have heard examples of those who even went so far as powder blue with large emblazoned gold stars.
Do you generally use Tolkien elvish runes to divulge the Hockey Gods secrets, or do you prefer the more mainstream ancient Sumerian scripts (before translating into modern math)?
When creating a protection circle before summoning any minor underworld quasi-deities (so as to acquire more advanced data), have you noticed any effect on said data as pertains to the choice of regular chalk and sea-salts versus sugared virgin blood and bat guano?
Any help in these matters is greatly appreciated.
As the saying goes, 'There are three types of lies; lies, damned lies and statistic's'
I'm so late to the party! but I'm still gonna go for it, don't judge me!
http://www.youtube.com/watch?v=wqNo5zQMgLc&feature=related
I completely agree with this. It takes so much patience to repetitively defend your position for the sole benefit of others, especially admists all of the sarcastic comments that can be found here.
Willis thanks for taking the time to explain and simplify the concepts behind these stats. Also, I didn't realize there were shorthand methods for some, like QualComp.
I would agree about not being able to watch the game any other way once you start paying attention to line matching or other coaching strategies. It's like the Matrix, once you see it, there's no turning back. For those that haven't taken the blue pill, ignorance is bliss.
Fenwick is our Matrix, Willis our Morpheus and Nuge our Neo. Something like that.
http://www.youtube.com/watch?v=bCUhFZnxoBU
@Jonathan Willis
Exactly right, Clyde.
I won't bother looking up the youtube for that quote since apparently we have someone for that now.
umm... Willis... I'm going to just go ahead and ask... Is there something objectionable about my question from the other day?
It seemed to me to be a perfectly reasonable novice question concerning the relation between two sets of statistical data that on their face appear at odds.
It was an attempt to suss out
1) the operational assumptions subtending the statistical data and their relations; and
2) the missing variable (from a novice perspective) that would account for what I acknowledged must be a superficial anomaly.
You gave me a coherent answer, that I repeated back to you, giving you the opportunity to correct my assumptions further. My recapitulation:
Now, however my comment is not only a driving factor in this article but you appear to offer either a bastardized quote or paraphrase of my initial quote:
so, what's the deal?? What does it all mean? I never said anything like that, and I certainly hope you didn't get that impression. If it's just a coincidence that this entire thread is circling around my question so be it. But, if you are trying to imply something... I'd like to hear the substance of the implication and the argument behind it.
@Romulus' Apotheosis
Terribly sorry if I gave you that impression! Your question was entirely legitimate - and your recap of my answer entirely as I intended it.
Plus/minus is just a good example of a statistic that gets misinterpreted, and often, by people who are new to looking at hockey through a statistical lens. I certainly meant nothing personal by using that same example in the new article.
Here's a smart stat.
100% of the 20 players on the Nov 17 game day Ottawa Senators roster will hear the chant of Go Oilers Go!! from the fans at Rexall Place.
unfortunately, I don't know how to Corsi or QualComp this :(
Go Oilers Go!
Actual video footage of the application of advanced stats in progress! Rare! http://www.youtube.com/watch?v=uDP9Zjevm5U
As anyone who studied stats in HS or University, stats are anything you want them to be. You can use them to prove any theory, whether merited or not. But that 'feature' of statistics is only exploited by those with ill intentions or empty heads. The stuff on this site is not in that category.