NHL and Analytics: Not as clear cut as we think

Math

Is the NHL opposed to analytics?

Not at all, but teams will not quantify their value equally all the time, just like pro and amateur scouts grade out players differently when they watch them.

The simple truth is analytics in hockey are not as clear cut as some want to believe.

This shouldn’t surprise anyone who follows hockey, whether you are a staunch believer in analytics or not. Check out twitter any day and you will see analytic people disagreeing with the value of a certain stat. The funny part for me is watching some stats guys rip on the NHL for being steeped in “old school thinking,” yet often those same people are the ones who steadfastly believe analytics are the best way to get an accurate evaluation of a player. 

Matt Pfeffer was an analytics consultant for the Montreal Canadiens.

In an interview with Ken Campbell of the Hockey News, he stated Shea Weber was an average player.

“In
my model that evaluates Shea Weber, very, very little of it has to do
with shot differential at this point in his career,” Pfeffer said. “With
his experience, you really need to only look at goal differential to
measure his impact. You only need Corsi if you don’t have a large enough
sample size to evaluate goals. My analysis of Shea Weber had
very little to do with Corsi. It’s easy to hate on Corsi, but (Weber)
is not a good goal differential guy either. He’s not pushing the needle
in terms of how many goals the Nashville Predators score and get scored
on when he’s on the ice. He’s good, he’s serviceable, but he doesn’t
really push the needle on either side.”

Pfeffer then added.

“There’s nothing wrong with being average in the NHL,” Pfeffer said. “An average NHLer is worth a heck of a lot and that’s what Shea Weber is.”

If Pfeffer believes Weber is an average player, then that might be the reason the Predators chose not to extend his contract. Pfeffer, like most people, believes the Habs lost the trade with the Predators for P.K Subban. I agree with him on that, but the suggestion Weber is average is absurd in my eyes.

Pfeffer was on record as saying he shared his views on Subban with management. Management went in their own direction, and some believe this differing of opinion cost Pfeffer his job. That might be the case, but it is just as likely Marc Bergevin and management simply didn’t agree with many of Pfeffer’s evaluations. We will truly never know, but I don’t buy the theory the NHL is backing away from analytics because one stats guy got fired.

TrumpFired

If analytics people want to be part of the NHL then they will need to understand how the business works. Coaches get fired. GMs get fired. Players get traded. Scouts get fired. Trainers get fired, and even analytics people will be let go.

Teams have made errors firing coaches and management, as well as making bad trades, and they’ve also made smart decisions in letting some people go. It is part of the business, and, if anything, Pfeffer’s firing shows me the NHL is actually taking analytics more seriously than they have before.

Maybe the Habs were wrong in their evaluation of Pfeffer’s work. We won’t know because we haven’t been privy to everything he compiled for them, or maybe Pfeffer’s analysis wasn’t what they wanted.

Very few scouts see players the same way, and I’ve spoken to many analytics people over the years and they too have differing opinions on stats and which ones are better or worse.

Pfeffer isn’t the first analytics guy to be let go and he won’t be the last. It happens in the business world. People are let go for various reasons. Maybe his passionate plea regarding Subban was the reason, or maybe there were a combination of factors.

Either way, it shouldn’t deter those who study analytics from thinking the NHL doesn’t value them, because that simply isn’t the case. Hard core stats people don’t agree on every statistic, so why should we expect every team to automatically agree with the presentation of their analytics team?

Ask a pro or amateur scout how often management has disagreed with them. It happens often.

Pfeffer is only 21 years old. He’ll likely have many other opportunities ahead of him, but I see his firing more in the context that the NHL is taking analytics more seriously, and now they will start questioning those who produce the numbers.

It is how pro sports works.

Recently by Jason Gregor:   

  • Monday Musings: Chiarelli overhauling Oilers
  • Patience for Puljujarvi
  • Pitlick: I’m sick of playing in the American League
  • Lucic ready to lead
  • Monday Musings: Why NHL keeps overpaying free agents
  • Is Milan Lucic worth it?
  • What will Larsson bring to the Oilers
    • Petrolero

      Maybe the guy got fired because his modelsnwere incorrect, his work might have simple been bad. Mamy bloggers thunk because they went to a summer course in basic statistics or read a few blogs on advamced stats they are qualified to develop predicting models or even come up. With their own stats.

      Just look at TSN’s Scott Cullen resume and education for comparison and even he gets it wrong at times.

      Hockey is simply too chaotic. I’m all for measuring the game but it really annoys me when bloggers, including some on this site, make evaluations and declare them as universal truths. That is just ignorant and doesn’t help the cause of taking advanced stats seriously.

      I think Gregor is right in that the Canadiens probably fired the guy because they are taking advanced stats seriously and realized this guy wasn’t doing a good job.

    • Oiler Al

      @ fisherprice … here’s another one for you!

      I saw a survey where the Canadian teams don’t apply heavy volume ,low percentage shot opportunities,and suffocating swarm systems, and that is the reason none of them made the playoffs this year.

      Any fool can spin nonsense ,naming only teams that have won the cup. All those teams could have won the SC without analytics.

      • fisherprice

        Yes, you’re probably right. Stan Bowman, who regularly talks about how analytics influence lots of his personnel and scouting decisions, and has 3 Stanley Cup rings as a GM is probably wrong.

    • freshpotofcoffey

      I don’t always agree with you Jason, but this article is bang on. It’s also worth noting that he didn’t get ‘fired’ per se, he just didn’t get offered another contract. There’s a big difference.

      For all we know he was a huge jerk around the office and no one liked to be around him. Maybe they couldn’t wait for his contract to end regardless of his thoughts on Subban and Weber.

      Of course I don’t know the guy, I wasn’t there, and I doubt this was the case. I bet he’s a great dude. But it’s a possibility. Which would suggest that people connecting the dots between “opposed the trade” and “got fired”, who also weren’t there, should maybe revisit some of their assumptions.

    • dsanchez1973

      The only comment in this thread worth reading is Rusty’s but as much as I agree with his buildup, I disagree with his conclusion.

      Hockey is a much harder game to measure than an individual 1-1 sport like baseball, or a highly incident distinct sport like football. In the course of a minute, a team can go from offense, to defense, and back again, during which the players on the ice may change in whole or in part.

      That doesn’t mean, however, that we should throw our hands up and say “it’s impossible” – early versions of advanced stats like Corsi or Fenwick were based on generating numbers from the available information. Numerous projects since then have been focused on generating additional underlying numbers that can be interpreted into advanced predictive stats. I have little doubt that it will continue on in this vein to the point where we do have a much more solid base of underlying numbers to draw conclusions from.

      And as for the whole “Eakins liked Corsi so Corsi sucks” – the truth is Eakins failure was 99% based on one thing and one thing only: Oilers PDO while he was coach, especially relating to Dubnyk forgetting how to play goal. Had Dubnyk continued his normal progress and delivered league average level goaltending or better, I believe you might still see Eakins as coach, no matter how many Edmonton sportswriters thought he was an arrogant clown.

      • Clyde Frog

        It is sad, but it’s comments like these that demonstrate just how limiting the whole “advanced” statistics movement is…

        The difference between a descriptive and predictive statistic is quite large and hoping to utilize hockey data in a way that allows for predictive statistics is largely a pipe dream at this point. The confidence levels for any model will be extremely low (also, please notice that no-one ever tells what their calculated confidence level would be).

        What allows baseball or even football to be mined so effectively to date is not that it is a team or individual sport, but that it has a distinct event beginning and end. You can quantify the variables and categorize them for each and every pitch/play, then build your data sets and population with a clear, agreed and replicable understanding of what occurred.

        Hockey suffers from the fact that there is no set beginning or end to an event. This forces an observer to aggregate data or to isolate actions within an event without context.

        This would mean that shots are largely treated equally even though we all understand them to very different. In baseball, a hitter facing a left hand knuckleballer will behave differently that when they face a right-handed pitcher with a mean slider, we would expect this and apply different models accordingly. In hockey though the data never gets this granular and thus we are forced to treat events as largely the same.

        There has been work done to try and quantify zone starts and other factors, but these have extremely limited worth and a large portion are subject to observational bias. Which degrades the data to a terrible degree lowering confidence levels further and making the same model compute different results depending on the observer who captured the data.

        Which is the single biggest problem with statistics in hockey, no two data sets are created equally. This leads to people applying models that confirm their biases, you see this all over blogs. Where people show different measures as indicators of a hockey player being “good” or “bad” and then largely explaining and interpreting what this means for the future. They will show one analysis then explain away the parts that don’t quite fit with another analysis.

        This sort of bias is the reason we have an argument at all. We can say. look this player is great and this says he is! Or this player is terrible and I have proof! People hunt for specific examples that reinforce their view point and dismiss all others.

        Shooting percentage is a great measure of how little understanding of what statistics is exists within the community. How many times have you seen someone trot out the league average shooting percentage and say something like this player is overachieving and this is unsustainable, they will regress to the mean?

        They never post the standard deviation which would tell you how likely a regression would be, nor do they discuss the distribution which would further tell you how much of an outlier this player is.

        They also have an incredibly hard time explaining a player who sustains a high percentage or heck, a player like Ryan Smyth who had the most famous “muffin shot” in the league. He averaged anywhere from 8 to 17% in a season and made Chris Pronger giggle like a school girl every time he teed up a slapper. Now I understand that he was a tip in artist and therefor his shots didn’t rely on sniping from the dot, but that is the point… You can’t compare Ryan Smyth and Alexander Ovechkin in the same breath, even though both were in a percentage of each other and both were better than the average…

        I could go on and on, but I think I’ll let it rest at that. We have an extremely limited understanding in the group who utilize the models of the basic tenants of statistics. Now, this isn’t an indictment of professionals as I am sure they could probably bury me in confidence levels, regression analytics etc. But it is an indictment of those who take a descriptive statistic, strip it of all the quantifying indicators and then pretend it can predict the future.

        PS. Please excuse the punctuation/grammar, I typed this up on my phone and don’t care to properly edit it!

    • Zarny

      People are so dramatic. Last year was the “summer of analytics” but one firing a year later and the NHL is opposed to analytics? Ummm no.

      Mr. Gregor is correct. Analytic people will get hired and fired just like anyone else. It means nothing regarding the overall inclusion of analytics in hockey.

      Analytics are here to stay because they work. No team will completely ignore analytics just as I doubt any team will ever make decisions exclusively based on analytics. They are a tool not the only tool. That some people have no idea what linear regression is won’t change that.

      Hockey analytics is still in its infancy in many ways. The raw data can certainly be improved and analytic people will be the first to tell you this. Implementation of a system like Sportvu would likely render Corsi and Fenwick obsolete as possession could be measured literally in lieu of using proxies. New or refined methods are being developed for different stats every day. Analytics will evolve just like everything else in the sport.

      The real trick with data and statistics is interpreting what they actually mean. Context. Not all data and statistics tell a story. The most successful teams will figure out what is important and what is not. And that will likely change over time.

    • Analytics are one part of describing a players talent. In my opinion, it should never be the only part. There are always other factors some of which are outside of the arena such as how they take care of themselves, how they treat their fellow players and some can’t necessarily be measured like how do they influence their teammates on the bench etc. Focusing on one or a few analytic to describe a player isn’t all that helpful. You also need the why on those stats.

    • Marty

      reasons why p.k subban got traded

      1) he’s annoying (on the ice and off it)

      2) he’s super skilled

      that’s all it is, I’d bergevin it too and trade p.k. all marc found out last year was that price is vezina, hart and norris in one