Draft pick valuation and the likelihood of goaltenders “making” the NHL
By Steve Burtch
7 years agoGoaltenders are an uncertain commodity at the best of times. Draft pick valuation has presented a similar challenge to analysis for a variety of reasons, but most obviously due to an inability to agree upon measures of success at the NHL level for skaters. Combining these two issues would seemingly test the projection abilities of any rational approach, but I decided to make a run at it just prior to this year’s draft.
Much draft pick valuation analysis in recent years has centered upon probability of skaters making it to the NHL and playing a certain number of games. Unfortunately this doesn’t seem reasonable when examining goaltenders as the number of goalies in the league is far lower than the number of skaters, with there being only 30 teams each having one starter and one backup per side. Regular starters may make a games played threshold but that number wouldn’t do a good job of discerning the results for the majority of goalies that get a taste of NHL action but wash out for a variety of reasons.
Additionally, it seems illogical to define goaltending success as “making” the NHL, even as a long term starter. As more analysis has been performed on the position, it has become clear that not all starters provide similar value, and some remain in their role due to inertia and contract obligations more than anything they are doing to prove their worth in the crease.
Alternatively, teams with goaltending depth may have an elite starter, but thus only provide limited opportunity to a back-up goaltender that could be starting in many other organizations. As a result, a games played threshold isn’t a reasonable measure of qualification when assessing drafted prospects. In order to identify a value added metric, I propose identifying Goals Saved Above Replacement (GSAR) for NHL goaltenders.
In my work on goaltender development trajectories I compiled a database of all NHL goaltenders dating back to the 1983-84 NHL season – when the NHL began recording save percentages. In my work I compared individual goaltender seasonal save percentages to league average save percentage to assess their value in comparison to the rest of the league.
In order to assess the draft value of goaltenders, we can thus set replacement level goaltending as 1 standard deviation below the weighted mean of the seasonal save percentage for the NHL. This would put the upper 83.8% of NHL goaltenders above replacement in any given year – in an NHL season where 70 goalies play games, this would work out to approximately 59 goaltenders, which is very close to the number of starter and backup positions in the NHL. As an example, in the 2015-16 season, 92 goaltenders played at least one game with 70 goaltenders playing nine or more games. 83.8% of 92 goalies would be 77, so -1 standard deviations seems like a reasonable cut-off for replacement level.
Using the defined replacement level threshold, I then tabulated GSAR at the seasonal and then career level for all NHL goaltender seasons from 1984 to 2016. This gives us our measure of goaltender value for the sake of draft pick valuation using all NHL goaltenders drafted between 1982 and 2006.
In order to evaluate the probable outcome of draft picks, we apply an Ordered Probit Regression model based on stratification of drafted goaltenders into 6 groups, as indicated below:
The threshold formula used was exponential due to the regularity of group size that resulted. The cohort of NHL goaltenders drafted had 357 members, and each threshold resulted in a grouping between 38 and 81 goalies for those that made it to the NHL. This seemed reasonable for the purposes of the model.
The model used Draft Slot as a predictor of the Career GSAR Group, and produced probabilities for a pick in each draft slot resulting in a goaltender of a given group. For example, a goaltender selected with the 21st overall pick has the shown probabilities of resulting in each stratified group:
Three goaltenders were drafted with the 21st overall pick since 1982: Jason Muzzatti, Mika Noronen and Tuukka Rask. They fall into groups 4, 3 and 5 respectively having produced 119.03, 58.82 and 562.30 Goals Saved Above Replacement in their careers.
The Expected Group outcome for each draft slot was then compiled and applied to an Estimation of Career GSAR, resulting in the following valuation curve.
Essentially this indicates that there is very little Expected Value in any goaltender chosen after the 2nd round of the NHL entry draft, and most good starters will be first round selections.
Contrary to what may perhaps be the common conception in analytics circles, this does not suggest that there is no point in drafting goaltenders early in the draft. It would simply indicate that if a goaltender is not warranting of early selection, there is little to be gained by his selection beyond organizational need.
Put another way – generational goaltending talents that are identified early in their development such as Carey Price (5th overall), Tuukka Rask (21st overall) or Cory Schneider (26th overall) are still entirely reasonable selections in the early rounds. Teams should avoid riskier selections that are essentially stabs in the dark in the mid-range of rounds 3 and 4, where skaters of higher value can still be obtained.
From 2003 to 2006 fifteen goaltenders were selected in the first two rounds of the draft. Of those fifteen, six became group 5 goalies, and another four became group 4 goaltenders. That’s a 66% return of legitimate NHL talent, with a solid proportion being top end starters. In the same 2003-06 time frame, seventeen goalies were picked in the 3rd and 4th rounds. Of those seventeen, one has developed into a group 5 goalie (Jonathan Quick), and two more are group 4 goalies (Ben Bishop and Thomas Greiss). Ten of the remaining picks have never faced a shot in the NHL. The return rate drops off a cliff after the end of the second round in a similar fashion to what we see with NHL skaters.
While the analytics community tends to be critical of the scouting and drafting of goaltenders, data from the past 30+ years would suggest that the scouting of NHL goaltenders at the top end of the draft is not significantly less reliable than it is for skaters. The decline in expected performance value is more rapid, but this may have more to do with the far greater restriction on available slots at the NHL level than anything to do with imperfect scouting.
Recent articles from Steve Burtch
