Dealing With Small Sample Sizes in Scouting
Addressing all of the shortcomings with trying to analyze scouts when the data is small, variable, and incomplete.
The limitations of this avenue of research and analysis is that the entire area is governed by small sample sizes. Players are scouted recruited, and picked so early in their developmental curves, and teams only have one career path in which to develop them properly. With 1000s of data points per year, and maybe even the ability to see how a player’s career unfolds across the multiverse with each team and each possible development path with each team, then maybe you can start to properly grade the scouts.
With that in mind, I’m going to try anyways for the sake of this project - more on that here.
To provide an example of just how small of a sample size we’re working with, I want to highlight Scott Wheeler of the Athletic. From 2015 (the first year with rankings) to 2019, he had an incredible track record in providing rankings that surpassed the NHL draft order and all of his peers for whom I have any data, with regards to ranking the top-10 of every draft. Then in 2020, he had another quality year relative to the average tracked scouting list and the NHL draft order, except for his number-10 ranked player Noel Gunler. Scott has differed from the consensus in a lot of places before, with many of those instances ended up as wins, but not in this case.
Gunler was ranked by 22 of the 28 lists I have for the 2020 draft, with 15 of the lists having him between 16th and 26th. 10th isn’t an outrageously different ranking, but when dealing with the “top-10” list, it makes a huge deal.
Gunler’s upside after being drafted 41st overall in the 2020 NHL draft still presents him with some runway, but his game has stalled thus far, and he had returned back to Europe after some mixed results in the AHL (in part the return was caused by Carolina’s lack of an AHL affiliate). That one ranking of Gunler at 10th dropped Scott all the way from the top of the ladder in terms of all-time top-10 rankings, down behind 20% of the other tracked scouts I have - still, being in the top-20% is still a testament to how exceptionally consistent Scott has been in the first round. Gunler getting his development back on track could help Scott jump back up, or a few more solid years would entrench him as one of the best at the top of the draft rankings.
Overall, this should highlight just how consistent scouts have to be, and will have to be in the future if they want to separate themselves from the rest of the public scouting community as it becomes more and more saturated. Luckily, they have the luxury of making mistakes, and sticking their necks out for prospects like Gunler, who may have specific developmental needs to hit a possible higher ceiling - and public scouts have zero control over any of that, whereas NHL team scouts could at least make recommendations to the drafting teams on how best to manage developmental curves and what certain players need to work on.
All of that underscores that it’s not just small sample sizes, but also having to put their lists together without the benefit of being part of the rest of the growth process for a player.
I’m dealing with the same thing for the first-round lists as well as the top-10, where many lists extend past the first round, and some of them have names that just miss who would impact a scout’s numbers (positive or negative). However, the cutoff for the first round is where some lists end, so it makes it impossible to evenly rank each source without a hard cut off after the first round.
Having scouts evaluated for their lists at one point in time, for what feels like an “arbitrary” length of 30-32 names, really leads to a grading that has to be taken with context. I know that the rankings lists that I will be producing are going to be small lists based on small sample sizes, but I do want to also provide context with them, to build up the deserving scouts who put in so much thought and effort.
My next post will highlight a few of those deserving scouts who have performed best in the top-10 slots of their draft lists. If you don’t want to miss it, hit the free subscribe button.