The Different Types Of Consensus Lists

Breaking down the details of a general, coordinated, and weighted consensus for scouting lists.

As I have noted previously, many of the public lists are group works or consensus lists of other public sources. A lot of these grade out very well when compared to lists from individuals, regardless of whether it’s a consensus list of various sources, or multiple scouts within one coordinated group. There are some key differences, so let me get into that, and why they all work.

The main issue with scouting comes down to the human elements of it. On one side, we have a human with non-linear development that we are trying to project while having no input into their development path or career opportunities. On the scout’s side of things, we have our own inherent biases that affect rankings and perception. Mix that together and there’s a whole ton of human uncertainty baked into scouting on the whole. Now, without being an NHL team it’s difficult to do anything about the scouted player’s human elements. However, what can be done on the scouting side is to work with other scouts in order to try to remove the inherent biases of the scouting team. That’s the ideal version anyways. 

Coordinated Consensus

When it comes to coordinated groups, be it a previously mentioned team like McKeen’s, FCHockey, DobberProspects, or any others, the idea is that with multiple scouts working together the biases are covered over and information is analyzed more accurately and evenly. This is covered off especially well by these aforementioned groups, as they designate certain scouts to cover certain areas, with crossover and overall scouts mixed in to help bring the individual areas together more cohesively.

The NHL draft order is also a version of a coordinated consensus, with each team putting their list together from a team of scouts. Interesting for the NHL though, is they seem to get in their own way sometimes, especially earlier in the draft, which I’ll get into a bit more at a later time. The difference with the NHL is that there are pressures to not just pick the best player, but one that might either fill a need, be the hometown kid, or have some other layer of politics to it. On top of that, these team “consensus” lists are then selected from in a predetermined order, meaning that a player could be first on 30 out of 32 lists, but if those two other teams pick first and second, that player won’t be selected until 3^rd overall. All that to say, each NHL team is working off of a parallel coordinated consensus list. However, the NHL draft itself ends up being a mixed amalgamation of all of those lists.

General Consensus

One other main type of consensus is a general consensus, which instead of balancing opinions within a group to come to an agreed upon order, takes whatever input lists there are and sorts them into the average “consensus” order. This is what Bob McKenzie does with his NHL scouts and executives that he polls, it’s what the Habs Eye on the Prize site has done with success as I mentioned before, and it’s what plenty of others do as an exercise to have a general idea of where players stand.

One of my favourite displays of consensus listing for NHL draft rankings is Scouching’s. He puts the list together as a visual with a bar chart, so you can easily see where the drop offs of tiers are, and what kind of range the player has with the error bars (I’ve linked his tweet with it if you click on the chart below).

Not only does it display the consensus list, but it gives plenty of other information to the reader about what tiers have formed, and what players might be wildcards come draft day and along their developmental arcs. As far as general consensus lists go, his might be the best.

Weighted Consensus

There are some who dive a little further into the nuance of stats and draft pick values, understanding that someone ranked at five on one list and 15 on another list has a higher probability of plus-value than someone ranked at 10 on the same two lists. That’s based on the idea that there’s a logarithmic correlation between the value of a draft pick, and how early the pick is.

Dom Luszczyszyn of the Athletic has an excellent breakdown of those numbers and the research behind it. Here’s the overall graph displaying just how steep the drop off is.

Getting back to the example I had above, the player that has a chance to provide the value of a fifth overall pick is the much more valuable selection than the player that is more likely to provide the value of a 10^th overall pick.

Blue Bullet Brad is the one who I have seen incorporate that best, providing proper weighting to each draft ranking list, and then combining the sum for each player into a weighted ranked consensus. His list has performed very well in the top-10 of drafts as I went over here, and is above the average across the first round as well.

The one issue with these consensus rankings, is that everyone’s list is assumed to be created equally, and given the same weight. As I’m starting to add some historical context to how well scouts and list makers have performed in the past, I have the ability here to add that context of which scouts should be given more weight in the consensus.

My end plan is for a weighted consensus, both by order and by input. That means valuing the input of certain scouts with a better historical accuracy a little more than others, and then for every list, grading the player values on a logarithmic scale. Hopefully, this gives us the best consensus list able to be produced at the time of the draft.

If you have any links to final rankings for me to include, please let me know.