5/27/2016
By Dave LackfordAs a moderator for warchant.com I see a lot of comments from subscribers claiming Rivals rankings are totally off and the only sites that can be relied upon for accurate recruit rankings are 24/7 and ESPN, and that Scout should be disbanded.  Personally, I respect the evaluators form all four sites and see them as hard working individuals and I prefer to use the composite ranking of the four sites as an indicator of a team’s/player’s future success.  I also bristle at halfcocked proclamations masked as assertions of fact.  If I learned anything from my Stats and Methods courses at Bucknell University it was that the only way to defend a proposed theory is to show your conclusion was based upon empirical research, with all of its curvy features as well as its warts.  Honestly, nobody that truly matters cares about your unwashed opinion.

With this in mind, I sought out to see how Rivals rankings of the FSU 2016 recruiting class stacked up against the other three sites.  While the ’16 FSU class is admittedly a small sample size considering these sites rank almost every D1 recruit in the country, I chose this group because the complaints I read about Rivals’ rankings (and Scouts’) were from FSU fans on a Rivals’ affiliate’s FSU message board in regard to their team’s class/individual player rankings.According to ESPN, FSU finished with the #1 class in the nation.  So of course, so it’s not surprising that FSU fans tend to think they have the best talent evaluators. Rivals and 24/7 had FSU finishing 2nd in the country behind Alabama.  Scout had FSU ranked 3rd behind Alabama and Ohio State, so naturally, this particular group of FSU fans think Scout’s rankings are a “joke” in their purely objective assessment, of course.Often, the perception of a site’s credibility is largely based upon where a particular recruit, who ends up becoming one of the fan bases’ favorites, is ranked.  I remember when Rivals dropped Malik Henry from 5-star status last season and the Premium Recruiting Board lost its collective minds. 

The most peculiar of the reactionary comments that week was how Rivals had jumped the shark long ago and how subscribers only read Rivals for the in-depth pieces and inside recruiting information.  This take seemed ill founded to me for a number of reasons.The most glaring flaw with that take was that Rivals was the last site to drop Henry.  While ESPN had Henry raked the highest on the national scale at #17 overall, Rivals wasn’t far behind rating him #34 overall. Both sites regarded him as the 3rd best QB in the nation behind Shea Patterson (Ole Miss) and Jacob Eason (UGA).  If Rivals’ ranking of Malik Henry was the impetus for so much vitriol toward their evaluators, it’s a poor reason for outrage.  It makes even less sense to say 24/7 is far better at evaluating talent, based on Henry, when Rivals was closer to ESPN, while 24/7 (Henry was #53 on their board) was closer to Rivals rating than ESPN’s.

Something has to give here.  How can Rivals subscribers think 24/7 is better than FSU when FSU rates a guy like Henry higher, when they are mad that Henry was dropped in Rivals’ rankings?  So,  I decided to examine how each site ranked each individual player. If ESPN and 24/7 are better than Rivals and Scout, per the discussion on the recruiting boards, then there should not be much variation between ESPN and 24/7’s rankings and Rivals and Scout should be similar.

I made a chart of every player FSU signed and how he was ranked by each site.

In order to compare the sites against one another, this chart will show how often they “agreed” on a player’s ranking.  In order for sites to be in agreement, they must have a player ranked in the same general area overall, at their position, or share the same star ranking while the other two sites do not.  To find a way to objectively compare the site’s rankings, I have constructed the following rubric:

For a site to agree on a player, they must have that player ranked within 20 spots nationally, 5 spots at his position, or two sites designate a player a certain number of stars, while the two other sites designate a different star status.

Legend:

• A yellow or green highlight on a player’s ranking for a respective site means that all sites “agreed” on a player.
• If one site had a player ranked 100 spots above or below the closest site’s OVERALL ranking, that site’s ranking is considered an outlier, and is denoted on the chart by a red highlight.
• If data collection issues made it impossible to determine if sites agreed on a player that phenomenon will be denoted by a blue highlight.
  • Kickers and Punters were not accounted for in this study.

DATA ANALYSIS

The chart below illustrates which players the four sites agreed upon, and which sites agreed with one or their peers more often than the others.

                                                                             AGREEMENT:
  24/7                              SCOUT                               ESPN                                    RIVALS
24/7 & Scout = 7        Scout & 24/7 = 7           ESPN & Rivals = 6        Rivals & 24/7 = 7
24/7 & Rivals = 7       Scout & ESPN = 6         ESPN & Scout = 6         Rivals & ESPN = 6
24/7 & ESPN = 3        Scout & Rivals = 2        ESPN & 24/7 = 3           Rivals & Scout = 2
​
    OUTLIERS: 
Low               High 
24/7 = 2          24/7 = 0
Scout = 4        Scout = 0
ESPN = 0        ESPN = 0
Rivals = 1        Rivals = 1

Number of time all four sites agreed on a player: 2
Number of times three sites agreed on a player: 3
Rivals and Scout agreed on Shavar Manuel’s ranking along with 24/7 but Rivals did not agree with ESPN.

Not once did Rivals and Scout agree on a player, unless a third site or all sites agreed.

Scout and 24/7 however, were the only two sites in agreement on a player five separate times: Brian Burns, Baeveon Johnson, Shavar Manuel, Juaun Williams, and Emmitt Rice.  24/7 and Scout had the exact same rankings for Burns and Williams.  Keeping with the Scout and 24/7 buddy system, the two sites ranked Shavar Manuel 2 spots apart in their overall rankings, and neither felt very highly of LB Emmitt Rice, as both tagged him as a 3 star, separated by 6 spots on their position rankings.  On the flipside, Rivals and ESPN both tabbed him a 4-star, with Rivals giving him the highest outlier rating at 186th overall.

Highest Rank for a player:             Lowest Ranking for a player: 
ESPN = 12                                                        Scout = 12
24/7 = 6                                                             24/7 = 6
Rivals = 3                                                          Rivals = 5
Scout = 2                                                           ESPN = 1 

Most 5 stars:                Most 4 stars:              Most 3 stars:
24/7 = 3                                ESPN = 20                         Scout = 8
Scout = 2                              Rivals = 19                        24/7 = 7
ESPN = 1                              Scout = 14                         Rivals = 3
Rivals = 1                             24/7 = 13                           ESPN = 3

​Once I crunched the data, I was shocked to see that 24/7 and Scout ranked the FSU players very similarly while Rivals and ESPN were closer to one another.  Therefore, the comments from Warchant subscribers that 24/7 and ESPN were the most accurate seems to be based purely on perception and not reality.  Here is the logic at play, if ESPN is the best, Scout is the worst, and 24/7 is right there with ESPN, then surely 24/7 and ESPN will rate the FSU class similarly while Rivals’ and Scouts’ ratings will agree.

That was not the case here.

While 24/7 gave the 2016 FSU class the most 5-stars with three, Scout was second with two.  Further, Brian Burns was FSU’s second rated 5-star according to both 24/7 and Scout and both sites ranked him exactly the same with 5-stars at #22 overall.  ESPN and Rivals only awarded one 5-star player to the ‘Noles.  One can fairly point out that ESPN gives out less 5-stars than any other site and that according to all the other sites Henry’s ranking at 17 would make him a 5-star, and Baveon Johnson’s overall ranking of 29th would also make him a 5-star, meaning both ESPN and 24/7 gave the 2016 FSU class three 5-stars.  But that would be a stretch, if ESPN wanted to give out 31 or 32 5-stars they would, but they specifically limited that number in order to give 5-star status a different distinction.

Moving on from 5-star status to 4-star status, again, ESPN and Rivals were similar.  ESPN gave FSU’s class 20 4-stars while Rivals’s 19 4-stars came in second. Scout and 24/7, once again, were very much alike, this time in regard to their allocation of stars to this class, with Scout awarding 14, while 24/7 brought up the rear with 13.  On the inverse, Scout and 24/7 gave out the most 3-stars to the Seminole’s class, Scout rated eight players as 3-stars while 24/7 gave out seven. ESPN and Rivals both saw three 3-stars in this year’s haul.

Maybe the perception of ESPN and 24/7 being the best talent evaluators came from the amount of FSU signees they rated higher than other sites.  ESPN gave a player the highest ranking out of the four sites 12 times. That explains why they deemed FSU to have the best class in the country.  24/7 did so six times, which is the closest to ESPN’s lofty opinion of Tribe16.  But, Rivals was right behind 24/7, giving the highest comparative ranking to one of FSU’s signees five times.  Scout of course was last, doing so only twice.  Is one recruit’s ranking really enough to overlook all of the other factors and claim that ESPN and 24/7 are close cousins while Rivals and Scout are the worst?  Nah.

Finally, the most telling stat is the number of times 24/7 and scout agreed on the same players’ ranking, which was seven times.  Guess who else 24/7 agreed with 7 times?  Rivals.  Who did 24/7 agree with the least? ESPN.  The two sites only agreed on the ranking of three prospects.  But the analysis doesn’t end there.  ESPN and Rivals agreed on six prospects.  They also agreed with Scout six times.  Meanwhile, Scout and Rivals only agreed on two prospects in this class, and that was only when all four sites or at least three sites agreed on the same player.

When you do the research it is clear that 24/7 and Scout are more alike than 24/7 and ESPN.  Further, Scout and Rivals are much more conflicted than any of the sites.

In conclusion, if ESPN is the best, and Scout is the worst, then 24/7 is more similar to Scout than ESPN and Rivals is more similar to ESPN than it is to Scout.  On the other hand, 24/7 and Rivals also have many similarities.  To say that one site has better evaluators of talent is inherently subjective.  This analysis attempted to apply the logic expressed by many on Warchant’s Premium Recruiting Board and support the assertion that 24/7 and ESPN are better evaluator of talent than Rivals.  Clearly some other form of logic must be used to support this notion because from the data used in this study, that claim cannot be supported.

Make sure you follow the All Sports Discussion Twitter account at @AllSportsDACC and please like our Facebook Page