2015 Hugo Analysis: Nominating Stats, Part 1
Time to dig into the nomination stats. Since Chaos Horizon is a website dedicated to award predictions, this is data we really need—2015 is going to be our best model for 2016, after all.
Let’s tackle this in a methodical and organized fashion. The 2015 nominating stats are included as part of the 2015 Hugo packet, easily available at the Hugo website or right here: 2015HugoStatistics. The first thing we can do is go back to the Sad Puppy and Rabid Puppy slates and see how many votes each of those texts got. I’ve divided this into three lists: joint Sad/Rabid selections, Sad selections, and Rabid selections.
Joint Sad and Rabid Picks, Number of Nominations in 2015 Hugos:
Novel
263 The Dark Between the Stars
387 Skin Game
372 Monster Hunter Nemesis
270 Lines of Departure
Novella
292 Flow
338 One Bright Star to Guide Them
338 Big Boys Don’t Cry
Novelette
259 The Journeyman
248 The Triple Sun
267 Championship B’tok
266 Ashes to Ashes
Short Story
230 Goodnight Stars
184 On a Spiritual Plain
226 Totaled
Best Related
206 Letters from Gardner
273 Transhuman and Subhuman
254 The Hot Equations
236 Wisdom from my Internet
265 Why Science is Never Settled
Best Graphic Story
201 Reduce Reuse Reanimate
Dramatic Long
314 Lego Movie
769 Guardians of the Galaxy
489 Interstellar
170 The Maze Runner
Dramatic Short
169 Grimm “Once We Were Gods”
170 The Flash “Pilot”
Editor Long
368 Toni Weisskopf
276 Jim Minz
238 Anne Sowards
292 Sheila Gilbert
Editor Short
236 Jennifer Brozek
217 Bryan Thomas Schmidt
279 Mike Resnick
228 Edmund Schubert
Professional Artist
173 Carter Reid
160 Jon Eno
188 Alan Pollack
181 Nick Greenwood
Semiprozine
229 InterGalactic Medicine Show
Fanzine
181 Tangent
208 Elitist Book Reviews
187 Revenge of Hump Day
Fancast
179 Sci Phi Show
158 Dungeon Crawlers Radio
169 Adventures in SF Publishing
Fan Writer
150 Mathew Surridge
156 Jeffro Johnson
175 Amanda Green
201 Cedaer Sanderson
Campbell
229 Jason Cordova
224 Kary English
219 Eric S. Raymond
If we toss out the Best Dramatic, Long Form as an outlier (the stat numbers are way high, indicating that far more than just the Rabid + Sad Puppies voted for Guardians of the Galaxy, as anyone would predict), we wind up with this as the following range:
387-150. That takes us from the most popular pick to least popular choice (Skin Game by Jim Butcher in Novel, down to Matthew Surrdige in Fan Writer). That’s the “effective joint Sad/Rabid Puppy vote,” or how many votes the Puppies delivered to the 2015 Hugo nomination process. That wide range reflects two things: the lack of popularity of categories like Fan Writer, and lack of slate discipline (not every Puppy voter voted for all the works on the slate). To illustrate how some people didn’t follow the slate, look at Best Novel:
387 Skin Game
372 Monster Hunter Nemesis
270 Lines of Departure
263 The Dark Between the Stars
All four are joint Rabid/Sad picks, but Skin Game and Monster Hunter Nemesis grabbed 100 more votes than the Kloos or Anderson. That means that least 25% of these voters were picking and choosing from the slate, not voting it straight down the line.
A couple number to parse: how do we know Skin Game (or any other nominee) didn’t pick up some non-Puppy voters? We don’t know that for sure, but we can look at the 2009 Hugo Nominating stats for references. That’s the last year where they released the complete list of everyone who got a vote. Small Favor, Butcher’s Dresden Files #10, only got 6 votes that year. Now, this year’s pool is bigger, and maybe people liked Skin Game more, but that looks like a relatively trivial number to me. Your mileage may vary.
On the flip side, how do we know that every Puppy voter voted for Skin Game? Again, we don’t know for sure—there could have been 500 Sad Puppies, and only 80% of them voted for Butcher. In this case, I don’t think it matters. We’re looking at “effective” strength: this is how many votes the Puppies actually delivered in the categories, not a potential estimate of their max number. The actual number of votes is what is useful in my predictions.
Conclusion: So, Chaos Horizon is concluding that the effective Sad/Rabid combined block vote was 387-150, with sharp decay by both the popularity of the chosen work and the popularity of the category. I think that number can explain some of the vitriol in the field: of the 387 people who voted for Skin Game, at least 200 of them didn’t vote all the way to the bottom of the slate. More people only voted part of the slate than voted the whole thing—thus opening up the door for all kinds of online arguments as to exactly how “slate”-like this whole thing was. Expect those to continue as we move into Sad Puppies 4.
On to the Sad Puppy picks. When all was said and done, the Sad Puppies only had a few picks that were not mirrored by the Rabid Puppies (8, in fact), so we’ll learn far less here.
Sad Puppy Picks, Number of Nominations in 2015 Hugos:
Novel
199 Trial by Fire
Short Story
132 A Single Samurai
185 Tuesdays With Molakesh the Destroyer
Dramatic Short
41 Adventure Time “The Prince Who Wanted Everything”
Didn’t make top 15 Regular Show “Saving Time”
Semiprozine
111 Abyss & Apex
100 Andromeda Spaceways In-Flight Magazine
Fan Writer
132 Dave Freer
If we toss out the “Dramatic Short” category as an obvious outlier (the Sad Puppy voters didn’t seem to have liked picking cartoon shows in that category, as “Regular Show” didn’t even make the top 15), we wind up with this as a range:
199-100. I think the Trial by Fire number (at 199) is a little inflated; Gannon did grab a Nebula nom for this series in both 2014 and 2015, and I expect he picked up a fair amount of votes outside the Puppy blocks. That 185 number for “Molakesh” might be the more solid estimate of the max Sad Puppy core; that story is from Fireside Fiction, a rather obscure venue. Neither Andromeda Spaceways nor Abyss and Apex placed in the Top 15 in the 2014 Hugos, and the cut off there was a mere 10 votes, so I think we can attribute the lion’s share of those votes to the Sad Puppies.
Conclusion: We only have 8 data points here, but we’ve got a 199-100 range, with the top end only happening in popular categories (Novel, Short Story). That’s a 50% difference from the highest voted to the lowest voted, perhaps suggesting that only 50% of the Sad Puppy voters voted straight down the slate. You could get that number even lower, though, if you counted the television shows that not even the Sad Puppies voted for.
Rabid Puppy Picks, Number of Nominations in 2015 Hugos:
Novel
196 The Chaplain’s War
Novella
172 The Plural of Helen of Troy
145 Pale Realms of Shade
Novelette
165 Yes, Virginia, There is a Santa Claus
Short Story
162 Turncoat
151 The Parliament of Beasts and Birds
Dramatic Long
100 Coherence
Dramatic Short
141 Game of Thrones “The Mountain and the Viper”
86 Supernatural “Dog Dean Afternoon”
Editor Long
166 Vox Day
Editor Short
162 Vox Day
Professional Artist
118 Kirk DouPonce
Fanzine
119 Black Gate
Fan Writer
66 Daniel Enness
Campbell
143 Rolf Nelson
A couple interesting outliers here. The Dramatic Television category seems strange; you’d have to imagine that more people voted for Game of Thrones than just the Rabid Puppies, and Supernatural only picked up a scant 86 votes. Even the Rabid Puppies didn’t follow VD’s instructions in Fan Writer, only voting 66 times for Daniel Enness. I think the most sensible explanation is that Rabid Puppy voters didn’t follow the recommended picks in these categories. If you get rid of those 3 outliers, you end up with a very tight grouping of:
196-100. The Torgersen is probably inflated from Sad Puppies; even though he didn’t include himself on his own list, I can imagine some Sad Puppies coming over to vote for him. He’d also had a prior Hugo nomination outside the Puppy process. The tightly grouped Vox Day number (166 and 162) might be an equally sensible top number for the Rabid Puppies group. We’re only take 20-30 votes difference, though, and we’d be splitting hairs. I’m a stat site, though, so if you want to split hairs, go ahead!
Conclusion: 196-100 seems safe, and not even the Rabid Puppies had perfect slate discipline. This surprised me, although I could probably be persuaded there was a core group of 166 (Vox Day’s editor nom) to 119 (the Fanzine/Professional Artist number) of Rabid Puppies that did stick pretty closely.
So that leaves us:
Nomination Estimates, Sad, Rabid, and Joint Puppy Picks (percentage calculated using 1595 total nominating ballots):
Joint: 387-150; 24.2% – 9.5%
Sad Puppy: 199-100; 12.5% – 6.3%
Rabid Puppy: 196-100; 12.2% – 7.4%
Let’s double check-the math. If we add the Rabid and Sad picks together, we wind up with 395-200. The joint picks is 387-150. Obviously, that top number looks great; those 8 extra votes would seem to fall within the margin of other votes Skin Game is likely to have picked up. 200 and 150 are quite a bit farther apart, but this might reflect the limited data set we have for Sad Puppy picks alone (8 data points) and Rabid Puppy picks alone (15) compared to joint Sad/Rabid picks (52). Some of the joint picks may have been unappealing to both the Sad and Rabid voters, as well as being in categories with low voter turnout (Fan Writer, Fan Cast, etc.). Take a look at this chart, showing how quickly the various voting groups decayed (excluding Best Drama, for the reasons stated above):
The chart just lines up the most popular pick to the least popular pick to take a look at the decay curve. Rabid and Joint alike fell off very quickly and then evened out. I think that reflects how much more popular the Best Novel category is than the rest of the Hugos. In 2015, it pulled in almost twice as many votes as the other fiction categories. Sad Puppies fell quickly the whole way down, but I don’t know if that reflects a greater variance amongst Sad Puppy voters or just a lack of data.
What does this all mean? That’s the big question. What it means for Chaos Horizon is that I can use these ranges and totals as I put together my 2016 prediction. The max number of nominating ballots was in Best Novel, where 1595 were cast; 5950 voted in the Hugo finals, an increase of almost 3.75. According to me previous analysis, here’s my final Puppy estimates:
Core Rabid Puppies: 550-525 (9.2% – 8.9%, using 5950 total votes for percentage)
Core Sad Puppies: 500-400 (8.4% – 6.7%, using 5950 total votes for percentage)
There are also some Puppy inclined neutrals; I’m not including them, because I don’t know if they’ll follow the Puppies into the nomination stage.
Those percentages are a little down from the nominating ballot, but not aggressively so. That’s what you would expect: the Puppies had the advantage of surprise in the nomination stage, while the push-back against them came in the final balloting. Much of the growth in the final ballot was from people wanting to vote specifically against the slates.
Boil this all down, and we now have a set of numbers to use in future predictions. In my next nominating analysis, I’ll be looking at how big the sweeps were for each category. With that data in place, I can then predict whether or not there will be sweeps (or in which categories) in 2016.
12 responses to “2015 Hugo Analysis: Nominating Stats, Part 1”
Trackbacks / Pingbacks
- - September 7, 2015
- - September 11, 2015
Chaos Horizon 
I think it is more helpful to model the Puppies as three groups: Sad, Rabid, and Loose. The Loose Puppies are willing to vote for either the Sad or Rabid slate, according to their preferences.
This explains the fact that in many categories the nominees on both slates get significantly less votes than the sum of the votes for candidates on only one.
The math works out if we assume that there are only about 60 hard votes each for the Sad and Rabid slates, but a larger group of 180 individuals that will vote for either at the top of the slate, decaying to an average of 140 for short story and down perhaps 50 for fancast.
In short story, this group splits among seven choices rather than five, so an average of 100 per candidate.
This agrees with the short story results, where the candidates on one slate average a bit under 160 (100+60), but those on both average a bit under 220 (60+60+100).
I’m most interested in the Best Novel category here at Chaos Horizon, and I always make sure that my ranges can account for that data. How does your 60/60/180 estimate (300 total) account for the 5 data points about 300, which are:
387 Skin Game
372 Monster Hunter Nemesis
368 Toni Weisskopf
338 One Bright Star to Guide Them
338 Big Boys Don’t Cry
Two from novel, two from novella, and 1 from editor. The Weisskopf is easy to explain; she got 50 votes back in 2013 and 169 in 2014, although 2014 she was a Sad Puppy 2 pick. That leaves the others. What data do you have to suggest that Butcher, Correia, Wright, or Kratman are picking up 90-40 votes outside the Sad/Rabid slates? Any estimate you give should at least account for all the data at hand. Those four authors are not showing up on previous year’s data, indicating their support was minimal. I have a hard time thinking that Wright suddenly picked up 40 neutral votes for a Castalia house published novella, but maybe you have a different explanation.
As for your claim, “This explains the fact that in many categories the nominees on both slates get significantly less votes than the sum of the votes for candidates on only one,” isn’t this also just as logically explained by people not following the slate exactly? That’s what the Sad Puppy 2 data suggests, as analyzed here. We saw Hoyt pick up half the vote of Correia just last year. I tend to roll over data analysis from last year, hence my “the picks decayed by choice popularity and category popularity” has last year’s numbers backing it up, as well as long-established Hugo trends of sharp category/popularity decay.
There may be a couple of different ways to slice the pie. I always prefer a broader range, which could account for a couple different analyses of the data, rather than aggressive pursuing a minimum or maximum.
Have you requested the anonymized data? It’s currently tied up in anonymization level drama, but it seems like it would really sharpen the focus of what you can determine about the slates. Be interesting to see how it matches the numbers you’ve teased from the limited data.
There’s a ton you could learn from those ballots, much of which I think exceeds the scope of why they wanted to release the data in the first place (to see if EPH works). I’m not sure they’re going to broadly release the data; most organizations try their hardest to keep their data protected. With a full nomination set, you could pretty much parse out the whole process: which authors are linked in voting, gender/age/genre biases, votes per ballot, all sorts of interesting stuff that the Hugos probably want to keep secret. They probably didn’t think through how much that data would show when they made the non-binding resolution, and are now having all sorts of second thoughts.
What I think would be learned is not just would EPH work, but is it needed 🙂
The problem with just using it to determine if EPH would work is how you define “work”
Interesting to compare that core Rabid nomibation number with the number of people who voted 1st for Vox D in the final ballot for Editor Ling (despite the Rabid preference supposedly being to vote 1 for Toni W).
There was definitely growth from the nomination stage to the final stage—but you’d expect that with the explosion of voters. The Puppies seemed to have grown less (500/200), or around 2.5 times, versus (5950/1827), or around 3.25 times, for the overall voting. Since the Puppies were the first movers in this kerfuffle, they had their greatest impact on the nomination stage.
In aggregate across the analysis – the Sads seemed to largely be doing what the Sad sponsors urged i.e. “here are our suggestions & support them if you agree.” Your analysis seems to back that in terms of a difference in voting behavior between Sad/Rabid.
I think that’s pushing the data a little. There seems to be a range of behaviors, going from almost following the slate verbatim to just voting for Jim Butcher at the top. We do seem to have a clear set of voters who voted the slate all the way (the 40 who voted for “Adventure Time” in Sad Puppies, the 60 who voted for Daniel Enniss from Rabid Puppies), so that (at the minimum) is 100 pure slate voters. If you’re taking about pure slate voters through the categories people care about the most (the fiction categories), the number might be as high as 200.
I was going to comment on this after seeing the ballots, but we hear they won’t be released. This was the most interesting news for me:
Novelette
259 The Journeyman
248 The Triple Sun
267 Championship B’tok
266 Ashes to Ashes
166 Yes Virginia (RP only)
A version of Wright’s (RP only) story was free on his blog.
The SP3 list linked directly to two free stories, Ashes to Ashes and Championship, and one might presume that many of the people who felt strongly enough to pay forty bucks to nominate those stories were also interested enough to make the single mouse click necessary to go and read them.
Yet The Journeyman and The Triple Sun, which were not free online, required an Analog subscription or issue purchase to read, and they still an got almost identical number of votes. If 300-400 voters chose one or both of Journeyman and Triple Sun, wouldn’t that still be less than 2% of subscribers of Analog, the largest SF magazine?
One major impact of the SP2 campaign was to get two Analog stories on the ballot in 2014. In 2007-2013, there were almost none (except two novelettes which got 6-7% of the nominations in in 2011).
If you wear “lockstep voting” glasses you might see hundreds of people willing to cynically nominate novelettes without even looking at them. But if you look at it wearing “an increasing percentage of the almost thirty thousand Analog subscribers have been motivated to participate in the Hugos” glasses, it would suggest that Analog stories will get more Hugo buzz in the future than in the recent past.