Estimating the 2016 Hugo Nominations, Part 4
Predicting how the “Sad Puppy” voters are going to nominate in 2016 is the most speculative part of all. The Sad Puppies drastically changed their approach, moving from a recommended slate to a crowd-sourced list. It’s an open question of how that change will impact the Hugo nominations.
What we do know, though, is that last nomination season the Sad Puppies were able to drive between 100-200 votes to the Hugos in most categories, and the their numbers likely grew in the finally voting stage. I estimated 450. All those voters are eligible to nominate again; if you figured the Sad Puppies doubled from the nomination stage in 2015 to now, they’d be able to bring 200-400 votes to the table. Then again, their votes might be diffused over the longer list; some Sad Puppies might abandon the list completely; some Sad Puppies might become Rabid Puppies, and so forth into confusion.
When you do predictive modelling, almost nothing good comes from showing how the sausage is made. Most modelling hides behind the mathematics (statistical mathematics forces you to make all sorts of assumptions as well, they’re just buried in the formulas, such as “I assume the responses are distributed along a normal curve”) or black box the whole thing since people only care about the results. Black boxing is probably the smart move as it prevents criticism. Chaos Horizon doesn’t work that way.
So, I need some sort of decay curve of the 10 Sad Puppy recommendations to run through my model. What I decided to go with is treating the Sad Puppy list as a poll showing the relative popularity of the novels. That worked pretty well in predicting the Nebulas. Here’s that chart, listing how many votes each Sad Puppy received, as well as the relative % compared to the top vote getter.
| Somewhither | John C Wright | 25 | 100% |
| Honor At Stake | Declan Finn | 24 | 96% |
| The Cinder Spires: The Aeronaut’s Windlass | Jim Butcher | 21 | 84% |
| Uprooted | Naomi Novik | 20 | 80% |
| A Long Time Until Now | Michael Z Williamson | 17 | 68% |
| Seveneves | Neal Stephenson | 15 | 60% |
| Son of the Black Sword | Larry Correia | 15 | 60% |
| Strands of Sorrow | John Ringo | 15 | 60% |
| Nethereal | Brian Niemeier | 13 | 52% |
| The Discworld | Terry Pratchett | 11 | 44% |
| Ancillary Mercy | Ann Leckie | 9 | 36% |
What this says is that for every 100 votes the Sad Puppy generates for John C. Wright, they’ll generate 36 votes for Ann Leckie. I know that stat is suspect because not everyone who voted in the Sad Puppy list was a Sad Puppy, and that the numbers are so small it was easy for one person to get boosted up the list by a small group of fans. Still, this gives us something. What I’ll do is plug this into my chart of 40%, 60%, and 80% using the 450 Sad Puppy estimate to come up with:
| Sad Puppies | |||
| Scenario | 40% | 60% | 80% |
| Voters | 180 | 270 | 360 |
| Ancillary Mercy | 65 | 97 | 130 |
| Uprooted | 144 | 216 | 288 |
| The Fifth Season | |||
| Aurora | |||
| Seveneves | 108 | 162 | 216 |
| Golden Son | |||
| Somewhither | 180 | 270 | 360 |
| The Aeronaut’s Windlass | 151 | 227 | 302 |
| Agent of the Imperium | |||
| Honor At Stake | 173 | 259 | 346 |
| A Long Time Until Now | 122 | 184 | 245 |
Does this make any sense? I’m sure many will answer no. But look closely: could the remnants of the Sad Puppies, no matter how they’re impacted by the list, generate 300-150 votes for Jim Butcher this year? I find it hard to believe that they couldn’t produce that number. Remember, Butcher got 387 votes last year in the nomination stage. Some of that was Rabid Puppies (maybe up to 200), but where did the rest come from? And will all the Sad Puppy votes for Butcher vanish in just a year?
How about that Somewhither number—is it too big? This could also model some Sad Puppies being swayed over to the Rabid Puppy side, as would the Seveneves number. The Novik and Leckie numbers could represent the opposite happening: Sad Puppies who joined in 2015 and are now drifting over to more mainstream picks and choices. I think I’d go conservative with this, staying in the 40% band to model the dispersion effect.
So now I have predictions for each of the 3 groups. If I combine those, I get 27 different models. Each model may be flawed in itself (overestimating or underestimating a group), but when we start looking at trends that emerge across multiple models, that’s where this project has been heading. In predictive modelling, normally you make the computers do this and you hide all the messy assumptions behind a cool glossy surface. Then you say “As a result of 1,000 computer simulations, we determined that the Warriors will win 57% of the time.” For the record, the Chaos Horizon model now says the Warriors will win 100% of the time and that Steph Curry will be nominated for Best Related Work.
We could go on and do 100 more models based on different assumptions and see if trends keep emerging. This kind of prediction is messy, unsatisfying, and flawed, and the more you actually understand the nuts and bolts behind it, the more it makes you doubt predictive modelling at all. Of course, the only thing worse would be if predictive modelling was 100% (or even 90% or 80%) accurate. Then we’d know the future with 100% accuracy. Come to think of it, wouldn’t that make for a good SF series . . . Better get Isaac Asimov on the phone. Maybe I should argue that this series is eligible for “The Best SF Story of 2017” Hugo.
Tomorrow we’ll start combining the models and see if anything useful emerges.
31 responses to “Estimating the 2016 Hugo Nominations, Part 4”
Trackbacks / Pingbacks
- - March 31, 2016
- - April 1, 2016
Chaos Horizon 
My prediction of the five nominees: Uprooted, Seveneves, The Aeronaut’s Windlass, Somewither, and Ancillary Mercy
Way to spoil the surprise! But that’s what I think as well.
Then here’s another question. Do you think any of the five will decline their nomination? I haven’t paid that close attention to them, but my guess would be the Stephenson might decline a Puppy-fueled nomination.
Butcher did not decline his last year and Somewhither is published by Castalia House, so I highly doubt Wright would decline his nomination.
That’s a good question to ask. I figure Wright would take it, no problem. Stephenson is an interesting case—given the success of his past novels in the Hugos, he could make an argument that he’d make it without Puppy help. Then again, Stephenson is so successful that he might not want the bother of being caught up in a culture war. Same could go for Butcher—he took it last year before the extent of the controversy was known. It might be tempting to say, “Thanks for the honor, but let’s give someone else a chance.” I wouldn’t be surprised if one or both of them turned it down, but I don’t have enough of a sense of them to make a guess either way.
I don’t think Jim Butcher has a decent chance at winning if he doesn’t decline the nomination. I think only Seveneves and Uprooted have a chance at winning, with Uprooted slightly more likely to win. So the real question is does Stephenson decline the nod. If he does I think Uprooted is almost a shoe in. The only pick for nominee I’m iffy on is Somewhither. Even if Jemisin’s fans rally and edge Somewhither out, I don’t think The Fifth Season has a chance at winning. That said, if Jemisin takes Wright’s place that pushes the odds in favor of Stephenson beating Uprooted for the Hugo by my reckoning, assuming he doesn’t decline the nomination.
I don’t think any of those five would decline. Novik and Leckie would likely be finalists anyway. Stephenson and Butler are popular regardless of Worldcon altogether so I don’t know why they’d really care. And Wright is happy to be the puppies’ pick.
Publically Butcher welcomed the nomination last year and rebuked those who criticized his nomination. Since none of his works prior to Sad Puppies was ever a finalist for any of the literary SF awards – what does he have to lose by turning down a nomination by his fans?
If a Hugos have any sales impact on Novels, it has not been clearly established.
Butcher also attends gaming cons and Baen, Larry Correia, John Ringo, etc….. frequently well represented at any big gaming con. Gencon is the biggest RPG con in the world and Baen is almost always there in force. There is a lot of market overlap between wargamers and RPG people. Wargamers are frequently the game masters of RPG games (I’ve read market studies on this). And wargamers as a group love Ringo, Correia and similar authors. I’ve written for almost every wargaming site on the web and correspond with these individuals often.
In sum, Butcher has no reason to turn down a nomination and his past actions have not indicated any inclination to do so. I met the guy at DragonCon when he released his first book in paperback (a very long time ago) and never had the opportunity to talk with him again. But Butcher’s RPG and gaming links go back to before he was an author.
Thanks for the info. Aside from having read half a dozen Dresden novels, I know basically nothing about the guy.
Source for your Butcher allegations?
I’ve read your musing on modeling nominations for 2 years now. Highly entertaining if you are a SF fan, statistics nut, and somebody who does market modeling.
You are so right that a weighted average of guesses does not improve your estimates unless your underlying assumptions are correct. I gave an exam today that incorporated problems in sales forecasting and when you have to discard previously accurate historical data due to a market shock. I remembered this blog when I was writing the exam.
My suspicion is that the Sad Puppies will have no measurable effect on the nominations this year. My reasoning is as follows:
1) Most Sad Puppies last year genuinely believed they represented the opinion of the majority of fans. The repudiation of the slates at Sasquan last year destroyed this belief. Those folks mostly dropped out, and the rest became Rabid Puppies, who merely want to destroy the awards and don’t care who gets one.
2) The list this year is nothing but a beauty contest. It has no political agenda whatsoever. There’s nothing in it to inspire anyone, one way or the other.
3) Assuming there were any high-minded puppies to begin with, the incessant inflammatory comments from the leaders of the SP4 group, together with their refusal to disassociate themselves from Vox Day, makes it impossible for them to want to stick with the group.
A milquetoast list with fire-breathing rhetoric. No matter which side of the controversy you come down on, SP4 leaves you asking, “what was the point of this?”
Why would anyone simply want to randomly destroy an award, especially SF fans? A fit of pique, a bout of insanity?
The truth is this has devolved into a pro vs. anti-Third Wave Feminist affirmative action movement. Somewhere in there art, fun and entertainment gets shoved off a cliff. But that was happening anyway. The reaction to that was predictable, since I know of no precedent where people ever read E. Rice Burroughs or R. Heinlein based on their weight, height, national origin or accompanying list of imaginary grievances. The idea anyone ever “enjoyed” their work because of the beheading of King Charles I or Germany losing WW I is too stupid to imagine, and yet that is exactly how SFF is being promoted today. The idea mid-twentieth American women buying 4.5 million copies of a single issue of a women’s magazine while the totality of SFF mags sold 1/8 of that the same month being “proof” “male elitism” “marginalized” women out of SFF is just one more log on the flames of stupidity.
This is a bed people just like you made, and no one is surprised when you get angry by having a prank act as a mirror showing your own image in it. You keep mentioning V. Day as if you are against some principled concept of biological racial and sexual supremacy, but everyone knows… that to social justice crusaders, that depends on the race and sex, doesn’t it? Even the biased Southern Poverty Law Center hasn’t gone that far off the rails, and do in fact list black supremacist groups and have a principled definition of “supremacist” and “hate groups” which takes no prisoners.
When you can put together a fairly accurate list of nominees based on nothing more than names and the non-fiction writings of those names, that is an award that is already corrupt. Asserting the Puppies corrupted the Hugos is laughable.
Greg:
Or many of the Sad Puppies think that literary SF is boring? You keep thinking that Sad Puppies = Vox Day. They are quite different. I’ve read your posts at various Sad Puppies sites. You just do not get it.
But kudos for trying and being reasonably polite communicating with those who do not share your views.
Different Topic: I went to your website concerning short fiction. It was very interesting. But you seem to omit most compilations of short SF fiction published in book form and not in magazines. In the course of a normal year, I only read short fiction in compilations – mostly those published by Baen and in their monthly bundles. Your site seems to omit these works which would be an oversight in short SF fiction that reaches a wide audience.
Your website, your choice.
We found ten original anthologies to review in 2015. This is the first I’ve heard about the Baen original compilations. Where do you find out about them?
Greg:
If you go to the Baen website. Go to the ebooks link. On ebooks there are monthly bundles. The monthly bundles are the earliest release point for Baen Publishing – usually.
Direct Link is:
http://www.baen.com/catalog/category/view/s/monthly-baen-bundles/id/1999
Couple of examples of current or upcoming Anthologies of short stories.
1] Ring Of Fire Universe (there are a bunch of these). Grantville Gazette VII is an example. Ring of Fire IV is another. Both are in the same universe.
2] Black Tide Rising – short stories in the John RIngo Zombie world.
3] Years Best Military and SF Adventure – 2015
4] Galactic Games
All of these are collections of short SF fiction that are scheduled for release in 2016 or were released in 2014-2015.
You seem to keep up with the Tor short fiction releases carefully and miss the Baen short fiction releases.
Admittedly, I don’t read much short SF fiction. But your site misses what I do read in an average year.
According to what Larry Correia and other Baen authors put on their blogs, Baen pays very well for short SF that fits. They seem to be attracting an increasing amount of short fiction and their releases have been accelerating from the success of the Ring of Fire releases. Scalzi even has a story in Black Tide Rising.
From the perspective of how an outside reader looks at your website, the Baen releases don’t get much chatter from the traditional literary SF reviewers nor from goodreads. So the metrics that you commonly use may not be readily transferrable to how the selection process you use is implemented.
Again, none of this is a criticism. Your website, your rules.
I looked at the bundles, but they’re almost all novels, not anthologies. The anthologies I found were all part of series, but I don’t review stories in a series unless they’re able to stand alone. (A story that can’t stand alone gets no more than two stars.)
Are you sure that Baen publishes very much original, non-series, short fiction?
greghullender says : April 1, 2016 at 1:19 pm
Are you sure that Baen publishes very much original, non-series, short fiction?
Probably not. Most of their short fiction publications are in a series or in a world created by somebody else. They maybe publish two anthologies a year not in a series. They have an annual publication of short military fiction.
Ah, you mean this one: “The Year’s Best Military SF & Space Opera (The Year’s Best of Military and Adventure Science Fiction Stories Book 1)” http://www.amazon.com/Military-Adventure-Science-Fiction-Stories-ebook/dp/B00WXL8UJM
I’d have been glad to review that one, had I known about it. If there is another one this year, I’ll make a point of it.
Thanks!
Greghullender: I speculate the opposite: puppies from last year saw the rejection as confirmation that their cause was just; the possibility that EPH will pass makes this their last chance to make a difference. More of them will participate more fully this year than last.
Total number of women’s magazine sold per month in America, 1955: 44.7 million
Total number of SFF magazines sold per month in America, 1955: 500,000?
Reaction from SFWA members 60 years later:
“How many of you refuse to… read something if it’s about Yet Another Straight White Man?”
“Not a single white man won an award tonight…”
“…only one award went to a white male.”
So what “cause” do the Puppies have, or are they just pushing back against nonsensical myths about “male elitism” (Le Guin, 1975) and “How To Suppress Woman’s Writing” (Russ, 1983) in SFF?
The real question is why weren’t women supporting SFF during the Golden Age and why are male authors and editors being blamed for that today?
I’ll give you a hint: it’s starts with the letter “F.” There’s your “cause.”
We’re getting too far off topic from the purpose of this website, which is to make predictions about the Hugo and Nebula awards, not settle the culture war.
Fair enough. I predict the next 10 Nebulas and Hugos will rotate Jemisin, Wong, de Bodard, Sam Miller, Older, Hurley, Leckie, Kowal, K. Ashante Wilson, Elliott, Samatar, Ahmed, El-Mohtar and Bear along with Uncanny, Strange Horizons and Lightspeed for reasons unknown.
Counterevidence: the relatively low participation on the SP website. How does it compare with the comments thread on Torgesen’s blog last year?
Counter-counter-anecdote: I personally was pushed by last year’s shenanigans to identify more fully as a Sad Puppy, when I’d previously called myself a Sad Puppies Fellow Traveler.
I don’t think anyone has the right to be all that confident in their predictions, but our host has done about as good a job as anyone can.
It might be noted that the exact SP4 vote count, as stated by The List linked at http://madgeniusclub.com/2016/03/17/the-list/, isn’t 100% accurate. In the comments on that post I pointed out, for example, that Ancillary Mercy only got 7 votes (which means it’s not even in the Top 10). Kate Paulk promised to fix the errors, but so far she seems to only have made cosmetic changes.
I count 9 votes for Ancillary Mercy. But it wouldn’t surprise me if there are other errors. Also, she took none of the ties into account.
Yes, this is the shakiest part of the data. Still, the list published was the list published, and there’s not a huge difference between 8, 9, and 10.
Going over the http://sadpuppies4.org/recommendations-best-novel/ page, I still only see seven separate users who voted on AM (these being: Snowcrash, Robmatic, Nathaniel Harada, Joe Sherry, Hypnotosov, Nota Bene, Gary D). However, Nathaniel Harada mentioned enjoying AM twice (http://sadpuppies4.org/recommendations-best-novel/#comment-611 and http://sadpuppies4.org/recommendations-best-novel/#comment-1442), and so did Joe Sherry (http://sadpuppies4.org/recommendations-best-novel/#comment-971 and http://sadpuppies4.org/recommendations-best-novel/#comment-1276) I can only assume Kate ended up counting votes twice, not realizing that some users might have posted several times.
Sorry if this seems pedantic, but I must admit that while I enjoyed the SP4 concept, the organization and execution – as can be seen from this example – leaves much to be desired.
That contributes to the impression that SP4 was a half-hearted effort. It looks like the organizers just wanted it to be over with. Of course the extremely low participation in the list gives the same impression. I’ll be surprised if there’s an SP5.
Yes, the Sad Puppies are sad. I really liked the setup of the website, but the lackluster participation makes it seem like it was just more trouble than it was worth.