Modeling Hugo Voting Campaigns

What’s Hugo season without some impassioned discussion? 2015 is shaping up to be just as vehement a year as 2014. As I’m fond of saying, Chaos Horizon is an analytics, not an opinion, website. While that line can be delicate—and I sometimes don’t do a great job staying on the analytics side—neutrality has always been my goal. I want to figure out what’s likely to happen in the Hugo/Nebulas, not what should happen. If you want to find opinions about Hugo campaigning, you’ve got plenty of options.

At Chaos Horizon, I try to use data mining to predict the Hugo and Nebula awards. The core idea here is that the best predictor of the future is the past. I make the assumption that if certain patterns have been established in the awards, those are likely to continue; thus, if we find those patterns, we can make good predictions. There are flaws with this methodology—it can’t take into account year-by-year changes in sentiment, nor shifts in voting pools, and data mining tends to slight new or emerging authors—but it gives us a different way of looking at the Hugo and Nebula awards, one that I hope people find interesting. An analytics website like Chaos Horizon is most useful when used in conjunction with other more opinion driven websites to get a full view of the field.

What I need to work on is figuring out how to model the effectiveness of a campaign like “Sad Puppies 3” on the upcoming Hugo awards. For those out of the loop, a quick history lesson: over the past several years, we’ve seen several organized Hugo “campaigns” (for lack of a better word) that have placed various works—for various reasons—onto the final Hugo slate. Larry Correia’s “Sad Puppy” slate (we’re up to Sad Puppies 3 in 2015) has been the most effective, but the campaign to place Robert Jordan’s entire series The Wheel of Time also worked remarkably well in 2014. We’ve also seen some influence from eligibility posts (such as in Mira Grant’s case) on the final Hugo slate.

At this point, I think the effectiveness of campaigning is clear. If an author (or group of authors and bloggers) decides to push for a certain text (or texts) to make a final Hugo slate, and if they have a large and passionated enough web following, they can probably do so. Whether or not that is good for the awards is another question, and one that I’m not going to get into here. A quick google of “Hugo Award controversy” will find you plenty of more meaningful opinions than mine.

Instead, I want to focus on this question: How much influence are campaigns likely to have this year? Let’s refresh our memory on the Hugo rules, taken right from the Hugo website:

Nominations

Nominations are easy. Each person gets to nominate up to five entries in each category. You don’t have to use them all, but you have the right to five. Repeating a nomination in the same category will not affect the result; for instance, if you nominate the same story five times for Best Short Story, it will count as only a single nomination for that story. When all the nominations are in, the Hugo Administrator totals the votes for each work or person. The five works/people with the highest totals (including any ties for the final position; see below) go through to the final ballot and are considered “Hugo Award nominees.”

So, from January 15th to March 10th, eligible WorldCon members are casting nominating ballots to choose the final slate for the 2015 Hugos. Who is eligible to vote?

Anyone who is or was a voting member of the 2014, 2015, or 2016 Worldcons by the end of the day (Pacific Time/GMT – 8) on January 31, 2015 is eligible to nominate. You may nominate only once, regardless of how many of those three Worldcons you are a member.

You can either be an attending member (i.e. someone who actually goes to the WorldCons) or a “supporting member,” which costs you $40 and allows you to participate in the Hugo process. That “supporting member” category has been the buzzed about issue. While $40 may seem like a lot, that nets you the “Hugo Voting Packet,” which includes most of the Hugo-nominated works (author/publishers decide if they’re included). If you like e-books, $40 is a decent bargain for a variety of novels, novellas, novelettes, and short stories. Supporting membership also lets you nominate for at least 2 years (the year you join and then the following year), even if you only get to vote on the final slate once. All around, that’s a pretty good deal.

EDIT (see comments): I’ve been told that the “Voting Packet” is not guaranteed by the Hugo Awards. This has been the practice for the last several WorldCons, but it depends more on rights-holders and the individual WorldCon committees as to whether that it will happen in any given year. So don’t join for the sole reason of grabbing a packet!

That’s a fair amount of info to wade through, and it shows how the Hugo nomination process is relatively complex. Nominations combine last year’s Hugo voters with a new crop of attending and supporting members. That year-to-year carry-over means you don’t start fresh, and this is why the Hugo often feels very repetitive: if voters voted for an author the previous year, they can vote for that author again. I’d go farther than that: they’re very likely to vote for that same author again. This is one of the reason I have Leckie predicted so high for 2015.

So, preliminaries aside, let’s start looking at some data. What did it take to get onto the ballot in 2014? In this case, I’m mining the data from the 2014HugoStatistics, which gives us all the gritty details on what went down last year.

Table 1: Minimum Votes + Percentages to Make the Hugo Slate, 2014

That chart puts into sharp relief why campaigns work: you can get a Hugo Best Novel nomination for fewer than 100 votes. The other categories are even less competitive: a mere 50 votes to make the Short Story final slate? Given the way the modern internet works, putting together a 50 vote coalition isn’t that difficult.

Now, how did the various texts from the Wheel of Time campaign and the Sad Puppy 2 campaign perform?

Table 2: Nomination Results for 2014 Hugo Campaign Books

A couple notes: Sad Puppy 2 got their top nominees in well above the minimums, particularly in Correia’s case. You can also see that, even within the Sad Puppy 2 campaign, different authors received different numbers of votes. 184 voted for Correia, but only 91 (less than 50%) followed his suggestion and also voted for Hoyt.

What can we learn from this chart?
1. Correia grabbed 11.5% of the vote and Jordan around 10%. Correia also ran a Sad Puppy 1 campaign in 2013 that netted him 101 votes and 9% (he placed 6th, just missing the final ballot). Using that data, I could predict 2015 in two different ways: I could average those campaigns out, and argue that a vigorous Hugo campaign will average around 10% of the total vote. While a campaign brings supporters in, it also brings in an opposition party that wants to resist that vote. 10% seems a pretty reasonable estimate. The other way to model this is to note that Correia’s number of voters increased from 101 in 2013 to 184, an impressive 80% increase. If Correia matches that increase this year, he’d jump from 184 to 330 votes. In an earlier post, I estimated the total nomination ballots for this year to be around 2350 (that’s pure guesswork, sadly). 330/2350 = 14.0%. Either way, the model gets us in the same ball park: Sad Puppies 3 is likely, at the top end, to account for between 10% and 15% of the 2015 Hugo nominating vote. For good or bad, that will be enough to put the top Sad Puppy 3 texts into the Hugo slate.
2. The data shows that the Sad Puppy 2 campaign fell off fairly fast from the most popular authors like Correia to less popular authors like Toregersen (60% of Correia’s total) and Hoyt (50% of Correia’s total) to Vox Day (33% of Correia’s total). Torgersen and Vox Day made the final slate based on the relatively weakness of the Novella and Novelette categories. While I don’t track categories like Novella, Novelette, or Short Story on Chaos Horizon (there’s not enough data, and I don’t know the field well enough), I expect a similar drop-off to occur this year. If you want to assess the impact of the whole Sad Puppy 3 slate, think about which authors are as popular as Correia and which aren’t.

If we put those two pieces of data together, we get my “Hugo Campaign Model”:
1. A Hugo campaign like “Sad Puppies 3” will probably account for 10-15% of the 2015 nominating vote.
2. The “Sad Puppies 3” slate will fall off quickly based on the popularity of the involved authors.

How does that apply to the 2015 Sad Puppy Novel slate? Brad Torgersen (running it this year instead of Correia) put forth 5 novels:

The Dark Between the Stars– Kevin J. Anderson – TOR
Trial by Fire – Charles E. Gannon – BAEN
Skin Game – Jim Butcher – ROC
Monster Hunter Nemesis – Larry Correia – BAEN
Lines of Departure – Marko Kloos – 47 North (Amazon)

Based on my modeling, I expect Monster Hunter Nemesis and Skin Game to make the 2015 Hugo slate. Butcher is even more popular than Correia. As such he should hold on to (or even improve upon) most of Correia’s campaign vote. The other authors are not as popular, and will probably hold on to between 60%-30% of the Sad Puppy 3 vote. They’ll probably wind up in the 8-12 spots, just like Hoyt did last year.

As for the other categories—you’ve got me there. If I had to guess, I’d pick the 2 most popular Sad Puppy 3 choices for each category (I don’t even know how to begin doing that) and predict them as making the final slate. That’s sort of how the math worked last year, with 2 “Sad Puppy” slate nominees making into Novelette and Novella. It’s a more robust slate this year, which might actually hurt the chances of more texts making it (by dividing the vote)

Of course, there could be a major change in the nominating pool this year: more voters mean less predictability. Still, given the lack of “huge” 2014 SFF books (The Martian probably isn’t eligible, and we didn’t get a Martin/Mieville/Willis/Bujold/Vinge/Stephenson book this year), I don’t anticipate it being particularly difficult to make the 2015 slate. It’s also hard to predict the “Sad Puppy” support doubling or tripling in just a year’s time. It’s also exactly difficult to imagine “Sad Puppy” support collapsing by 50% or 75% percent. We’ll find out soon enough, though.

So, that’s the model I’m going to us to handle Hugo campaigns. Satisfied? Unsatisfied?