The PEN/Faulkner association has announced that We Are All Completely Beside Ourselves has won the 2014 PEN/Faulkner award for Best Novel.
That’s quite an honor for Fowler. The PEN/Faulkner is one of the biggest literary fiction awards of the year, along with the Pulitzer, National Book Award, and National Book Critics Circle Award. This win means that Fowler is going to be a force to reckon with on the literary fiction award circuit this year.
What does this mean for the Nebula, though? No Nebula book nominee has ever won an award like this. Does this make Fowler a frontrunner? Or does her acceptance by the lit fic world mean that the novel isn’t SF? Will Nebula voters feel that since Fowler has already been rewarded with a major award, she doesn’t need the Nebula? Or will they bandwagon to show that the Nebula is choosing “major” texts?
I’ll add the PEN/Faulkner to same year awards, and it will factor in the formula. I suspect that Fowler is in the same boat she was in before the award–highly regarded, but her vote will be split by people not finding her genre-y enough. If there were more obvious SF elements in Beside Ourselves, it would be a frontrunner. Since there’s discussion about its SFness, it falls behind Gaiman’s more obviously fantasy novel.
The last of our two indicators. These focus on how our novels are doing this award season. Are they being nominated? Are they winning? There’s got to be some correlation—if you can win one award, you must be more likely to win another.
Unfortunately, these indicators are fraught with problems. Most of the major SF and F awards come out after the Nebula. In particularly, fantasy awards are often clustered in the back half of the year, including the World Fantasy and the British Fantasy.
The Hugo Nominations come out fairly early (April), and are a good indicator of whether or not they’ll win. Since 2000, 8 out of 13 winners were also nominated for the Hugo award. The indicator is actually a little better—in two years, 2004 and 2007, none of the Nebula Nominees was nominated for a Hugo.
The other indicator I want to work with is the broader idea that being nominated for a major award, such as the Clarke, the Tiptree, the Philip K. Dick, the Campbell, etc., is a good indicator. My list of major awards are the ones identified by SFADB, and are as follows: World Fantasy, British Fantasy, British SF, Campbell, Arthur Clarke, Philip K. Dick, Stoker, and Tiptree. Those cover most of the awards that get a lot of press. 10 of the previous 13 years, our eventual winner has claimed at least a nomination for one of those other awards, with the Campbell being the most reliable. Now, the Campbell is only for SF, and that puts our fantasy writers at a disadvantage. However, we already know fantasy is at a disadvantage. I’m going to have to figure out how to weigh this one. Since this is the first year of prediction, it’s a great trial run to see when award nominations come out. For now, I’m pinning this at 9/13. This tosses out the Jo Walton year, since Among Others (2012) was nominated for the World Fantasy and British Fantasy, and those nominations happened after the Nebula was awarded.
Indicator #11: The novel is also nominated for a Hugo in the same year. (72.3%)
Indicator #12: The novel is nominated for at least one other major SF/F award that same year. (69.2%)
We’re still early in award season, so not many have come out yet–just the Tiptree, the Clarke nominations, and the Philip K. Dick nominations. Here’s how our books are doing. Nominations are noted in the chart below; in case anyone wins, I’ll mark that with *:
Leckie is obviously leading the way. I suspect she’s going to be a favorite on the award circuit this year: a big, ambitious SF novel that deals with gender issues is always going to get a lot of attention. Don’t read too much into the lack of nominations for Gaiman–the major fantasy awards haven’t moved yet.
So we’ll have to keep an eye on these indicators as more information comes out.
So that’s our 12 indicators! Now on to weighting!
This is more of a speculative indicator: surely the kind of rankings a book receives (both in terms of how many rankings and total score) gives us some indication of how it’ll do. I’m thinking specifically of the rankings on Amazon and Goodreads as solid indicators of public reaction to a text. Take a look at the Amazon ranking of Martin’s A Game of Thrones (4.4) versus A Feast for Crows (3.6). The first book is widely beloved; the fourth in his series is considered a disappointment. That’s just an example to show how different such scores can be varying on whether the book is liked or not. Presumably, a book has to be well liked to win the Nebula.
Unfortunately, there’s no historical data for this, as we can’t go back in time and see how the Nebula nominees were ranked/scored on Goodreads or Amazon when they were nominated. After the nominations–and particularly the winner–comes out, this drives new readers to the books, which taints any statistical significance those rankings might have.
Nonetheless, we can start collecting data and see if any correlation show up in future years, thus 2015 and beyond. Right now, the weight of Indicator #10 will be 0, as there is not enough data to be reliable or meaningful yet.
Goodreads and Amazon provide us with two useful reader rankings: number of rankings (showing how many people have read the book), and then an actual score (whether they liked the book). More readers + more positive reaction has to equal more wins, doesn’t it?
Indicator #10: The novel is frequently reviewed and highly scored on Goodreads and Amazon.
Let’s take a look at this year’s nominees, as of right after the nominations:
Where AMZ = Amazon, and GR = Goodreads.
What does this data mean? Who knows? A couple things seem clear: the easiest way to get high scores is to not be rated much. I suspect some kind of correlation can be worked up, but we don’t have enough data to make that clear. Interestingly, our frontrunners (Gaiman and Leckie) are middle of the pack when it comes to scores, but Gaiman obviously has a clear advantage to how often his book has been read. For all we know, the most read book every year might win the Nebula. We’ll have to wait for some more years of data to see.
Now we’re on to the critical reception of these books. Critics play an important role in bringing attention to novels, and give us a good indication as to whether or not these books are well respected. While the internet does provide year end critics’ lists, these are a relatively new phenomena, and many of them don’t have long histories.
I took a look at a number of possible year end critics lists. Locus Magazine’s Recommended Reading was by far the best of the bunch. It has a long history, and the eventual Nebula Winner has appeared on those lists 12/13 times since 2000, with the only exception being Asaro’s win for The Quantum Rose in 2002.
Beyond that list, nothing has the same kind of reliability or history. I looked at the NPR and Publisher’s Weekly lists, but the Nebula winners haven’t been consistently appearing on their SF/F lists. A couple online blogs are more promising: Tor.com and io9.com, but these don’t have long histories. io9 has three years of history (including this year), and Tor.com only two. Still, they’ve managed to include the winners on their list, so that’s something. What I’m going to do is give a low weight to this indicator and see if it improves over time.
Indicator #8: Nominee appears on the Locus Magazine Recommended Reading List (92.3%)
Indicator #9: Nominee appears on the Tor.com or io9.com Year-End Critics’ list (100%, but low weight due to lack of history)
So where does that put us for this year?
As we might expect from previous indicators, Gaiman and Leckie rise to the top, appearing on all the recommended lists for the year. Everyone else was divided. Fowler, Griffith, and Wecker likely missed some lists because of the lack of genre elements in their work, not the quality of their writing. This same confusion over genre, however, will likely extend to the voters and hurt them in the final tally.
Once again, two more indicators clearly “won” by Gaiman and Leckie.
So far, we’ve just looked at awards history. Now, we need to look at how the books have been received. Indicators #6 and #7 use some of the popular “Best Book of the Year” votes as an indicator. If a book isn’t well known, it likely won’t receive the most votes for the Nebula award.
The Locus Awards are one of the longest running and most reliable reader polls out there. It’s a well informed audience, and their track record is strong. Of the last 13 years, since 2000, all but one eventual Nebula winner has made the Locus Award list. These are lists of the top 20 novels of the year in a variety of genres (Science Fiction, Fantasy, Young Adult, First Novel, etc.). It doesn’t seem to matter where you place on the lists–we’ve had novels that have been in 1st place, novels in 15th place, novels on the Young Adult list. You just have to show up somewhere.
Unfortunately for us, the Locus Award results came out later in the year, so they won’t be available for some time.
Although it lacks the long history of the Locus, the Goodreads Choice Awards attracts tons of voters, and gives us perhaps the best sense of the “general public’s” reaction to a book. Although only in place since 2009 (4 years data), every Nebula award winning book has placed on either the Goodreads SF or F year end Reader’s Top 20 list. Who knows if this will continue in the future–we can address that by proper weighting–but it seems like a good indicator for right now.
So, that leaves us with these two Indicators:
Indicator #6: Nominee places in the Locus Awards. (92.3%)
Indicator #7: Nominee places in the Goodreads Choice Awards. (100%)
A pretty telling chart. Most of our books this year were not all that popular. Gaiman and Wecker broke through to the mainstream, but everyone else languished. I think this indicates some real problems for our other strong contenders (Fowler, Griffith). Did enough genre readers pick up their books? Even Leckie comes across poorly. The disparity between Gaiman, with more than 40,000 votes, and Leckie, who pushes a little over 300, is staggering.
Another category that gives Gaiman a clear win, and one that’ll push Wecker up in the formula.
This is a simple one: is the Nebula Award for Best Novel biased towards science fiction or fantasy?
The SFWA (Science Fiction and Fantasy Writers) began as a SF organization, and it still prefers SF titles in these awards. Since 2000, SF novels have won 9/13 times, for a robust 69.2%. If you include the previous 20 years (1980-1999), SF novels have won 26/33 times, for a staggering 78.8%. We can even see this bias last year, when 2312 won over 5 fantasy novels.
Interestingly, fantasy novels don’t suffer in terms of nominations. Since 2000, there are 36 F noms versus 43 SF noms. While you could quibble probably with the categorization of a couple novels, this is roughly equal. Even this year we have 4 fantasy novels squaring off against 4 SF novels.
So, that makes:
Indicator #5: Nominee is a science fiction novel (69.2%).
This year is an interesting mix of fantasy and science fiction nominees:
This is the first time Gaiman doesn’t have a clear advantage. While there are 4 science fiction novels, Fowler’s book pushes the limits of genre. If the SF voting crowd is looking for an alternative, they might coalesce around Leckie.
Indicators #1 and #2 looked at nomination history. Let’s move on to winning. If you’ve won previously, does that increase your chances of winning again?
This wasn’t as straightforward as I would have liked. Winning a Nebula was not necessarily a ticket to winning another. 7 of 13 authors who won the Nebula for Best Novel since 2000 were winning it for the first time. Oddly, the other 6 were winning at least their second Nebula for Best Novel. So, I discarded the idea of previously winning as a meaningful indicator, and instead integrated the idea of “Previously had Won a Nebula Award for Best Novel.”
The more correlated category–and I parsed through Hugo Wins, Hugo + Nebula Wins, and so forth–seemed to be how honored you’d been over your career as a writer, defining this as the sum of Nebula Win + Nominations + Hugo Wins + Nominations. This gave a good measure of the profile of an author, and the most honored nominee won a strong 53.9% of the time, with little statistical significance to your winning chances based on placing 2, 3, 4, or below on this ranking.
Both of these pass the eye test, although they are probably a little too dependent on each other. However, in a Linear Opinion Pool, this dependence can be taken care of with proper weighting. So the Indicators worked out like this:
Indicator #3: Has previously won a Nebula award for best novel (46.1%)
Indicator #4: Was the most honored nominee (Nebula Wins + Nominations + Hugo Wins + Nominations) (53.9%)
Gaiman and Fowler far outstrip the other nominees in terms of career honors. Fowler even beats Gaiman on the Nebula nomination front, although Gaiman pulls ahead when the Hugo is factored in.
Gaiman and Griffith have previously won the Nebula for Best Novel. Griffith’s previous win should get Hild a good look by a lot of readers. Gaiman likely doesn’t need more eyes on Ocean, but he is undeniably well known to this voting audience.
By looking at previously Nebula winners and their award history, it was easy to come up with some strong indicators. Of the past 13 years (back until 2000), 11 out of 13 winners had previously been nominated for a Nebula. This makes sense; to be competitive for this award, voters need to know who you are. Previous nominations mean direct exposure to the voting pool, and thus voters are more likely to know those authors, to have read their books, and to vote for them.
Likewise, being previously nominated for a Hugo is a strong predictor, with 10 out of 13 winners having at least one previous Hugo nomination. It’s not as strong, but then why should the Nebula voters listen to the Hugo voters?
Interestingly, prior nominations was more reliable than the broader category of “prior wins.” This may have something to do with the number of different Nebula and Hugo categories.
That’s a strong correlation, and it actually is even stronger than that. Some of the people that snuck wins without many prior nominations were huge names: Neil Gaiman for American Gods in 2003 and Michael Chabon for Yiddish Policeman’s Union in 2008. Gaiman was incredibly well known for his comic Sandman, and did have a Nebula nominee for some of his film work. Chabon had previously won the Pulitzer Prize, and was a widely acclaimed author from the field of literary fiction. Their winning books got huge marketing pushes, and were some of the most widely buzzed about texts of that year. Don’t worry: later indicators are going to take that fame and buzz into account.
But we’re looking at only prior nomination history in Indicators #1 and #2. In fact, Chabon is the only author in the past 13 years to win without at least one prior Hugo or Nebula nomination.
What are statistics, though, without some outliers?
The lesson: you better be somewhat known if you want to win the Nebula. The safest route is to have already been nominated for a Nebula (winning doesn’t actually mean as much, although that will factor in Indicators #3 and #4). Second best is to be nominated for a Hugo. Failing that, win a Pulitzer.
So, to put this in mathematical terms:
Indicator #1: Has previously been nominated for a Nebula.
Indicator #2: Has previously been nominated for a Hugo.
Those numbers aren’t as good as they look, because every year multiple authors have prior nominations, so they have to split up that 84.6% or 76.9%. However, let’s look at this year’s nominees:
N. Wins stands for prior Nebula wins, N. Nominations for prior Nebula Nominations, H. Wins for prior Hugo wins, and H. Noms for prior Hugo nominations.
The analysis is pretty easy here. Our prior Nebula nominees–Gaiman, Fowler, Griffith, Nagata, and Samatar–come out ahead of the newbies Gannon, Leckie, and Wecker. Fowler and Gaiman look particularly impressive considering their number of wins and nominations.
The Hugo nominees make the difference between our well-known and emerging authors even more apparent. Only Gaiman, Fowler, and Griffith have previously been nominated for the Hugo.
A safe bet, then, based solely on these two indicators, would be that either Gaiman, Fowler, or Griffith would win. So, from Indicators #1 and #1 alone, those three pull out into the lead.
From a methodology standpoint, I convert the overall chances (that 84.6% versus 15.4% number), and then distribute that over the various nominees. Our Nebula prediction table comes out like this:
Chances to Win Based Solely on Indicator #1: Previously Nominated for Nebula
Those odds are considerably different than the “drawing out of a hat” odds of a flat 12.5% each. Hugo odds are similar, but even better for the winners, with Fowler, Gaiman, and Griffith having a robust 25.6% each, and the rest ringing in with a paltry 4.6%.
To predict future Nebula wins on past Nebula patterns, let’s note several things:
1. We don’t have much data: The Nebula does not release voting numbers (the Hugo does), so we don’t have any sense of who finished second versus sixth. This makes it harder to predict.
2. The data is unreliable: Nebula voting patterns have changed a great deal over the past 20-30 years. While the Nebula goes back to 1966, can we really use what happened in the 1970s–back before Fantasy was part of the SFWA, and the whole field of SF was much smaller and more insular–to predict an award in 2014?
3. The Nebula is erratic: There have been some rather unpredictable Nebula awards in the past 15 years, with the award going to lesser known books like The Quantum Rose or The Speed of Dark. Good for them, but bad for a predictive model.
So, what does that mean? It means that we don’t have enough reliable information to run some of the more complex statistical models (i.e. the kind of Bayesian models made popular by people like Nate Silver).
I looked over several possible statistical modeling methodologies, and I decided, at least for the purpose of this blog, that a Linear Opinion Pool makes the most sense. Frequently used in risk management, a Linear Opinion Pool is a way of aggregating expert opinions (predictions) and then weighting them to come up with a predictive average. Crudely, here’s what it looks like:
Final % of a Book Winning the Award = Expert #1’s % Guess * Weight for Expert #1 + Expert #2’s % Guess * Weight for Expert #2 + Expert #3’s % Guess * Weight for Expert #3, and so on.
Here, the Weight is a measure of how reliable we think each expert is, and all those weights have to add up to 1.
Bored? It gets worse. Analogy time! Imagine a game of dice with a standard six sided dice. If we believe the dice aren’t fixed, then the % for any given number 1-6 is the same, a simple 16.67%. That would be the chance of correctly picking the Nebula winner if we just drew the name out of a hat.
Now, we know that the Nebula dice are weighted–certain numbers have a better chance of coming up than others. This is where are experts come in. Let’s say one expert watches the game for a while, and then decides that the dice are heavily weighted in favor of the numbers 1 and 2. He then draws up a nice table for us:
Chance of Rolling a #1: 20%
Chance of Rolling a #2: 20%
Chance of Rolling a #3: 15%
Chance of Rolling a #4: 15%
Chance of Rolling a #5: 15%
Chance of Rolling a #5: 15%
We have three things we could do here. We could roll the dice a bunch of times and then check to see if he was correct. We can only do that, though, if we have access to the dice. Second, we could trust the expert, and use his percentages to make a bet. Lastly, we could reject the expert, and go back to our simple 16.7% equal chances.
What if we have more than one expert? This new expert thinks the game is more heavily fixed towards rolling a #1. His probability chart comes out like this:
Chance of Rolling a #1: 25%
Chance of Rolling a #2: 15%
Chance of Rolling a #3: 15%
Chance of Rolling a #4: 15%
Chance of Rolling a #5: 15%
Chance of Rolling a #6: 15%
Who do we trust? If we trusted them equally, we could average the two probabilities to come up with a 22.5% chance of rolling a 1. A Linear Opinion Pool allows us, though, to trust one expert more than another. So let’s say I trust Expert #1 90% and Expert #2 10% because Expert #2 is drunk, I could then compute an average of .9*20+.1*25 = 20.5% chance of rolling a 1.
This is the model I’m going to use to predict the Nebula, with 14 different “Experts” providing probability tables from data mining previous award results and predictors.
While I can get more into the math if anyone wants, let’s run down the advantages of a Linear Opinion Pool:
1. The math is easy to understand and transparent, unlike more complex statistical measures.
2. The formula can be easily tweaked by changing the weights for the experts.
3. Unreliable data can be tossed out more easily because we can see if parts of the formula are messing everything up.
4. Expert probabilities don’t rely on the same kind of enormous data sets other statistical models do.
5. Since the Nebula award is inherently erratic, this doesn’t hypothesize the process is more predictable than it actually is.
So, the next steps are to start building Probability Tables based on past Nebula results. After that, we have to weight them. Then, we finalize the prediction. Onward!
Let’s get straight to the initial list:
1. Neil Gaiman, The Ocean at the End of the Lane (21.7%)
2. Ann Leckie, Ancillary Justice (13.7%)
3. Karen Joy Fowler, We Are All Completely Beside Ourselves (13.6%)
4. Nicola Griffith, Hild (12.2%)
5. Helene Wecker, The Golem and the Jinni (11.9%)
6. Linda Nagata, The Red: First Light (11.5%)
7. Sofia Samatar, A Stranger in Olondria (10.8%)
8. Charles E. Gannon, Fire with Fire (4.6%)
No real surprises. Gaiman is the clear favorite: a genre superstar writing a blockbuster novel that critics and readers loved. Who would have thought it? Ancillary Justice might be higher than you’d expect. My guess is that Leckie will emerge as the clear SF alternative to Gaiman, and we can’t ignore the Nebula’s bias towards SF writers. While Leckie lacks any Nebula history, her book did very well on year-end votes and critics’ lists. Fowler and Griffith did well given their award history but the reception to their books was mixed, largely because of their lack of strong genre elements. They particularly did poorly on voted (readers’) lists.
There’s still several more parts of the formula to be integrated and weighted, and some big indicators are on the horizon: the Hugo nominations and the Locus Readers’ Poll will be coming out later in the year. More discussion of methods to come, and then I’ll get into the formula.