Amazon vs. Goodreads Reviews

Time to get technical! Break out the warm milk and sleeping pills! Earlier in the week, I took a look at Amazon and Goodreads ratings of the major Hugo candidates. An astute viewer will notice that those rankings don’t exactly line up (nor do the number of ratings, but I’ll address that in a later post). Which is more representative? Which is more accurate?

At Chaos Horizon, I strive to do things: 1. To be neutral, and 2. Not to lie. By “not lie,” I mean I don’t try to exaggerate the importance of any one statistical measure, or to inflate the reliability of what is very unreliable data. So here’s the truth: neither the Goodreads or Amazon ratings are accurate. Both are samples of biased reading populations. Amazon over-samples the Amazon.com user-base, pushing us towards people who like to read e-books or who order theirs books online (would those disproportionally be SFF fans?). Goodreads is demographically biased towards a younger audience (again, would those disproportionally be SFF fans? Worldcon voters?). Stay tuned for more of these demographics issues in my next post.

As such, neither gives a complete or reliable picture of the public reaction to a book. If you follow Chaos Horizon, you’ll know that my methodology is often to gather multiple viewpoints/data sets and then to try to balance them off of each other. I’ve never believed the Hugo or Nebula solely reflects quality (which reader ratings don’t even begin to quantify). At the minimum, the awards reflect quality, reader reception, critical reception, reputation, marketing, popularity, campaigns, and past voting trends/biases. The Chaos Horizon hypothesis has always been that when a single book excels in four or five of those areas, then it can be thought of as a major candidate.

Still, can we learn anything? Let’s take a look at the data and the differences in review scores for 25 Hugo contenders:

Table 1: A Comparison of Reader Rankings on Goodreads and Amazon.com
Review Comparison Chart

The column on the far right is the most interesting one: it represents the difference between the Amazon score and the Goodreads score. As you can see, these are all over the place. There are some general trends we can note:

1. Almost everyone does better on Amazon than Goodreads. The average Amazon boost is .19 stars, and only three books out of 25 scored worse on Amazon than Goodreads. Amazon has a higher bar of entry to rate (you have to type a review, even if it’s one word; Goodreads lets you just enter a score), so I think more people come to Amazon if they love/hate a novel.
2. There doesn’t seem to be much of a pattern regarding who gets a higher ranking bump. Moving down the top of the list, you see a debut SF novel, a hard SF novel, an urban fantasy novel, a YA novel, etc. It’s a mix of genres, of men and women, of total number of ratings, and of left-leaning and right-leaning authors. I’d have trouble coming up with a cause for the bumps (or lack thereof). So, if I had to predict the size of a bump on Amazon, I don’t think I could come up with a formula to do it. I’ll note that since Amazon bought Goodreads, I think the audiences are converging; maybe in a few years there won’t be a bump.
3. If you want to use either ranking, you’d have to think long and hard about what audience each is reflecting, and what you’d want to learn from that audience’s reaction. It would take a major effort to correlate/correct the Amazon.com audience or Goodreads audience out to the general reading audience, and I’m not sure the effort would be worth it. Each would require substantial demographic corrections, and I’m not sure what you would gain from that correction. You’d have to make some many assumptions that you’d wind up with a statistics that is just as unreliable as Goodreads or Amazon.

I think “Reader Ratings” are one of the most tantalizing pieces of data we have—but also one of the least predictive. I’m not sure Amazon or Goodreads tells you anything except how the users of Amazon or Goodreads rated a book. So what does this mean for Chaos Horizon, a website dedicated to building predictive models for the Hugo and Nebula Awards?

That reader ratings are not likely to be useful in predicting awards. Long and short of it, Amazon and Goodreads sample different reading populations, and, as such, neither are fully representative of:
1. The total reading public
2. SFF fans
3. Worldcon Voters
4. SFWA voters
Neither is 100% useful (or even 75% . . . or 50%) reliable in predicting the Hugo and Nebula awards. So is it worth collecting the data? I’m still hopeful that once we have this data and the Hugo + Nebula slates (and eventually winners), I can start combing through it more carefully to see if it comes up with any correlations. For now, though, we have to reach an unsatisfying statistical conclusion: we cannot interpret Amazon or Goodreads ratings as predictive of the Hugos or Nebulas.

Advertisements

Tags: ,

9 responses to “Amazon vs. Goodreads Reviews”

  1. Jain says :

    Besides the points you mentioned in your above analysis, it’s also worth noting that Amazon and Goodreads have different suggested rating scales. On Amazon, 1 star = hated it, 2 stars = didn’t like it, 3 stars = it was ok, 4 stars = liked it, and 5 stars = loved it. On Goodreads, however, 1 star = didn’t like it, 2 stars = it was ok, 3 stars = liked it, 4 stars = really liked it, and 5 stars = it was amazing. So anyone on Amazon who enjoyed a book (and who’s following Amazon’s suggested rating scale) has to rate it 4 or 5 stars, while anyone on Goodreads who enjoyed a book (and who’s following GR’s suggested rating scale) can rate it 3, 4, or 5 stars.

    Of course, there’s no way of knowing how many users of either service follow the suggested rating scales and how many of them follow a more idiosyncratic rating scale. But I think it’s extremely likely that at least part of the reason books tend to score higher on Amazon is because some users *are* using the different suggested rating scales as their guides.

    • chaoshorizon says :

      Great point, and thanks. I’ll admit I’ve been using both Amazon for at least a decade and Goodreads for 2-3 years, and I don’t know either had a recommended scale! But that could definitely account for some of the difference between the two.

      • D.A. says :

        Hovering over the stars on goodreads shows the suggested scale. Any data imported from other booksites to goodreads that uses half star ratings, incidentally, will round up the half stars to further skew the rating (files from Visual Bookshelf, Calibre, Leafmarks, etc.). goodreads also does not require reviewing a book in order to rate it and has more active reviewers to further skew things.

        Just curious, where are you getting your age demographics? I’d be curious about links. General Internet or reader statistics or actual Amazon/goodreads demographics? Does Amazon provide customer demographics and if so breakout by who does/doesn’t review books? Does goodreads post somewhere even though not requiring demographics being required in having an account—even for chose not-to-be private profiles and for profiles that have to private under goodreads TOS because belong to 13-17 year olds under direct parental supervision?

      • chaoshorizon says :

        Keep in mind Chaos Horizon has a very narrow focus: seeing if various data sets are useful in predicting the Hugo/Nebula awards. So I’m not necessarily investigating whether Amazon is more reliable than Goodreads, or vice versa. The point here was to see if the two websites were consistent with each other: i.e. are books always rated higher on Amazon? Are books always rated higher on Goodreads? Is there a relationship that could be worked out? Does that score have any correlation to the Hugos? Unfortunately, the data seems to be pretty useless, at least for my purposes.

        This post came after two other posts on Amazons versus Goodreads versus Bookscan, located here and here. I was pulling demographic data for Goodreads off Quantcast (specific links are in those earlier posts). Amazon used to be ranked by Quantcast, but Amazon has since pulled their data: information like that is too valuable for big companies to share! I was forced to scramble to an older article to estimate demographic data from a few years back. Again, these are only rough estimates, and not necessarily an investigation of whether Amazon or Goodreads should be trusted more. Rather, the narrow research question was: are Amazon and Goodreads similar enough to be valuable in predicting the Hugo and Nebula awards?

      • D.A. says :

        Are the star ratings in your chart weighted to reflect that on retail site Amazon an okay read is ★★★☆☆ (a disliked read ★★☆☆☆) while on crowd-sourced goodreads an okay read is ★★☆☆☆ (a disliked read is ★☆☆☆☆)?

        Or are they just the book’s averages star rating on both sites with no converting/weighting?

      • chaoshorizon says :

        It’s just the raw data, no correction applied. I always prefer raw data: if people want to apply corrections, they can go ahead and do that themselves. Making a correction like that would force you to make a lot of assumptions (everyone uses the suggested rating scale, for instance), that would seem to me just as problematic as not applying a correction. Furthermore, if there was a consistent difference between how Goodreads and Amazon users used the ranking system, that would likely show up in a consistent relationship between the two score columns. If you look at the (limited, and only applied to one genre) data given, that relationship seems all over the place. Some books with high scores have big ratings differences. Some books with high scores have minimal rating differences. Some books with lots of ratings have big rating differences. Some books with lots of ratings have small differences. That “all over the place” look of the data shows (to me, at least), that there isn’t a consistent relationship between Amazon and Goodreads ratings, at least in the context of Hugo/Nebula contenders.

        If you’re looking at the broader question—how do Goodreads scores correlate with Amazon scores—it would probably be worth it to take 100 books, apply your suggested correction, and see if you come up with a smoother curve than I did. It would be an interesting project, particularly when looking at books outside the narrow range of SFF titles I chose.

  2. Erin @ Paperbackstash says :

    Also, I’ve heard Amazon can remove some negative reviews. With their character count requirement, you have to say a certain amount to leave an Amazon review, so some won’t bother. Not so with Goodreads.

    I’m also curious how you define accuracy. The ratings are a little different but don’t seem dramatically so.

    • chaoshorizon says :

      In the context of Chaos Horizon—a Nebula/Hugo prediction blog—accuracy means how predictive the review scores are for the Hugo/Nebulas, or if you want to be slightly broader, whether or not these scores reflect the opinions of the WorldCon/SFWA voting audiences. That’s a very narrow definition, suitable only the purposes of this blog. Small differences, particularly when they’re not consistent from book to book, means that the data is pretty useless in predicting the Hugos/Nebulas.

  3. 5ngela says :

    I often heard that Amazon and Goodreads use different rating but I think most reader use the same rating for Amazon and Goodreads. So this information really come as surprised for me

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Xeno Swarm

Multiple Estrangements in Philosophy and Science Fiction

AGENT SWARM

Pluralism and Individuation in a World of Becoming

Rare Horror

We provide reviews and recommendations for all things horror. We are particularly fond of 80s, foreign, independent, cult and B horror movies. Please use the menu on the top left of the screen to view our archives or to learn more about us.

Space and Sorcery

Adventures in speculative fiction

The BiblioSanctum

A Book Blog for Speculative Fiction, Graphic Novels... and more!

The Skiffy and Fanty Show

Running away from the thought police on wings of gossamer and lace...

Relentless Reading

"A Veritable Paladin of Blogging!"

MyLifeMyBooksMyEscape

A little about me, a lot about books, and a dash of something else

Far Beyond Reality

Science Fiction and Fantasy Reviews

Andrew Liptak

three more from on high

Reading SFF

Reading science fiction and fantasy novels and short fiction.

Eamo The Geek

The Best In Sci-Fi And Fantasy Book Reviews by Eamon Ambrose

The Other Side of the Rain

Book reviews, speculative fiction, and wild miscellany.

Read & Survive

How- To Read Books

Mountain Was Here

writing like a drunken seismograph

The Grimdark Review

The very best of fantasy.

From couch to moon

Sci-fi and fantasy reviews, among other things

%d bloggers like this: