Writing Blog: What good are bad reviews?
May. 3rd, 2016 08:12 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
hrjUsually I like to focus this blog on the creative part of the writing process, but I'm in an unusual pause at the moment so I thought I'd talk about the analytic end. I know the common wisdom in mainstream publishing is that an author should pay no mind to reviews and ratings. At most, we should do comic readings of our one-star reviews to show how little we care. (Only cry in private behind locked doors.) So this essay isn't really for anyone whose book came out from a major publisher. But I have this weird bi-cultural existence, suspended between what I consider my "home" writing community (mainstream SFF) and the community in which I was published (small-press/self-published lesbian fiction), so I get a lot of opportunities to compare and contrast. This essay is for people who don't have mainstream publication and for people who may be bewildered by some of those cultural differences.
There is major anxiety within the LesFic community (quite possibly within all marginal publishing communities) around the crowd-sourced rating-and-review sites like Amazon and Goodreads. A big reason for this anxiety is that it's what they (we) have: they don't get a big publicity blitz, they don't get bookstore placement, they don't get advance reviews in all the highly-respected sites. What they do get is an aggregate of individual reader opinions when those readers are motivated enough to post them.
In my experience, this anxiety is expressed in two major ways. There is a strong community pressure that readers--that is, readers "within the community"--should only ever review and rate books that they absolutely love, and therefore that they will rate highly. This philosophical position is expressed explicitly by many LGBT review sites, and in social media forums for LGBT book communities. This anxiety walks hand in hand with the second: the tendency to react to less-than-perfect reviews as a personal attack. Given a supposed "community standard" of only reviewing/rating books you love, there is an interpretation that to rate a book badly (where "badly" is anything less than a four-star review…or sometimes less than a five-star one) must have been done out of personal malice against the author. That either the reviewer is deliberately giving a false opinion (because, of course, the book must be objectively excellent!), or that, even if they genuinely didn't care for it, the act of publicly expressing that opinion could only come from personal malice.
Viewed from within the community (and it is very much an expression of the assumption that the reader/writer/publisher nexus is a community whose purpose is to support each other against the world) this can look a lot more reasonable than it does from outside the community--where it tends to look fairly toxic.
But beyond the damage to the usefulness of ratings/reviews when only glowing opinions are authorized, there is a damage to authors' perceptions of their own work. Express skepticism of the usefulness of all-five-star ratings and some authors will loudly proclaim that their book is so great that of course it earned all those five-star ratings.
No. I believe that almost every book can earn some genuine and sincere five-star ratings. But no book is universally beloved. Let me repeat that with emphasis: NO BOOK IS UNIVERSALLY BELOVED.
Because I wanted to throw some data at this essay, I took a look at Amazon for the top 100 sellers in Historical Fantasy and the top 100 sellers in Lesbian Romance. You know who has spent a very long time in the top 10 books sold in Historical Fantasy? Diana Gabaldon's Outlander. Do you know how many one-star reviews Outlander has on Amazon? 749. Seven hundred and forty-fucking-nine one-star reviews (4% of the total). No book is universally beloved.
Do you know what unfavorable reviews and ratings mean? They mean that your book is engaging readers who are outside your narrow inner-core target audience. Not just that it's reaching them, but it's engaging them sufficiently to express their opinion in public. And up to a certain point (I'll talk about that point later) the more reviews you get, the lower your average rating is. Because the more people you engage, the more likely you are to engage people who may have liked your book but didn't absolutely love it. Sure, that hurts. It would be nice to be universally beloved. NO BOOK IS UNIVERSALLY BELOVED.
Ah, but I'm a data person, so I ask myself, is it possible to quantify to what extent a less-than-perfect average rating reflects getting your book in the hands of people outside your core target audience? Let's see.
I went through the top 100 Amazon sellers in the Historical Fantasy category and recorded the number of reviews and the average star rating. Then I calculated the average number of reviews at each rating. I mention Diana Gabaldon above because I ended up pulling her out as her own little category for the analysis. And then I plotted those pairs of data.
(click for full-size version)
I had to put Diana Gabaldon on a separate y-axis that differed by nearly an order of magnitude from the rest of the data. But here's the take-away: from around an average rating of around 4.3 on up, a lower rating correlates with a larger engagement (expressed as overall number of reviews). This holds true for the overall average of that top 100 and it holds true in the specific case of Diana Gabaldon. The books that had only five-star reviews? (And keep in mind, these are ones that are currently in the top 100 sellers of the category.) None of them had more than four (4) reviews. Your first, most engaged reviewers are quite naturally going to be people from your core target audience. But an extremely high average is a sign that you haven't expanded far beyond that (yet).
Now let's take a look at the same sort of data for the Amazon category of lesbian romance. (In this case, I looked specifically at Kindle sales because for small press books the dynamics of e-book versus paper are peculiar.) I also cut the data off at a rating of 4.0, not only to compare better with the historical fantasy data (for which that was the lowest average rating) but because the values below that represented only one or two books each and so are less reliable for trending purposes.
(click for full-size version)
And what do we find? Pretty much the same thing. In this case, a rating of 4.4 and up correlates very closely with the average number of reviews at that rating. No book that had an average five-star rating had more than five (5) reviews. Interestingly, if you look at the plot for the maximum number of reviews at each average-rating point, you get the same effect: a very strong correlation between number of reviews and a lower average rating.
Now, of course, at some point this effect breaks down. At some point a lower average rating does start to reflect people's opinion of the specific book, even in the aggregate. And that seems to start somewhere in the lower 4's depending on the data set. And these trends are looking at aggregate behavior. It doesn't mean that there's no difference at all between a 4.4 rating and a 5.0 rating. What it means is that the meaningful difference between a book that has a 5.0 rating with X number of reviews, and a book in the same marketing category with a 4.4 rating and 50X number of reviews is not necessarily one of quality. What it means is that the second book is reaching outside its core audience. And is engaging them.
As someone who has been working very hard to reach outside what my publisher believes to be my core target audience, this is what I keep reminding myself when I get a "meh" rating or review. It means that I've succeeded.
There is major anxiety within the LesFic community (quite possibly within all marginal publishing communities) around the crowd-sourced rating-and-review sites like Amazon and Goodreads. A big reason for this anxiety is that it's what they (we) have: they don't get a big publicity blitz, they don't get bookstore placement, they don't get advance reviews in all the highly-respected sites. What they do get is an aggregate of individual reader opinions when those readers are motivated enough to post them.
In my experience, this anxiety is expressed in two major ways. There is a strong community pressure that readers--that is, readers "within the community"--should only ever review and rate books that they absolutely love, and therefore that they will rate highly. This philosophical position is expressed explicitly by many LGBT review sites, and in social media forums for LGBT book communities. This anxiety walks hand in hand with the second: the tendency to react to less-than-perfect reviews as a personal attack. Given a supposed "community standard" of only reviewing/rating books you love, there is an interpretation that to rate a book badly (where "badly" is anything less than a four-star review…or sometimes less than a five-star one) must have been done out of personal malice against the author. That either the reviewer is deliberately giving a false opinion (because, of course, the book must be objectively excellent!), or that, even if they genuinely didn't care for it, the act of publicly expressing that opinion could only come from personal malice.
Viewed from within the community (and it is very much an expression of the assumption that the reader/writer/publisher nexus is a community whose purpose is to support each other against the world) this can look a lot more reasonable than it does from outside the community--where it tends to look fairly toxic.
But beyond the damage to the usefulness of ratings/reviews when only glowing opinions are authorized, there is a damage to authors' perceptions of their own work. Express skepticism of the usefulness of all-five-star ratings and some authors will loudly proclaim that their book is so great that of course it earned all those five-star ratings.
No. I believe that almost every book can earn some genuine and sincere five-star ratings. But no book is universally beloved. Let me repeat that with emphasis: NO BOOK IS UNIVERSALLY BELOVED.
Because I wanted to throw some data at this essay, I took a look at Amazon for the top 100 sellers in Historical Fantasy and the top 100 sellers in Lesbian Romance. You know who has spent a very long time in the top 10 books sold in Historical Fantasy? Diana Gabaldon's Outlander. Do you know how many one-star reviews Outlander has on Amazon? 749. Seven hundred and forty-fucking-nine one-star reviews (4% of the total). No book is universally beloved.
Do you know what unfavorable reviews and ratings mean? They mean that your book is engaging readers who are outside your narrow inner-core target audience. Not just that it's reaching them, but it's engaging them sufficiently to express their opinion in public. And up to a certain point (I'll talk about that point later) the more reviews you get, the lower your average rating is. Because the more people you engage, the more likely you are to engage people who may have liked your book but didn't absolutely love it. Sure, that hurts. It would be nice to be universally beloved. NO BOOK IS UNIVERSALLY BELOVED.
Ah, but I'm a data person, so I ask myself, is it possible to quantify to what extent a less-than-perfect average rating reflects getting your book in the hands of people outside your core target audience? Let's see.
I went through the top 100 Amazon sellers in the Historical Fantasy category and recorded the number of reviews and the average star rating. Then I calculated the average number of reviews at each rating. I mention Diana Gabaldon above because I ended up pulling her out as her own little category for the analysis. And then I plotted those pairs of data.
(click for full-size version)
I had to put Diana Gabaldon on a separate y-axis that differed by nearly an order of magnitude from the rest of the data. But here's the take-away: from around an average rating of around 4.3 on up, a lower rating correlates with a larger engagement (expressed as overall number of reviews). This holds true for the overall average of that top 100 and it holds true in the specific case of Diana Gabaldon. The books that had only five-star reviews? (And keep in mind, these are ones that are currently in the top 100 sellers of the category.) None of them had more than four (4) reviews. Your first, most engaged reviewers are quite naturally going to be people from your core target audience. But an extremely high average is a sign that you haven't expanded far beyond that (yet).
Now let's take a look at the same sort of data for the Amazon category of lesbian romance. (In this case, I looked specifically at Kindle sales because for small press books the dynamics of e-book versus paper are peculiar.) I also cut the data off at a rating of 4.0, not only to compare better with the historical fantasy data (for which that was the lowest average rating) but because the values below that represented only one or two books each and so are less reliable for trending purposes.
(click for full-size version)
And what do we find? Pretty much the same thing. In this case, a rating of 4.4 and up correlates very closely with the average number of reviews at that rating. No book that had an average five-star rating had more than five (5) reviews. Interestingly, if you look at the plot for the maximum number of reviews at each average-rating point, you get the same effect: a very strong correlation between number of reviews and a lower average rating.
Now, of course, at some point this effect breaks down. At some point a lower average rating does start to reflect people's opinion of the specific book, even in the aggregate. And that seems to start somewhere in the lower 4's depending on the data set. And these trends are looking at aggregate behavior. It doesn't mean that there's no difference at all between a 4.4 rating and a 5.0 rating. What it means is that the meaningful difference between a book that has a 5.0 rating with X number of reviews, and a book in the same marketing category with a 4.4 rating and 50X number of reviews is not necessarily one of quality. What it means is that the second book is reaching outside its core audience. And is engaging them.
As someone who has been working very hard to reach outside what my publisher believes to be my core target audience, this is what I keep reminding myself when I get a "meh" rating or review. It means that I've succeeded.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2016-05-03 04:38 pm (UTC)no subject
Date: 2016-05-03 04:56 pm (UTC)no subject
Date: 2016-05-03 04:52 pm (UTC)You'd think Tolkien would fall under "Universally beloved." I certainly spent enough time with LOTR and "The Hobbit" in my childhood and teens. In 2002 on my drive across country to move here, I pulled "Fellowship" out before I packed, thinking it would be nice to revisit. I spent a lot of time griping to motel room walls about how desperately the Professor needed an editor!
Tastes vary *and* they change.
no subject
Date: 2016-05-03 04:57 pm (UTC)no subject
Date: 2016-05-03 07:51 pm (UTC)By the way - Tolkien's prose is one of my favorite things about his works, so there is always the matter of taste.
no subject
Date: 2016-05-03 10:46 pm (UTC)no subject
Date: 2016-05-03 10:00 pm (UTC)no subject
Date: 2016-05-03 10:46 pm (UTC)Your excellent blog
Date: 2016-05-04 09:19 am (UTC)Re: Your excellent blog
Date: 2016-05-04 04:45 pm (UTC)If nothing else, running these numbers makes it easier for me to not obsess over "OMG why did this person give me a 3-star rating? Where did I fail?" It's hard *not* to obsess when you can't comfort yourself with big royalty checks. So if I can turn even a 1-star rating into a strategic plus in my mind, it helps.
no subject
Date: 2016-05-04 04:18 pm (UTC)When I'm looking at reviews I ignore five star as they rarely tell me anything about the book to help me in deciding whether to buy or not. Mostly they reinforce the description but rarely do they contain what I consider useful information when I'm book buying. I look at three star and lower. Three star are usually the most useful reviews as many include the pros and cons. Quite a few one star reviews have sold me on a book. The 1-3 star reviews are also helpful in warning about PTSD triggers as well as tropes I prefer to avoid. While it's anecdotal, when I mention this, a number of people frequently chime in with "me too". Another thing 1-3 star reviews do is help the right readers find your books or prevent readers who are going to leave negative reviews due to content/tropes/PTSD triggers.
Unfortunately getting stats about readers like me is harder.
no subject
Date: 2016-05-04 04:39 pm (UTC)I keep trying to explain this phenomenon in LesFic circles: that reviews that sincerely explain *why* a book does/doesn't work for a reader are far more valuable than ones that praise it to the skies in generic terms. "[Author] blew me away again with her fabulous writing and characters" doesn't give me any reason to try a book. "I enjoyed how [author] explored [topic] but was the [plot feature] didn't really work for me" is far more valuable.
The real problem in these discussions comes in the smaller readerships. Small press/self-published books often have a relatively smaller potential market overall, so the apparent effect of individual ratings/reviews *feels* more important, even when the aggregate correlations still hold.
no subject
Date: 2016-05-04 05:47 pm (UTC)The real problem in my opinion is how little most authors educate themselves on the entire publishing industry so they believe myths. Too many people believe the false overnight success stories. Discoverability has always been a problem. It still is. Just as much for the majority of 1st time trad published books as for small press and self-published books. Self-publishing has an advantage over the others as authors can see immediately the effects of marketing and PR while small press and trad published authors have a harder time knowing if what they did worked or not. This depends on whether you can access your daily sales at multiple retailers.
Getting your first one star and three star reviews are times to celebrate. You've joined the big leagues. I tell my clients to look up their 5-10 favorite all time authors. I've yet to have one come back to me and say "they only have 4 & 5 star reviews". I'm also curious as to when being average became the end of the world. Shouldn't the majority of books be average? Isn't that what average means? I hate Amazon puts 3 star as part of bad/critical reviews as it should be 3 categories: good, neutral, critical.
Maybe too much of a rant? I've spent 5 years learning the industry (something to do while bedridden). The amount of misinformation/misunderstanding I see by authors published differently is depressing.
no subject
Date: 2016-05-04 08:18 pm (UTC)Sometimes it isn't a matter of believing myths, but of not realizing they've joined a different club with different rules and expectations. And, yes, if they don't educate themselves about the differences, there will be conflicts and unhappiness. (I have a bit of the opposite culture-clash: having developed my expectations in the context of mainstream SFF culture, and internalized that culture, I'm constantly being thrown for a loop by things I encounter in my publishing context and the LesFic reading community.)
And, yes, the mis-match between Amazon and Goodreads star ratings create a fair amount of anxiety.
no subject
Date: 2016-05-05 05:41 am (UTC)I don't have any good solutions. Most of my advice revolves around:
1. Ignore your reviews (hard I know but unless you intend to use feedback to improve your writing don't read)
2. Focus on getting your books written and published
3. Plan your social media time carefully so it's not being a time suck and emotional downer and is either helping build your author fan base or your getting support and/or having fun
4. Figure out who knows what they are talking about and hang out with them while staying away from those who don't because they become a time suck and leave one emotionally drained (hardest step)
Great Information
Date: 2016-05-04 05:35 pm (UTC)Re: Great Information
Date: 2016-05-04 08:22 pm (UTC)I don't see the Amazon data as the be all and end all, but rather as an index. It's a useful index because it's so public, but generalizing from it is only an approximation. I've been tracking Amazon sales data and then correlating against my royalty statements to see to what extent Amazon provides a useful real-time index to overall sales, and my preliminary conclusions are that it isn't useful at all! (In part probably because the nature of my readership and where they buy hasn't really settled down yet.)
How not to respond to reviews
Date: 2016-05-04 06:37 pm (UTC)The first review I did the author responded in a long comment why I was WRONG WRONG WRONG about her book, and her posse then chimed in and slagged off the reviewer.
The second review I did, for a book with a total of four 5-star reviews, I rated 4 stars while giving it quite a good review, and two seconds after it was posted, the review was voted down. Who but the author would have done that?
I no longer review for that organization and have pretty much quit reviewing anything at all. Our intent was to HELP indie authors, and I'm sure many authors have appreciated the feedback they've received, but it only took those two for me to chuck it in. I'm happy to help, but not if I'm going to be abused for it.
Clearly, in a club anyone can join, there will be folks like this, but please don't be one. Take HRJ's advice and see the advantage of having a variety of opinions attached to your work. I buy more books from reading 1-star reviews than 5-star reviews (which I don't usually read at all).
Catherine M. Wilson
Shield Maiden Press
Re: How not to respond to reviews
Date: 2016-05-04 08:26 pm (UTC)