[If you want to skip the discussion of survey mechanics and go right to the results, scroll down to the section with that label.]
In my corporate career, I was a survey professional designing auxiliary sample frames, and subsequently hybrid selection methods, to help compensate for differential non-response. I realize there’s a lot of jargon in that sentence, but that’s the fastest way to say it, and anyway it’s not the point. The point is, these results need major caveats. And I’m going to throw in a nice, juicy rant just for good measure.
First of all, it’s important to note that these numbers don’t MEAN anything in some absolute sense. Nothing. Zip. Zilch. This is what’s called an opt-in survey, which means the only people who filled it out are those who were interested in it — sort of like an ESPN poll or Cosmo online question.
To understand why those are bunk, you have to understand a little about survey non-response. If you’re taking a survey of cancer patients — for example, to determine the efficacy of various treatments — and you send out a survey, the people who respond will disproportionately be those in better health. (If you’re in the hospital dying of cancer, you tend not to want to fill out mail surveys from strangers.) The results of your survey, then, will be biased. If you don’t compensate for that in your methodology — by weighting the responses, by oversampling the less responsive group, by using differential incentives, and so on — you’re going to draw erroneous conclusions about cancer patients with potentially disastrous results, especially if your client is a policymaker.
Non-response bias isn’t the only challenge facing survey research professionals — if your sampling frame (basically, the big list of names or addresses or whatever that you draw from) is missing a chunk of your target population, or contains numerous phantom duplicates (such as individuals with multiple personal phone numbers), you’ll get biased results from differential probability of selection — but non-response is the big one.
The problem in my case is that my target population is opaque and I couldn’t design a sample frame with good coverage even if it were clear. This is because my target population is something ephemeral like “All English-speaking genre-fiction readers aged 18+ who have a reasonable probability of enjoying books similar to mine.” (I’m not writing for everyone. No one is. That’s stupid.)
This is not a unique challenge of course. In fact, most marketers have to resort to various proxies — some combination of known characteristics: age, gender, location, education, income, race/ethnicity, language spoken in the home, number of televisions, presence of pets, etc. — to get at the bulk of people likely to buy their product. But the real world is messy, and sometimes there isn’t great overlap between the whole of your customer base and the proxies available to you.
This occasionally gets marketers into hot water when, for example, they market their product to one gender at the expense of the other and the torch-and-pitchfork crowd objects. People, especially those on the internet, like to think marketers target specific demographics because marketers are evil and sexist. Thing is, I’m sure some are, but usually the reason is because they know something you don’t: the sales figures. If 85% of your sales come from white college-educated males aged 18-54, and you have a limited budget, then regardless of how you personally would like the world to be, who are you realistically gonna target with those limited dollars?
It’s like asking a fisherman to spend his day covering the entire lake equally, to be fair, even though he knows for a fact that most of the fish are in one nook on the north side, and even then only in the mornings, and oh by the way, there are almost none in the middle, and so covering the whole lake, or even any part but the north nook in the mornings, is a giant waste of time and money. Reasonable people fish when and where the fish are. That doesn’t necessarily make them evil — although, again, some of them might be (for other reasons).
Recall in my example I said the marketers know that 85% of their customer base shares certain characteristics, and that the real world is messy. That last 15% — obviously these are just hypothetical numbers — will contain a hodgepodge of different groups, none of whom were specifically marketed to. If you are in one of those groups, you might assume there are a whole lot of people out there like you and that therefore those marketers are just being silly, or willfully ignorant, or sexist, or whatever. And again, they might be. Such things happen. My only point is that it’s a logical fallacy to extrapolate from your experience — a sample size of one — to the world at large, and that marketers are generally greedy people, and they have lots of data, and if the data suggested there was a large untapped market — i.e., money to be made — they usually go after it.
But then, it’s usually not clear what people will buy, and so sometimes they play it unnecessarily safe. After all, if there’s one thing you learn in the survey business, and especially in the smaller industry of public opinion polling, it’s that there’s a BIG difference between what people say they want and what they actually DO. People like there to be fair and equal consumer product options for all races and genders because that’s nice and it makes the world seem fair and such things are free to them. (They’re someone else’s problem to create and fund.) But consumers only see what is or isn’t on the shelf whereas marketers know what we actually and regularly support with dollars.
You see this behavior everywhere. In politics, for example, citizens report time and time again — going all the way back to the advent of scientific polling in the 1930s — that they want to “throw the bastards out.” And yet, in practice they overwhelmingly vote for the local incumbent (or they don’t vote). Every. Damn. Election. They like their elected official just fine. It’s all those other voters who are the problem!
This is not just me bitching by the way. This is an actual, measurable fact of the world. AND it’s me bitching.
Side Note: What people ESPECIALLY don’t like, more even than the status quo, is when you point out their occult hypocrisies — such as how, at any point that actually matters, they tend to actively support the status quo. That knocks fact and self-image out of alignment, which in turn creates cognitive dissonance, or at least it would if they didn’t compensate by attacking the messenger (they just don’t understand how the world works, you see) rather than addressing the discrepancy. But that’s another story.
To date, I haven’t been able to identify which granular proxies best fit my target audience — What corollary media do they consume, such as what types of music? What websites do they frequent? Are they gamblers? What products do they buy in addition to my books, such as lube and adult diapers? — but I know the global ones. They’re English-speakers, of course, with slightly more women than men, I’d guess, since women make up something like 65-70% of the fiction market. I don’t write Young Adult, so they’re mostly going to be persons 18+ — not that some teenagers wouldn’t like my books, but again, given limited resources, you gotta fish where you reasonably expect the bulk of the fish are. Most of them probably own some kind of e-reading device since those folks tend to be heavier readers. Those folks also tend to care less if a book is traditionally versus indie-published. And so on and so forth.
But even after all that, I’m still left with a big — and therefore expensive — market to target, and so for this study I kept to my little pond and ignored the big lake, let alone the giant ocean that is the paid survey market.
Discussion of Results
An N of 33 is about the bare minimum for making any conclusions, no matter how slight. Samples of 30-35 are about where your margin of error slips into the acceptable range, assuming “average” variance. (A highly variable population of study requires a higher sample size to achieve the same level of confidence.) In short, there were enough responses here to validate a top-line analysis, but ONLY a top-line analysis.
The actual ratings mean nothing because, following my example above, I’m missing 100% of the “less healthy cancer patients” — those who don’t know or are on the fence about my books. The people who responded to this are those who already like and engage with my work. Thus, that the results are generally positive is both predictable and meaningless. It’s like saying “the people who like my books like my books.” Well, duh.
As I mentioned, the value of this survey is not the absolute numbers but their comparison, first between the categories measured, but also over time. I can, for example, repeat this exact survey at a later date and see if there is any significant change as my audience (hopefully) grows — what we call a longitudinal measure. I could (and probably will) add demographic markers in the future so that I can track responses by gender or age or average length of dildos purchased. But for now, I’m limited to top-line comparisons.
Respondents rated me highest on Originality followed by Action and then Plot. Premise, Characterization, and Pace were all basically tied for fourth place. Several of those are related, which shouldn’t be surprising (that’s co-linearity). All six of those together pretty much cover everything I emphasize in my writing since they are also what I enjoy reading. So I appear to be hitting what I’m aiming at.
At the other end, respondents rated me lowest in Emotional Development (my ex would agree!) followed by Descriptions and Setting. The latter was the subject of a blog post earlier this year after I had a small epiphany about setting, so all other things being equal, I would expect that to improve over time based on some recent changes I introduced.
As for the other two, my general emphasis on action, plot, and faster pace limits how much time I can realistically devote to expanded descriptions and characters’ deep emotional development, a point noted by several respondents in the open-ended comments section. I am not writing literary fiction, nor do I want to. However, that doesn’t mean those aren’t growth areas for me. Of course they are.
What I found most interesting, though, was not that those two categories were lowest — I expected that — but that the relative difference between them and my strengths was comparatively slight, and less than I expected: a low-to-high range of 3.82-4.48.
Recall, because of the nature of this survey, those numbers don’t mean anything in the absolute (even though I framed the question as if they did). Also, since all the respondents are already engaged with my work, the high-to-low range is artificially compressed. But still, what this says to me is that “the people who presently like my books don’t see a huge gap between my strengths and my weaknesses.”
It very well could be that, if more of the “less healthy cancer patients” responded, that gap would grow. That seems likely. (However, it’s also possible the gap would remain narrow but the whole range would slide down.) But if we assume a basic honesty among the respondents, then at the very least I probably don’t have a single catastrophic problem, which is significant — because it was always possible I could have.
That conclusion is not proved, given the limits of this survey, but it seems likely, especially since it’s completely corroborated by the Overall measure, where every respondent, despite whatever else they flagged as needing improvement, still put their total reader experience exclusively in the top two categories, suggesting none of my weaknesses are so bad as to inhibit their enjoyment of the books. Great!
A quick note about Grammar/Spelling/Mechanics: That’s less a measure of me than of Karen, my editor. I included it because it’s part of the reader experience — a big part — and because I can control the outcome — by getting a new editor, for example. Not surprisingly, she earns high marks, so no action needed. (I would have been shocked if the results were different.)
Final Summary: Nothing proved — the response pattern and sample size limit all but the highest-level suppositions. However, there were no surprises. I have room to grow in those areas outside my traditional emphasis, one of which was already being addressed, but there is probably no catastrophic deficit. Both time and further analysis will tell. Overall, a passable first measure.
That’s it! And by the way, if anyone needs help designing their survey, I’m happy to answer questions.