We’re currently working on updating and developing survey items for different age groups for our research into university access interventions. We’re arranging a piloting session with a local school, where we’ll test out some survey items on our target age group to see whether they interpret them in the way we intend. One of the particular questions we’ve been discussing is
“I read for pleasure at home”
with response options 1 – 5, where 1 is ‘never’ and 5 is ‘very often’ (with 2 – 4 unspecified). Will the respondents all be thinking of the same thing when they choose a particular response – or is one person’s ‘often’ equivalent to someone else’s ‘sometimes’ or ‘rarely’?
The three of us in the meeting talked about how we would answer the question.
I’d pick 4. I read fiction most weeks in normal work time, more frequently when I’m on holiday (although not ‘most days’ as I first typed!); very frequently if I’m in the middle of a novel I’m really enjoying and for sustained periods if it’s the week before my six-weekly-ish book club. But sometimes the book club will come round and I’ve barely started the novel, and I won’t have read another in that month either. But I read the news every single day, and read blog posts and academic papers. But I didn’t really count that, as I interpret ‘read for pleasure’ as sitting down and reading fiction (although if pressed then I’d include e.g. popular science, biography etc too). (There’s some interesting attempts to define ‘reading for pleasure’ in this National Literacy Trust report.)
But the main reason I picked 4, which I think might be most interesting when related to question interpretation, is that I see myself as a clever, book-y sort of person – the kind of person that researchers would expect to read a lot. Instead of just answering the question in a straightforward way, I was – partially sub-consciously – trying to answer what the hypothetical researchers were really trying to find out.
One of my colleagues did something similar: she reads fiction daily, but answered 3 (half-way between ‘very often’ and ‘never’) because “it isn’t very high-quality fiction”. She “enjoys some of the journal articles” she reads, but didn’t count that as ‘reading for pleasure’.
My other colleague was more straightforward: she answered 2, because although she likes reading she doesn’t get the chance to read very often. We’re both in the same book club, so I know she’s read many of the same novels as me over the past year – but I saw myself as a frequent reader and she saw herself as close to never.
Is there a term in the literature for the phenomenon of respondents over-interpreting the question? I don’t think it’s the same as social desirability bias – I’m not picking my answer because I want to look good, but because I think I’ve understood what the researchers are really getting at and I want my answer to bring them to an accurate conclusion about me overall. But of course I don’t actually know what the researchers are getting at, so my over-thinking could introduce bias.
We’ve decided to ask our piloting group the question with two different scales: the subjective ‘very often’ – ‘never’ and objective frequency measures (daily, weekly, monthly etc.). We can then check to see whether respondents interpret these consistently. We’re also going to ask specifically about this question in the focus group discussions. We’re also planning to ask students about what they read (e.g. fiction, non-fiction, comics, news) to help us decide whether we should ask about this in future (and if it matters), and how they read it (e.g. books, kindle, smartphone). Findings to follow in the coming weeks…