Self-Reporting Limitations and How to Mitigate

…or, “I often pick my nose in public!”

Most market research relies on subjects to tell the truth and respond to questions as honestly as possible. We set the stage by making the survey voluntary, encouraging different opinions in a focus group or letting participants know that there are no wrong answers.

But some questions are harder than others to answer honestly. When a respondent knows that his answer is “wrong” (morally, legally, ethically or socially), he may tend to offer the “right” answer instead. After all, there are no repercussions to him for answering in a normative way. Yet, as great interrogators might say, “We have ways of making you talk!”Right-vs.-Wrong

By its very nature, market research centers on self-reporting. There is evidence that the more personal the format, i.e. an in-person interview, the more likely subjects are to self-report falsely. We have witnessed increased truthfulness in online surveys, which are far less personal.

During the research phase of a past campaign, we asked subjects, over the phone, how often they brush their teeth. Nearly everyone knows that ‘twice a day’ is the right answer, and as a result, an overwhelming majority offered this answer. Interestingly, the survey population has a very high incidence of cavities. Brushing-TeethIf they truly brushed twice a day, they likely wouldn’t experience such a great degree of tooth decay. When we conducted an online survey of the same audience, we noticed significantly fewer ‘twice a day’ answers. The impersonal nature of an online survey allowed respondents to answer more truthfully than they did on the phone.

In another case, we probed into rampant abuse of handicapped parking spots. Here again, subjects knew full well that they were doing the wrong thing. In focus groups, these participants denied ever having parked illegally in a disabled parking spot.images

To get people to open up, we decided to try something new: the moderator admitted to having done it on occasion herself. Once this person in a position of authority confessed to her own counter-normative behavior, the subjects readily recounted their own incidents and motivations.

The most credible market research findings will always be based upon observational studies, especially where stigmatized or socially unacceptable behavior is concerned. But when more in-depth insights are required, consider research tactics that allow subjects to report freely, honestly and guiltlessly.

Three Ways to Test a Campaign

If you are like most marketers, you lose sleep thinking about the various ways your campaign could run amok, blow up or otherwise fail. Let’s face it, the list of reasons things could go wrong is nightmarishly long: wrong files delivered to media outlets, talent not showing up on time, legal team nixing the product name at the eleventh hour or the client accidentally ingesting drugs (yes, that really happened). There seems to be no shortage of new, innovative ways for the proverbial wheels to come off of your marketing machine.

wheels-off-hummer

Rest assured. You can control, to a great degree, the extent to which your campaign will resonate with your target using any of these three basic testing methods.

Focus Groups
Hold small, in-person focus groups–dyads or triads are best–with members of your target to hear feedback or to conduct A/B testing of potential campaign directions. Conduct at least three focus groups so that you can discard one; sometimes participants are just not very participatory. Ask questions such as: What is your first reaction to this? How likely are you to take the suggested action? How likely are you to talk about this campaign with a friend?

Online Testing
Conduct an online survey of 10 questions among the target audience. If you do not have emails for this audience (via client contacts, partners, friends or family), respondent panels can always be purchased online. Don’t be afraid to post images or links as part of the survey, but be sure to test it thoroughly. Take the survey in test mode several times and ask others to take it as well. Input different answers to make sure that all questions, branch logic and programming are working correctly. Don’t forget to delete your test “completes” before launching.

Non-Traditional Testing
Promotional events and tradeshows offer the perfect opportunity to test your campaign. Use your party, website launch or booth as the venue to solicit campaign feedback. In some cases, you may be able to test different messages or sales pitches. You’ll get more consistent, reliable results if you use documented, consistent questions or discussion points for each respondent.