Part I of this series examines the four main types of survey bias and the best practices market researchers can follow to avoid bias. In Part II, we focus on one of the most difficult aspects of research design — how to recognize (and fix!) the types of questions that lead to bias among respondents. 

Good questions are vital to meaningful insights

Even for the most seasoned market researchers, developing effective survey questions requires careful attention and considerable time. From avoiding jargon and steering clear of colloquialisms to ensuring consistency of rating scales and keeping things simple, there are numerous design elements to consider. Making sure the survey questions are structured in a manner that reduces the likelihood of bias among respondents is also essential to obtaining accurate information from participants and arriving at meaningful business insights.  

Understanding the question types that lead to bias

As noted in Part I, the order of the questions can lead to bias among respondents and must be scrutinized as part of the design process. Equally important is the structure of each question. Here we take a look at five question types that, if left unchecked, could lead to biased responses from survey participants. 

#1. Assumptive or Loaded Questions

Loaded questions assume a respondent behaves in a certain way or has had a specific experience. For example, “What prescriptions do you take?” is a loaded question because it assumes the person has been prescribed medication. 

This can be solved by applying conditional skip logic and first asking “Do you take any prescriptions?” Those who answer “yes” are then directed to questions that gather more detail, such as how many and what types of medications the respondents take. Those who answer “no” are moved to the next section of the survey, ensuring that only those that take prescriptions are asked to provide more detail. If for some reason you’re unable to use skip logic in your survey, another approach is to include the option “I do not take any prescriptions” in the list of responses. 

#2. Leading Questions

The main problem with leading questions is that you’re putting words in the mouths of your respondents and, thereby, forcing them to select an answer option that they may not agree with completely. You can avoid this by making sure that none of your questions are suggestive in any way. 

Consider the question, “Would you recommend our easy-to-use health care app to others?” It suggests to respondents that they believe the app to be easy-to-use when many of them may not feel that way. Leading questions can be corrected by remaining objective and always providing a balanced selection of positive, neutral, and negative options for each question. Or, you can also allow respondents to describe their experiences in their own words as a means for eliminating the bias that occurs with leading questions. 

#3. Double-barreled Questions

Double-barreled questions cause problems because they ask respondents to provide their opinions about two different things in one question, making it difficult for respondents to give accurate answers. Double-barreled questions are often the result of trying to fit too many questions into a single survey. Unfortunately, relying on shortcuts to reduce the total number of survey questions will negatively affect data quality

For instance, instead of asking, “How would you rate your appointment and payment experience today?” break the question into two: 

• “How would you rate your appointment experience?”

• “How would you rate your payment experience?”

#4. Double-negative Questions

Double negative questions are exactly like they sound: two negative words used in the same question. It’s a structure that can cause respondents to misinterpret what is being asked. Researchers can prevent this from occurring by wording all questions in a neutral manner.

Be on the lookout for negative words such as “not,” “no,” and “don’t,” especially when paired with prefixes like “un,” “non,” or “mis.” 

• Don’t: “Do you agree with the statement that the instructions for setting up the heart monitoring device were not too unclear?” 

• Do: “How clear were the instructions for setting up the heart monitoring device?” Then, provide a balanced answer key that allows respondents to rate their experience accurately. 

#5. Dichotomous Questions

These are questions that only have two choices, such as “yes or no” or “true or false.” While there will be times that these questions are relevant, especially in setting up conditional skip logic, the risk that comes with dichotomous questions is that the range of responses is extremely restricted. A good rule of thumb is to see if you can ask a similar question that offers more options. This helps you get more detailed insights for the same amount of respondent effort.

To illustrate, the question “Do you know what diabetes is?” with only a “yes or no” option can create confusion because respondents are left to determine what exactly is meant by “know.” Some respondents will select “yes” because they’ve heard of diabetes but don’t really know much more. Yet, others who also have heard of the condition but don’t know much about it will interpret that as indicating they should choose “no.”

A better approach is to ask, “How much do you know about diabetes?” followed by a single-selection of responses that participants can use to rate their level of understanding: 

• A little

• A fair amount

• A great deal

• Nothing

• I’m not sure

Remaining vigilant about the structure and wording of each survey question is vital to the design and execution of  effective surveys. When in doubt, always go back to the focus of your survey and assess your questions through that lens. Then, eliminate the questions you don’t need and work to optimize the ones that are most important to your desired insights. And, to double-check for bias, do at least one test run of the survey before you distribute it. You’ll be able to gauge its performance and fix problems with the structure and wording of your questions.