BOLD Educational Software
Survey Design Tips
Visit our Web Store
Return to B.A.R.R.C.

 
Survey Design Tips
Review your problem statement

The first step in any survey is determining what the problem is you are trying to solve.
If you do not have a clear vision of your problem statement, you don't understand what you need to do.
Remember that a problem is something that needs to be solved.
It is NOT your purpose statement.
Review your purpose statement The purpose of the study will help you determine whom you will survey and what you will need to ask them.
If your purpose is vague, the results will be difficult to interpret.
Define your research variables

You cannot design your survey without clearly understanding the operational definitions of your research variables. Typical research variables fall into 5 categories:

  1. Nominal level variables: unique, unordered groups (e.g., gender, ethnicity)
  2. Ordinal level variables: unique, ordered scale with no distinct measurement between levels (e.g., rank - first, second, third)
  3. Interval level variables: unique ordered scale with a set level of measurement between levels (e.g. temperature)
  4. Ratio level variables: unique ordered scale with a set level of measurement between levels and an absolute zero on the scale (the absence of the variable: e.g., income)
  5. Rich qualitative data: answers to open-ended question (e.g., describe what happened, how do you feel about . . .)
Operationalize your research variables Once you've defined your variables, determine how you will measure them. Remember, many social/psychological variables are measured by multiple questions, the responses are summed and the final variable is a summed score from all of the questions. For example, don't just ask, "Do you get nervous before tests?" with a yes/no response. Ask a few questions that measure more directly how nervous do respondents get: does your heart start to race? do you palms get sweaty? do you consider skipping class because there is a test?
Define your population Determine the group you would like to generalize your results to. Is your population small (e.g., one school, one business, one town)? Is your population larger (e.g., all schools in the state, all small businesses in New England)? Is your population very large (e.g., all high school students in the United States)?
Select your sample Decide whom you can include in your research. This group is called the target population. Make sure that your 'target population' (those you can reach in the sampling process) are representative of the population you wish to generalize to.
Determine your sample size A small, representative sample will reflect the population from which it is drawn. Larger samples do not necessarily represent the population. Random sampling is the best way to adequately ensure that your sample responses reflect the population as a whole. Use the sample size calculator to assist.
Select your sampling method Determine how you will select your sample. There are a variety of ways, but the two main categories are random and nonrandom. Usually it's best to select a random sample because it will be far more likely to truly represent your population. However, there are a variety of nonrandom sampling techniques that are appropriate for certain types of research. What is usually not a good idea though, is to select a convenience sample, or worse, to survey the entire population and allow the respondents to self-select. This is a sure way to get a biased (nonrepresentative) sample.
Determine how to survey your sample Once you've identified your population, will you survey them in person, by mail, by email, telephone, online? There are benefits and disadvantages to each method.
Create your survey

Make sure your survey questionnaire fits the distribution medium. You (probably) cannot show a picture by phone unless all your respondents have blackberries. Email or web surveys make it hard to ask follow-up questions. Personal questions are often best asked anonymously (e.g., by web survey).

Keep the survey short and simple. If it takes more than 15 minutes to respond to the survey, a lot of your potential participants will 'opt out'.

Start with an introduction or welcome message. A well-written introduction or welcome message will encourage respondents to complete the survey.

Allow “Don't Know”,“Not Applicable”, "Other", or "None" responses. Statistically, this is better for your analysis than a blank response, where you don't know if the respondent missed the question by accident.
Select appropriate question types Open-ended or Closed? While in general, open-ended are appropriate for qualitative studies and closed-ended are appropriate to quantitative studies, you can include some open-ended questions in a quantitative or mixed-methods study, and you can include some closed-ended questions in a qualitative study to clarify responses.
Writing open- and closed-ended questions

Open-ended questions do not have one definite answer. These are usually more appropriate to qualitative studies.

Closed-ended questions have a finite set of answers from which the respondent chooses. One of the choices may be "Other." You may want to allow respondents to write in an optional response if they choose "Other." Closed-ended questions are easy to standardize, and data gathered from closed-ended questions are easier to analyze using descriptive and inferential statistics.

Review the purpose of the study Before writing open- or closed-ended questions, review your problem statement, purpose statement, research questions, and the definitions of your research variables. If any of your survey questions goes beyond the scope of these items, it doesn't belong in your survey! One exception: you may want to collect demographic data such as age, gender, education, income level, to help you explain who your respondents were and support the assumption that the sample is representative of your population.
Styles of closed-ended questions

Likert-type scale: Used to discover respondents' feelings or attitudes. Respondents indicate how closely their feelings match a statement on a rating scale. The number at one end of the scale represents "Strongly Disagree," and the number at the other end of the scale represents "Strongly Agree." Decide whether or not to include a neutral response in the middle. Neutral responses are not always good to include because it does not force your respondents to decide whether they agree or disagree. To get a broader scale, it's often better to ask repondents to choose on a scale of 1 to 7, with 1 being "Strongly disagree" and 7 being "strongly agree". If you are planning a Pearson correlation, regression, or other statistical measure that assumes a linear relationship between variables, a larger scale (5-7 possible responses per question, and sum at least 3 questions per variable) is better than a smaller scale (3-4 possible responses and only 1 or 2 questions per variable).

Multiple-choice: Ask respondents to pick the best answer or answers from among all the possible options.
Multiple choice questions can ask nominal (male/female), ordinal (do you prefer coffee or tea), interval (What year did you begin working full time?), or ratio (how many months have you been employed at this firm?).
Pilot test your survey

There are two ways to pilot test your survey. One is to find content experts to review and evaluate your questions. Content experts can help ensure that you have construct validity: that you are measuring what you intend to measure. It's usually best to use content experts first. Then find a small sample from your research population who will NOT be included in your final study. Often 10 pilot participants are sufficient. Ask the participants to take the survey then provide feedback. Ask your pilot participants:

  1. Were you able to understand the questions?
  2. For multiple choice questions, did all the answers make sense? Were any answer options missing?
  3. How long did it take you to complete the survey?
  4. What recommendations do you have to improve the survey?
  5. Were the instructions clear?
Do a "practice" analysis

To be sure you really understand the operationalization of your variables and the analysis, open your statistics software (Excel, SPSS, NVivo, or other software), enter some 'dummy' survey responses and conduct the analysis. Are the variables appropriate for the analysis (e.g., do you have a grouping variable with 2 groups for a t-test, 3 or more groups for ANOVA)?

Make appropriate changes, or consult with a professional researcher before continuing. It's best to discover problems with your survey and methodology BEFORE you receive IRB/ARB approval!

Need help with research methodology, survey design, data analysis, or editing?
Contact us at editing@bold-ed.com or survey@bold-ed.com

Return to the BOLD Educational Software webstore
Return to the BOLD Educational Software ACADEMIC RESEARCH RESOURCE CENTER

Return to the B.O.S.S.

BOLD Online Survey System

Copyright BOLD Educational Software 2009
by Diane M. Dusick, Ph.D.
All Rights Reserved