For the Statistics Canada publication, see Survey Methodology.
A field of applied statistics of human research surveys, survey methodology studies the sampling of individual units from a population and the associated survey data collection techniques, such as questionnaire construction and methods for improving the number and accuracy of responses to surveys. Survey methodology includes instruments or procedures that ask one or more questions that may, or may not, be answered.
Statistical surveys are undertaken with a view towards making statistical inferences about the population being studied, and this depends strongly on the survey questions used. Polls about public opinion, public health surveys, market research surveys, government surveys and censuses are all examples of quantitative research that use contemporary survey methodology to answer questions about a es do not include a "sample," they do include other aspects of survey methodology, like questionnaires, interviewers, and nonresponse follow-up techniques. Surveys provide important information for all kinds of public information and research fields, e.g., marketing research, psychology, health professionals and sociology.
A single survey is made of at least a sample (or full population in the case of a census), a method of data collection (e.g., a questionnaire) and individual questions or items that become data that can be analyzed statistically. A single survey may focus on different types of topics such as preferences (e.g., for a presidential candidate), opinions (e.g., should abortion be legal?), behavior (smoking and alcohol use), or factual information (e.g., income), depending on its purpose. Since survey research is almost always based on a sample of the population, the success of the research is dependent on the representativeness of the sample with respect to a target population of interest to the researcher. That target population can range from the general population of a given country to specific groups of people within that country, to a membership list of a professional organization, or list of students enrolled in a school system (see also sampling (statistics) and survey sampling). The persons replying to a survey are called respondents, and depending on the questions asked their answers may represent themselves as individuals, their households, employers, or other organization they represent.
Survey methodology as a scientific field seeks to identify principles about the sample design, data collection instruments, statistical adjustment of data, and data processing, and final data analysis that can create systematic and random survey errors. Survey errors are sometimes analyzed in connection with survey cost. Cost constraints are sometimes framed as improving quality within cost constraints, or alternatively, reducing costs for a fixed level of quality. Survey methodology is both a scientific field and a profession, meaning that some professionals in the field focus on survey errors empirically and others design surveys to reduce them. For survey designers, the task involves making a large set of decisions about thousands of individual features of a survey in order to improve it.
The most important methodological challenges of a survey methodologist include making decisions on how to:
- Identify and select potential sample members.
- Contact sampled individuals and collect data from those who are hard to reach (or reluctant to respond)
- Evaluate and test questions.
- Select the mode for posing questions and collecting responses.
- Train and supervise interviewers (if they are involved).
- Check data files for accuracy and internal consistency.
- Adjust survey estimates to correct for identified errors.
Main article: Survey sampling
The sample is chosen from the sampling frame, which consists of a list of all members of the population of interest. The goal of a survey is not to describe the sample, but the larger population. This generalizing ability is dependent on the representativeness of the sample, as stated above. Each member of the population is termed an element. There are frequent difficulties one encounters while choosing a representative sample. One common error that results is selection bias. Selection bias results when the procedures used to select a sample result in over representation or under representation of some significant aspect of the population. For instance, if the population of interest consists of 75% females, and 25% males, and the sample consists of 40% females and 60% males, females are under represented while males are overrepresented. In order to minimize selection biases, stratified random sampling is often used. This is when the population is divided into sub-populations called strata, and random samples are drawn from each of the strata, or elements are drawn for the sample on a proportional basis.
Modes of data collection
Main article: Survey data collection
There are several ways of administering a survey. The choice between administration modes is influenced by several factors, including
- coverage of the target population,
- flexibility of asking questions,
- respondents' willingness to participate and
- response accuracy.
Different methods create mode effects that change how respondents answer, and different methods have different advantages. The most common modes of administration can be summarized as:
- Mail (post)
- Online surveys
- Personal in-home surveys
- Personal mall or street intercept survey
- Hybrids of the above.
There are several different designs, or overall structures, that can be used in survey research. The three general types are cross-sectional, successive independent samples, and longitudinal studies.
In cross-sectional studies, a sample (or samples) is drawn from the relevant population and studied once. A cross-sectional study describes characteristics of that population at one time, but cannot give any insight as to the causes of population characteristics because it is a predictive, correlational design.
Successive independent samples studies
A successive independent samples design draws multiple random samples from a population at one or more times. This design can study changes within a population, but not changes within individuals because the same individuals are not surveyed more than once. Such studies cannot, therefore, identify the causes of change over time necessarily. For successive independent samples designs to be effective, the samples must be drawn from the same population, and must be equally representative of it. If the samples are not comparable, the changes between samples may be due to demographic characteristics rather than time. In addition, the questions must be asked in the same way so that responses can be compared directly.
Longitudinal studies take measure of the same random sample at multiple time points. Unlike with a successive independent samples design, this design measures the differences in individual participants’ responses over time. This means that a researcher can potentially assess the reasons for response changes by assessing the differences in respondents’ experiences. Longitudinal studies are the easiest way to assess the effect of a naturally occurring event, such as divorce that cannot be tested experimentally. However, longitudinal studies are both expensive and difficult to do. It’s harder to find a sample that will commit to a months- or years-long study than a 15-minute interview, and participants frequently leave the study before the final assessment. This attrition of participants is not random, so samples can become less representative with successive assessments. To account for this, a researcher can compare the respondents who left the survey to those that did not, to see if they are statistically different populations. Respondents may also try to be self-consistent in spite of changes to survey answers.
Questionnaires are the most commonly used tool in survey research. However, the results of a particular survey are worthless if the questionnaire is written inadequately. Questionnaires should produce valid and reliable demographic variable measures and should yield valid and reliable individual disparities that self-report scales generate.
Questionnaires as tools
A variable category that is often measured in survey research are demographic variables, which are used to depict the characteristics of the people surveyed in the sample. Demographic variables include such measures as ethnicity, socioeconomic status, race, and age. Surveys often assess the preferences and attitudes of individuals, and many employ self-report scales to measure people’s opinions and judgements about different items presented on a scale. Self-report scales are also used to examine the disparities among people on scale items. These self-report scales, which are usually presented in questionnaire form, are one of the most used instruments in psychology, and thus it is important that the measures be constructed carefully, while also being reliable and valid.
Reliability and validity of self-report measures
Reliable measures of self-report are defined by their consistency. Thus, a reliable self-report measure produces consistent results every time it is executed. A test’s reliability can be measured a few ways. First, one can calculate a test-retest reliability. A test-retest reliability entails conducting the same questionnaire to a large sample at two different times. For the questionnaire to be considered reliable, people in the sample do not have to score identically on each test, but rather their position in the score distribution should be similar for both the test and the retest. Self-report measures will generally be more reliable when they have many items measuring a construct. Furthermore, measurements will be more reliable when the factor being measured has greater variability among the individuals in the sample that are being tested. Finally, there will be greater reliability when instructions for the completion of the questionnaire are clear and when there are limited distractions in the testing environment. Contrastingly, a questionnaire is valid if what it measures is what it had originally planned to measure. Construct validity of a measure is the degree to which it measures the theoretical construct that it was originally supposed to measure.
It is important to note that there is evidence to suggest that self-report measures tend to be less accurate and reliable than alternative methods of assessing data (e.g. observational studies; for an example, see.
Composing a questionnaire
Six steps can be employed to construct a questionnaire that will produce reliable and valid results. First, one must decide what kind of information should be collected. Second, one must decide how to conduct the questionnaire. Thirdly, one must construct a first draft of the questionnaire. Fourth, the questionnaire should be revised. Next, the questionnaire should be pretested. Finally, the questionnaire should be edited and the procedures for its use should be specified.
Guidelines for the effective wording of questions
The way that a question is phrased can have a large impact on how a research participant will answer the question. Thus, survey researchers must be conscious of their wording when writing survey questions. It is important for researchers to keep in mind that different individuals, cultures, and subcultures can interpret certain words and phrases differently from one another. There are two different types of questions that survey researchers use when writing a questionnaire: free response questions and closed questions. Free response questions are open-ended, whereas closed questions are usually multiple choice. Free response questions are beneficial because they allow the responder greater flexibility, but they are also very difficult to record and score, requiring extensive coding. Contrastingly, closed questions can be scored and coded much easier, but they diminish expressivity and spontaneity of the responder. In general, the vocabulary of the questions should be very simple and direct, and most should be less than twenty words. Each question should be edited for "readability" and should avoid leading or loaded questions. Finally, if multiple items are being used to measure one construct, the wording of some of the items should be worded in the opposite direction to evade response bias.
A respondent's answer to an open-ended question can be coded into a response scale afterwards, or analysed using more qualitative methods.
Order of questions
Survey researchers should carefully construct the order of questions in a questionnaire. For questionnaires that are self-administered, the most interesting questions should be at the beginning of the questionnaire to catch the respondent’s attention, while demographic questions should be near the end. Contrastingly, if a survey is being administered over the telephone or in person, demographic questions should be administered at the beginning of the interview to boost the respondent’s confidence. Another reason to be mindful of question order may cause a survey response effect in which one question may affect how people respond to subsequent questions as a result of priming.
The following ways have been recommended for reducing nonresponse in telephone and face-to-face surveys:
- Advance letter. A short letter is sent in advance to inform the sampled respondents about the upcoming survey. The style of the letter should be personalized but not overdone. First, it announces that a phone call will be made, or an interviewer wants to make an appointment to do the survey face-to-face. Second, the research topic will be described. Last, it allows both an expression of the surveyor's appreciation of cooperation and an opening to ask questions on the survey.
- Training. The interviewers are thoroughly trained in how to ask respondents questions, how to work with computers and making schedules for callbacks to respondents who were not reached.
- Short introduction. The interviewer should always start with a short introduction about him or herself. She/he should give her name, the institute she is working for, the length of the interview and goal of the interview. Also it can be useful to make clear that you are not selling anything: this has been shown to lead to a slightly higher responding rate.
- Respondent-friendly survey questionnaire. The questions asked must be clear, non-offensive and easy to respond to for the subjects under study.
Brevity is also often cited as increasing response rate. A 1996 literature review found mixed evidence to support this claim for both written and verbal surveys, concluding that other factors may often be more important. A 2010 study looking at 100,000 online surveys found response rate dropped by about 3% at 10 questions and about 6% at 20 questions, with drop-off slowing (for example, only 10% reduction at 40 questions). Other studies showed that quality of response degraded toward the end of long surveys.
Survey methodologists have devoted much effort to determining the extent to which interviewee responses are affected by physical characteristics of the interviewer. Main interviewer traits that have been demonstrated to influence survey responses are race, gender, and relative body weight (BMI). These interviewer effects are particularly operant when questions are related to the interviewer trait. Hence, race of interviewer has been shown to affect responses to measures regarding racial attitudes, interviewer sex responses to questions involving gender issues, and interviewer BMI answers to eating and dieting-related questions. While interviewer effects have been investigated mainly for face-to-face surveys, they have also been shown to exist for interview modes with no visual contact, such as telephone surveys and in video-enhanced web surveys. The explanation typically provided for interviewer effects is social desirability bias: survey participants may attempt to project a positive self-image in an effort to conform to the norms they attribute to the interviewer asking questions. Interviewer effects are one example survey response effects.
- ^George Beam. The Problem with Survey Research (2012) New Brunswick, NJ: Transaction, p. xv.
- ^"WhatIsASurvey.info". WhatIsASurvey.info. Retrieved 2013-10-03.
- ^ abGroves, R.M.; Fowler, F. J.; Couper, M.P.; Lepkowski, J.M.; Singer, E.; Tourangeau, R. (2009). Survey Methodology. New Jersey: John Wiley & Sons. ISBN 978-1-118-21134-2.
- ^ abcdefghijklmnopqrstuvwxyzaaabacadaeafagahaiajakalamanaoapaqShaughnessy, J.; Zechmeister, E.; Jeanne, Z. (2011). Research methods in psychology (9th ed.). New York, NY: McGraw Hill. pp. 161–175.
- ^ abMellenbergh, G.J. (2008). Chapter 9: Surveys. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research Methods: A consultant's companion (pp. 183–209). Huizen, The Netherlands: Johannes van Kessel Publishing.
- ^Prince et al., 2008)
- ^Lynn, P. (2008) "The problem of non-response", chapter 3, 35-55, in International Handbook of Survey Methodology (ed.s Edith de Leeuw, Joop Hox & Don A. Dillman). Erlbaum. ISBN 0-8058-5753-2
- ^Dillman, D.A. (1978) Mail and telephone surveys: The total design method. Wiley. ISBN 0-471-21555-4
- ^De Leeuw, E.D. (2001). "I am not selling anything: Experiments in telephone introductions". Kwantitatieve Methoden, 22, 41–48.
- ^Bogen, Karen (1996). "THE EFFECT OF QUESTIONNAIRE LENGTH ON RESPONSE RATES -- A REVIEW OF THE LITERATURE"(PDF). Proceedings of the Section on Survey Research Methods. American Statistical Association: 1020–1025. Retrieved 2013-03-19.
- ^"Does Adding One More Question Impact Survey Completion Rate?". 2010-12-10. Retrieved 2017-11-08.
- ^"Respondent engagement and survey length: the long and the short of it". research. April 7, 2010. Retrieved 2013-10-03.
- ^Hill, M.E (2002). "Race of the interviewer and perception of skin color: Evidence from the multi-city study of urban inequality". American Sociological Review. 67 (1): 99–108. doi:10.2307/3088935. JSTOR 3088935.
- ^Flores-Macias, F.; Lawson, C. (2008). "Effects of interviewer gender on survey responses: Findings from a household survey in Mexico". International Journal of Public Opinion Research. 20 (1): 100–110. doi:10.1093/ijpor/edn007.
- ^Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B.; Van Strien, T. (2011). "BMI of interviewer effects". International Journal of Public Opinion Research. 23 (4): 530–543. doi:10.1093/ijpor/edr026.
- ^Anderson, B.A.; Silver, B.D.; Abramson, P.R. (1988). "The effects of the race of the interviewer on race-related attitudes of black respondents in SRC/CPS national election studies". Public Opinion Quarterly. 52 (3): 1–28. doi:10.1086/269108.
- ^Kane, E.W.; MacAulay, L.J. (1993). "Interviewer gender and gender attitudes". Public Opinion Quarterly. 57 (1): 1–28. doi:10.1086/269352.
- ^Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B. (2011). "Interviewer BMI effects on under- and over-reporting of restrained eating. Evidence from a national Dutch face-to-face survey and a postal follow-up". International Journal of Public Health. 57 (3): 643–647. doi:10.1007/s00038-011-0323-z. PMC 3359459. PMID 22116390.
- Abramson, J.J. and Abramson, Z.H. (1999). Survey Methods in Community Medicine: Epidemiological Research, Programme Evaluation, Clinical Trials (5th edition). London: Churchill Livingstone/Elsevier Health Sciences ISBN 0-443-06163-7
- Adèr, H. J., Mellenbergh, G. J., and Hand, D. J. (2008). Advising on research methods: A consultant's companion. Huizen, The Netherlands: Johannes van Kessel Publishing.
- Andres, Lesley (2012). "Designing and Doing Survey Research". London: Sage.
- Dillman, D.A. (1978) Mail and telephone surveys: The total design method. New York: Wiley. ISBN 0-471-21555-4
- Engel. U., Jann, B., Lynn, P., Scherpenzeel, A. and Sturgis, P. (2014). Improving Survey Methods: Lessons from Recent Research. New York: Routledge. ISBN 978-0-415-81762-2
- Groves, R.M. (1989). Survey Errors and Survey Costs Wiley. ISBN 0-471-61171-9
- Griffith, James. (2014) "Survey Research in Military Settings." in Routledge Handbook of Research Methods in Military Studies edited by Joseph Soeters, Patricia Shields and Sebastiaan Rietjens.pp. 179–193. New York: Routledge.
- Leung, Wai-Ching (2001) "Conducting a Survey", in Student BMJ, (British Medical Journal, Student Edition), May 2001
- Ornstein, M.D. (1998). "Survey Research." Current Sociology 46(4): iii-136.
- Prince, S. a, Adamo, K. B., Hamel, M., Hardt, J., Connor Gorber, S., & Tremblay, M. (2008). A comparison of direct versus self-report measures for assessing physical activity in adults: a systematic review. International Journal of Behavioral Nutrition and Physical Activity, 5(1), 56. http://doi.org/10.1186/1479-5868-5-56
- Shaughnessy, J. J., Zechmeister, E. B., & Zechmeister, J. S. (2006). Research Methods in Psychology (Seventh Edition ed.). McGraw–Hill Higher Education. ISBN 0-07-111655-9 (pp. 143–192)
- Soeters, Joseph; Shields, Patricia and Rietjens, Sebastiaan.(2014). Routledge Handbook of Research Methods in Military Studies New York: Routledge.
- Surveys at Curlie (based on DMOZ)
Research Methods » Questionnaire
by Saul McLeod, updated 2018
A questionnaire is a research instrument consisting of a series of questions for the purpose of gathering information from respondents. Questionnaires can be thought of as a kind of written interview. They can be carried out face to face, by telephone, computer or post.
Questionnaires provide a relatively cheap, quick and efficient way of obtaining large amounts of information from a large sample of people. Data can be collected relatively quickly because the researcher would not need to be present when the questionnaires were completed. This is useful for large populations when interviews would be impractical.
However, a problem with questionnaire is that respondents may lie due to social desirability. Most people want to present a positive image of themselves and so may lie or bend the truth to look good, e.g., pupils would exaggerate revision duration.
Questionnaires can be an effective means of measuring the behavior, attitudes, preferences, opinions and, intentions of relatively large numbers of subjects more cheaply and quickly than other methods. An important distinction is between open-ended and closed questions.
Often a questionnaire uses both open and closed questions to collect data. This is beneficial as it means both quantitative and qualitative data can be obtained.
Closed questions structure the answer by only allowing responses which fit into pre-decided categories.
Data that can be placed into a category is called nominal data. The category can be restricted to as few as two options, i.e., dichotomous (e.g., 'yes' or 'no,' 'male' or 'female'), or include quite complex lists of alternatives from which the respondent can choose (e.g., polytomous).
Closed questions can also provide ordinal data (which can be ranked). This often involves using a continuous rating scale to measure the strength of attitudes or emotions. For example, strongly agree / agree / neutral / disagree / strongly disagree / unable to answer.
Closed questions have been used to research type A personality (e.g., Friedman & Rosenman, 1974), and also to assess life events which may cause stress (Holmes & Rahe, 1967), and attachment (Fraley, Waller, & Brennan, 2000).
They can be economical. This means they can provide large amounts of research data for relatively low costs. Therefore, a large sample size can be obtained which should be representative of the population, which a researcher can then generalize from.
- The respondent provides information which can be easily converted into quantitative data (e.g., count the number of 'yes' or 'no' answers), allowing statistical analysis of the responses.
The questions are standardized. All respondents are asked exactly the same questions in the same order. This means a questionnaire can be replicated easily to check for reliability. Therefore, a second researcher can use the questionnaire to check that the results are consistent.
They lack detail. Because the responses are fixed, there is less scope for respondents to supply answers which reflect their true feelings on a topic.
Open questions allow people to express what they think in their own words. Open-ended questions enable the respondent to answer in as much detail as they like in their own words. For example: “can you tell me how happy you feel right now?”
If you want to gather more in-depth answers from your respondents, then open questions will work better. These give no pre-set answer options and instead allow the respondents to put down exactly what they like in their own words.
Open questions are often used for complex questions that cannot be answered in a few simple categories but require more detail and discussion.
Lawrence Kohlberg presented his participants with moral dilemmas. One of the most famous concerns a character called Heinz who is faced with the choice between watching his wife die of cancer or stealing the only drug that could help her. Participants were asked whether Heinz should steal the drug or not and, more importantly, for their reasons why upholding or breaking the law is right.
Rich qualitative data is obtained as open questions allow the respondent to elaborate on their answer. This means the research can find out why a person holds a certain attitude.
Time-consuming to collect the data. It takes longer for the respondent to complete open questions. This is a problem as a smaller sample size may be obtained.
Time-consuming to analyze the data. It takes longer for the researcher to analyze qualitative data as they have to read the answers and try to put them into categories by coding, which is often subjective and difficult. However, Smith (1992) has devoted an entire book to the issues of thematic content analysis the includes 14 different scoring systems for open-ended questions.
Not suitable for less educated respondents as open questions require superior writing skills and a better ability to express one's feelings verbally.
Designing a Questionnaire
With some questionnaires suffering from a response rate as low as 5%, it is essential that a questionnaire is well designed.
There are a number of important factors in questionnaire design.
Make sure that all questions asked address the aims of the research. However, use only one feature of the construct you are investigating in per item.
The longer the questionnaire, the less likely people will complete it. Questions should be short, clear, and be to the point; any unnecessary questions/items should be omitted.
Run a small scale practice study to ensure people understand the questions. People will also be able to give detailed honest feedback on the questionnaire design.
The researcher should ensure that the answer to a question is not influenced by previous questions.
The language of a questionnaire should be appropriate to the vocabulary of the group of people being studied. Use statements which are interpreted in the same way by members of different subpopulations of the population of interest.
For example, the researcher must change the language of questions to match the social background of respondents' age / educational level / social class / ethnicity etc.
Make sure it looks professional, include clear and concise instructions. If sent through the post make sure the envelope does not signify ‘junk mail.’
The researcher must ensure that the information provided by the respondent is kept confidential, e.g., name, address, etc.
This means questionnaires are good for researching sensitive topics as respondents will be more honest when they cannot be identified. Keeping the questionnaire confidential should also reduce the likelihood of any psychological harm, such as embarrassment.
Participants must provide informed consent prior to completing the questionnaire, and must be aware that they have the right to withdraw their information at any time during the survey/ study.
Problems with Postal Questionnaires
The data might not be valid (i.e., truthful) as we can never be sure that the right person actually completed the postal questionnaire.
Also, postal questionnaires may not be representative of the population they are studying?
- This is because some questionnaires may be lost in the post reducing the sample size.
- The questionnaire may be completed by someone who is not a member of the research population.
- Those with strong views on the questionnaire’s subject are more likely to complete it than those with no interest in it.
Benefits of a Pilot Study
A pilot study is a practice / small-scale study conducted before the main study. It allows the researcher to try out the study with a few participants so that adjustments can be made before the main study, so saving time and money.
It is important to conduct a questionnaire pilot study for the following reasons:
Check that respondents understand the terminology used in the questionnaire.
Check that emotive questions have not been used as they make people defensive and could invalidate their answers.
Check that leading questions have not been used as they could bias the respondent's answer.
Ensure the questionnaire can be completed in an appropriate time frame (i.e., it's not too long).
Fraley, R. C., Waller, N. G., & Brennan, K. A. (2000). An item-response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78, 350-365.
Friedman, M., & Rosenman, R. H. (1974). Type A behavior and your heart. New York: Knopf.
Holmes, T. H., & Rahe, R. H. (1967). The social readjustment rating scale. Journal of psychosomatic research, 11(2), 213-218.
Smith, C. P. (Ed.). (1992). Motivation and personality: Handbook of thematic content analysis. Cambridge University Press.
How to reference this article:
McLeod, S. A. (2014). Questionnaire. Retrieved from www.simplypsychology.org/questionnaires.html