EH Home About Us Communities Radiation Medicine Water Sewage Programs Newsroom Data

This is a full transcript of the online presentation. For the presentation itself, go here.

Begin Transcript:

Transcript for the presentation: Survey Design Overview

Slide 1 

Today I will be talking about survey design. I will be presenting a general overview of the topic as it relates to PACE EH process.

Slide 2 

The goal of using surveys in the PACE EH process is to evaluate the community’s perception of the intervention (i.e installation of street lights) and its effectiveness. In addition, we would also like to measure the community’s physical and mental health. The best way to do this would be through biomonitoring – the collection of medical information on each individual. However, this method is costly and invasive. So, instead we have respondents fill out surveys, through which we can evaluate the community’s physical and mental health. We hope to see a change in the community’s physical and mental health associated with the intervention.

Slide 3

I will speak on four different sections of survey design: types of surveys, along with their benefits and limitations; sample frames; question design and word usage; and survey layout and design.

Slide 4 

There are four general survey types: personal interviews, telephone surveys, mail surveys, and I have combined email and internet surveys into one category.

Slide 5 

Personal interviews are when an interviewer speaks face to face with a respondent. The interviewer asks the respondent specific questions from the questionnaire according to set guidelines.  Many times this interview occurs in the respondent’s home.

The benefits of this interview time include the ability to obtain in-depth and comprehensive information. For example, if a respondent indicates that the air quality in the home is not good and smells funny the interviewer may be able to ask the respondent to elaborate on “smells funny.” Potentially asking the respondent to provide an example of the smell or asking the respondent to name a similar smell.

The interviewer is able to use props, pictures, or cards for interactive questions. The use of these items can help to keep the respondents interest in the survey process. In addition, the use of these items may also assist the respondent in understanding the question and/or jogging their memory.

The interviewer is also able to utilize the respondent’s non-verbal cues. He or she may be able to provide the needed clarification to a question without being asked. This is especially helpful if the respondent appears to be confused about a question but may feel unsure about asking for clarification.

A limitation to personal interviews is that they are quite costly. In addition, to the usual costs such as design and data entry, money needs to be available for travel and salaries for trained interviewers.

This survey type is also time intensive. It requires that interviewers spend time interviewing  each individual plus time spent traveling to and from each interview.

A survey type which uses an interviewer is subject to interviewer bias. No matter how specific the questions and guidelines are each interviewer conducts personal interviews differently. Interviewers provide verbal and, more importantly, non-verbal cues to the respondent. How an interviewer appears when asking the questions and receiving the answers may influence the responses the respondent provides. In addition, how the interviewer probes for further information or provides clarification to questions may also influence the respondents answer. Trained interviewers can reduce, but not eliminate, the amount of interviewer bias.

Finally, there is a lack of anonymity. Respondents may be worried about confidentiality and therefore may be less than forthcoming with their answers, especially when dealing with sensitive issues such as income, or socially undesirable behaviors and ideas. In addition, the respondent may also wish to appear to the interviewer in a favorable light. He or she may over report socially acceptable behaviors or thoughts and underreport socially unacceptable behaviors or thoughts. For instance, in a PACE-EH community the respondents may over report their satisfaction with the community improvements just because they admire the PACE-EH Project Coordinator who is conducting the survey.

Slide 6 

Telephone interviews are when the interviewer asks respondents questions over the phone from a set questionnaire. Respondents provide answers according to the instructions given by the interviewer.

One of the benefits of talking directly with the respondent is the ability to probe for detailed information. It also allows the respondent to ask for clarification when they do not understand a question or the instructions provided by the interviewer.

For telephone interviews, the turn around time from interview to collection is quick. Especially, if the responses are entered into a computer system during the interview process.  

Again, with all survey types that involve an interviewer we are faced with the limitation of interviewer bias. Remember the bias can be reduced, but not eliminated, with the use of trained interviewers.

Moving from personal interviews to telephone interviews we lose the ability to utilize props, pictures or cards to assist the respondent in jogging their memory or understanding a question.

Telephone interviews are a costly endeavor. While they do not include the cost of travel time this type of survey still includes one or more interviewer’s salary.

Finally, while telephone interviews provide more anonymity than personal interviews (i.e. the interviewer can not see the respondents house, identify them on the street, etc.) the method is still not completely confidential. The interviewer was able to get in touch with the respondent and the respondents responses can be linked to a particular phone number. Because there is not the promise of absolute confidentiality the respondent may still not want to answer sensitive questions or provide completely accurate information.

Slide 7 

In mail surveys, respondents fill out the surveys on their own by reading the questions and following the provided instructions. Typically the survey is mailed to the individual; however, there are other methods of delivery, such as handing the survey to the respondent. The important idea with this survey type is that the respondent fills out the survey on their own.

Let’s go over some of the benefits.

Mail surveys provide the anonymity that the previous survey types did not. Respondents may be more willing to complete the survey and provide accurate responses to all the questions since they perceive complete confidentiality.

Since, mail surveys are presented to respondents on sheets of paper it is very easy to include pictures or images to help the respondent understand the questions or jog their memory.

Another benefit of this survey type is the lack of an interviewer. Since there is no interviewer – there is no interviewer bias. In addition, the costs of conducting the survey are greatly reduced since we no longer have to pay an interviewer’s salary.

One of the limitations of mail surveys is a lengthy turn around time. Researchers must mail out surveys to respondents and then wait for the respondents to mail the surveys back. This can take many weeks and does not include the time required for follow-up for those respondents who do not return the survey.

There is also the inability to probe for detailed information or to provide clarification when needed. The only data that you will receive is from the questions written on the survey – nothing else. So it is imperative that the researcher thinks carefully about the goals of the survey, the data required to meet these goals, and if the questions asked will gather this data.

Slide 8 

Finally email and internet surveys. Respondents are either emailed the survey or provided with the location of the survey online. They provide responses to the questions according to the given instructions.

With email and internet surveys the researcher is able to customize the survey to each respondent. The respondent is only allowed to answer the questions that apply to them. For example, a respondent is asked if they have school age children. If the answer is yes, the respondent would then be asked about the method of transportation their child uses to get to and from school. The respondent would also be asked about the safety of their child traveling to and from school. If the respondent has said no – they do not have school age children – then they would not have been presented with any questions pertaining to school age children.

Email and internet surveys have the fastest distribution method of all the survey types. As soon as the respondent is contacted (whether through email, a website, or a flyer with a web address) they can be directed to a website to fill out the survey.

One of the greatest benefits of this survey type is automatic data entry. Since, the interview method is electronic the responses are entered electronically directly into a database.

There are two major limitations to email and internet surveys.

These surveys are computer programming intensive. The branching features which allow for individual survey customization require a knowledgeable programmer or an expensive computer program to create the survey so that it runs smoothly and works properly.

In addition, they may not represent your target population. Not all individuals have access (or easy access) to the internet or the respondent may not be computer savvy. So, when choosing to use email or internet surveys – or any survey type – the researcher must be aware of the characteristics population they wish to interview. These characteristics may lead the researcher to choose one survey method over another or simply print their survey in Spanish or provide a Korean speaking interviewer.

The rest of this presentation leans toward creating paper surveys.

Slide 9 

Sampling frames.

Let’s say this box with all the little yellow diamonds represents a population, how about all the residents of Flagler county. All the residents of Flagler county would be our general population. Now let’s say that we are curious about parents’ perception of childhood lead poisoning. Our target population or the people we want to survey would be all the parents in Flagler county. Of course we will not be able to survey all of the parents in this county – we don’t have the funds or the man power. So, instead we take a representative sample of the population, say 2,000 parents.

Slide 10           

There are two main issues that we have to be aware of when creating the sampling frame.

One: everyone in the target population must have a know chance to participate. So, if there are one hundred thousand parents in Flagler county then each parent has a one in one hundred thousand chance of being selected to participate in our survey.

Two: no one outside the target population should be included in the survey. If our target population is parents in Flagler county we do not want individuals who are not parents participating or parents who reside outside Flagler county.

Problem two is fairly easy to overcome, it can be as simple as asking respondents if they are parent and if they reside in Flagler County. If the answer is no to either question then survey is not administered.

Slide 11           

Addressing problem one is more difficult. In an ideal world we our sampling frame would consist of a list of all the parents in Flagler County. Our sample of 2,000 parents would be randomly selected from this list, giving every parent in Flagler county a known chance to participate.

However, it is unlikely that a list of all the parents in Flagler county exists. It is more likely that the sampling frame is created from a list of parents whose children are enrolled in public school. We would then randomly select 2,000 parents from this list. In this sampling frame each parent in Flagler County would not have a known chance to participate. Instead only those who have children in public school would have a know chance to participate.

Slide 12           

When creating the sample frame one of the first things a researcher needs to do is define the population. What group of people do you want to gather information from? It is imperative that the sample frame represent the population, (i.e the same proportion of males and females, black and whites, etc). If the population in the sampling frame is not representative of the target population then the results of the survey and the conclusions made therein can not be generalized to the target population. Meaning that if we interviewed only parents in Flagler County with children in public schools, we could not say that our results and conclusions are the opinions of all the parents in Flagler County. Because parents of home school children or parents of toddlers may have a different opinion. Second, it is important to define the sample area. This can be census blockgroups, school district boundaries, counties, etc. A defined sample area will assist the researcher in identifying the physical location of their target population. It may also provide demographic information on the target population which will help in creating a representative sample, in other words creating the sampling frame. The researcher will also need to figure out how to contact the respondents and if the method of contacting the respondent will obtain a representative sample of the population. For instance, if using phone numbers obtained from a telephone listing service it is important to find out if cell phone numbers are included in the list. If cell phone numbers are not included in the list, the sample will not represent the population because a number of individuals, typically the younger generation, are now using cell phones exclusively instead of traditional land lines. Another example is contacting the respondent during business hours (approximately 8 to 5.) It has been shown that women are more likely than men to be at home during this time period. The survey would then have a disproportionate number of females and not represent the true proportion of females in the population.

Here is a real life example of a sampling frame. In Indian River County a survey was conducted in a PACE EH community. The general population was Indian River County. The target population was all the adults in West Wabasso, the PACE EH community. We wanted to know how the adults in the community perceived the changes made over the past two years.

Slide 13           

The west Wabasso community was defined by census blocks as highlighted in the blue-gray area. In this case the target population and the sampling frame were the same. We attempted to survey all the adults in the community. And managed to capture approximately 93 percent. The proportion of males and females responding to our survey was approximately the same as the target population.

Slide 14           

Alright, let’s move on to survey questions. In general, we want each question to be concise and to the point. However, it may not always be possible to keep each question short, especially for written or internet surveys where the respondent does not have a chance to ask for clarification. Sometimes a researcher needs to provide that clarification in the question. For instance: “About how long has it been since your last visited a doctor for a routine checkup? A routine checkup is a general physical exam, not an exam for a specific injury, illness, or condition.” In this question the researcher has defined the word “checkup” to make sure each respondent interprets the word the same way.

When writing survey questions or choosing questions for your survey, first figure out what the objectives or purpose of the survey is. Then assign one of these three categories to each question– must know, useful to know, and nice to know. The data obtained from questions in the must know category are essential to meeting the objective of the survey. The data obtained from questions in the useful to know category are helpful in meeting the survey objectives and provide a more accurate picture. The data obtained from the questions in the third category is exactly what is sounds like – nice to know but not necessary to meet the survey objectives. After each question is categorized, immediately throw out any question categorized nice to know. Questions categorized useful to know can be included only if the survey is not too long or complicated.

Finally, when writing questions figure out how the data obtained from the question will be analyzed. If you can not figure out a way to analyze the data then go back to the drawing board and either rephrase the question or start from the begin and reassess what type of information you would like to get out of the question.

Slide 15           

Here is an example from the west Wabasso survey. We asked respondents to agree or disagree with the following statement: “My children miss fewer school days due to illness than 2 years ago.” We also allowed respondents to mark not applicable. As you can see in the pie chart 36 percent of respondents marked not applicable. It turns out that older respondents were more likely to answer not applicable they younger respondents. Probably because they did not have children in school. If we had thought about why an individual might have marked not applicable ahead time we would have added a filter question asking respondents if they had school age children. Only those who answered yes would be instructed to answer the question about children missing school.

Slide 16           

There are three general types of survey questions. Behavioral or factual questions ask a respondent various facts about themselves or how they behave. For example: “Have you used public bus transportation within the past 30 days?” or “Are you limited in any way in any activities because of physical, mental, or emotional problems?”

Knowledge questions deals with a respondent’s knowledge of a particular topic or their cognitive skills. An example of this question is “Do you know firsthand of a hazardous waste site located in your community?”

Finally, there questions that deal with the respondent’s psychological state or their attitude about a particular subject. For instance: “Thinking about your community as a whole, how safe do you feel your community is from crime?” or “Do you believe there is a problem in your community with roaming dogs?”

Slide 17           

There are two different types of responses to survey questions – open ended question and closed ended questions. Let’s discuss closed ended questions first. Closed ended questions have pre-determined categories that respondents can choose from. The first example provides multiple choices to the question “What is the main source of drinking water used in your home?” When using multiple choice responses the researcher has to make sure they include all possible responses and that each category is mutually exclusive. Here there are three choices however, if this questions had only two responses; a city, county or town water system, or a small water system operated by a home association, then individuals who obtained their water from a private well would feel frustrated and may not continue filling out the survey, they may not answer the question or they may choose one of the two answers even though neither one describes their situation.

The second example has interval responses. It is important to make sure when using interval responses that the intervals do not over lap. An overlapping interval would be 5 to 10 years with last category being 10 years or more.

Closed ended questions also include agreement or response scales. In the third example we have a response scale. The respondent can circle a number between very bad and very good.

Slide 18           

The previous questions could also have been done with labels instead of numbers; very good, good, neither good nor bad, bad, and very bad.

When using response or agreement scales the researcher wants to obtain results that have variability within the response categories. If all the responses were in one category there would be nothing to analyze. For instance if our question only had three categories - very good, neither good nor bad, and very bad - the majority of people would answer neither good nor bad. In this case people are forced to choose an extreme value or an average value. If we eliminated the extreme categories by changing very good to good and very bad to bad we would be more likely to obtain variability in the results. 

Slide 19           

Open ended questions. These are questions where the respondent writes in their answer. The first example requires the respondent to write in a particular number. In this case how many days per week they use the community’s public recreation areas. Using this response format the researcher is able to create response categories in the analysis phase dependent on the type of results they receive.

Our second example allows the respondent to provide an answer in their own words. Allowing the respondent to answer in their own words can potential provide the researcher with a lot of valuable information they would not otherwise receive. However, this type of open ended question is difficult to analyze. The researcher has to read each response and then determine how the results will be summarized and used in the analysis.

Open ended questions are especially useful if the researcher is uncertain how individuals will respond. As an illustration, let’s say we had a community that we did not know very well and we wanted to know what environmental issue was the most important to the residents. On our survey we asked the closed ended question: “Which environmental issue do you think needs the most attention. We provided four choices; water, septic, air quality, and other. However, in our community the major problem was animal control issues and fears about radon in their homes. Therefore the majority of respondents from this community would mark other or choose one of the three listed choices even though this was not the environmental issue that was the most important to them. The data from our survey would provide unusable results or incorrectly access the concerns of the community. In this case, it would have been better if we had utilized an open ended question.

Slide 20           

Certain adjectives and adverbs have been shown through research to be interpreted differently dependent on the person. When writing questions you want to avoid using these words. Instead use adjectives that have been shown to have consistent meanings for different individuals.

Slide 21           

When writing questions be aware of the word “or” and the word “and.” This use of these words could potentially lead to a single question with two different topics. In other-words a double barreled questions. For example: “Do you feel the air in your home and work environment is adversely affecting your health?” If an individual answers yes then are saying the air in their home is affecting their health, the air at work is affecting their health, or the air at home and at work is affecting their health. Because we are unsure of what the respondent meant, we are shooting in the dark when trying to target intervention methods to improve air quality. It would have been more effective if we had split the question into two different questions.

Occasionally it is ok to use the word “or” and the word “and” but the researcher must examine all aspects of the question and be sure that the data received supports the survey objectives.

When writing survey questions we want to avoid creating double negatives and potential question confusion. For example, let’s say we asked a respondent to answer yes or no to the following statement “I do not think there is noise pollution in my community” This translates to “No, I do not think there is noise pollution in my community” meaning the person believes there is noise pollution in their community or “Yes, I do not think there is noise pollution in my community” meaning the person does not think there is noise pollution in their community. It would be easier to have asked the question without the word ‘not.’

Slide 22           

Unless you are giving a survey to a homogenous population avoid using slang, jargon, or acronyms. A homogenous population may be a group of doctors. For this population feel free to use technical medical terms in your questions. Or if surveying health department employees it probably ok to use CHD instead of County Health department.

When wording questions use specifically defined time periods. For example; “How many times in the last week did you use your community’s public recreational areas?” Do you, as the researcher, mean: a work week, which is 5 days; the past 7 days starting with yesterday; or the past 7 days starting with the previous Sunday. It would have been better if the question had asked “How many times during the past 7 days did you use your community’s public recreational areas?”

Make sure you as the researcher clearly understand the meaning of all the words you use. When using the word government, is it important to the survey objectives for the respondent to provide information related only to federal government or all government in general?

Slide 23           

When designing the layout of the survey, place general questions before more specific related questions. i.e. Ask the questions related to the respondents opinions about their local police before asking questions about the level of crime in the community.

Instructions on how and which questions to answer in a written survey should be in a different font and/or format then the font and format used for the question. This will help the respondent to differentiate the survey instructions from the questions and be more likely to follow the instructions.

Number all the survey questions. This will alert the respondent to having skipped a question (i.e. answering question 10 after question 7.) Question numbering also makes it easier for the data entry person to correctly enter the responses into the database.

Slide 24           

Do not try to condense your survey to fit on one page. Putting a lot of information onto a single sheet makes it appear harder and more complicated to complete, especially if the researcher reduced the font size to make all the information fit. It is better to create your survey to allow for enough room to write in responses to open ended questions (always remembering that hand written responses require more room than computer type) and use a readable font, which would be in general a 12 point font or a 10 point sans serif font. If you want the survey to fit on one sheet of paper – cut questions. This is where your question categories ‘must know’ and ‘need to know’ come in handy.

Slide 25           

For closed ended questions a vertical layout is easier to read than a horizontal layout. However, to save space it is sometimes useful to use a horizontal layout. If this is the case make sure that question choices are clearly labeled and there is substantial space between the choices. Notice in this box that this is difficult to determine where the mark for a good response should be. On the line before good or the line after good. Instead, a horizontal layout should look like this.

Slide 26           

I hope you have picked up from the previous two slides that the appearance of your survey is very important. Respondents are more likely to fill out an attractive, well laid-out survey, which is easy to read and understand than one that is not.

Slide 27           

It is very important to introduce yourself and your organization to the respondent. Then explain to them why you are conducting a survey and how the results will be beneficial to them or their community. This way the respondent knows who you are and has a reason to fill out the survey.

Slide 28           

Finally, make sure to thank the respondent for their participation.

Slide 29           

Ok let’s talk about response bias.

There are four main categories of response bias; memory, motivation, communication and knowledge.

Knowledge – individuals may not know the answer to the question they are presented with but may hazard a guess anyways.

Communication – The respondent may also not understand the question or may interpret the question differently than the researcher intended.

Motivation - Respondents may not be motivated to tell the truth. This may be because of potential social repercussions or wanting the researcher to view them in a better light.

Finally, as we all know people forget things whether it be the time an event occurred, how an event transpired, how they felt last week, where they placed their keys, or what they had for breakfast. A respondent’s memory greatly affects the outcome of the survey. Two important types of memory bias are, forward telescoping and backward telescoping. Forward telescoping is the inclusion of events from a previous period. Forward telescoping leads to an over-reporting of events or activities. Backward telescoping, pushing the event back into a previous time period leads to the under-reporting of events or activities. The important thing to understand about forward and backward telescoping is that it refers to over-reporting and under-reporting of events. For example, a individual may indicated that they had trouble sleeping 15 nights during the past 30 days when in actuality they had trouble sleeping only 10 nights during the past 30 days. This would be the over-reporting of an event or forward telescoping. Research has shown that forward telescoping occurs more often than backward telescoping.

In addition, people may have trouble reconstructing past events. Basically, what this means is that people examine the present, their feelings, thoughts, activates, and project backwards to fill in pieces of incomplete memory. People also remember that there has been change over time and tend to exaggerate the amount of change. For this reason when evaluating PACE EH projects, in which interventions can occur over many years, we recommend using a pre and a post survey. The results of the pre intervention survey will be compared to the results of the post intervention survey to determine if the community views the intervention as effective.

Slide 30           

Now let’s continue looking at our real world example. In the community of West Wabasso located in Indian River County. A survey was conducted after the interventions. The respondents were asked seven community improvement questions in which they were instructed to answer if the situation now compared to two years ago had improved, not changed, or worsened. The respondents were also provided with the fourth choice, unsure

Slide 31           

There were also five quality of life question used in the analysis. Respondents were asked to agree, disagree or they could mark not applicable.

Slide 32           

In general, respondents felt their quality of life had improved over the past two years due to the changes in their community. Four community change issues showed a statistically significant association with improved quality of life.

Removal and/or demolishing of abandoned houses and cars

Provision of high-quality police service

The installation of street lights

The creation of safe places to walk/exercise outdoors

Now there are several limitations to the survey and the methods used to conduct it. However, as you can by the results the information obtained through the use of a survey was invaluable and provided data which can be used to back up the anecdotal evidence provided by the community.

Slide 33           

Survey methodology is a very large field of statistics with years of research behind it. In this presentation I have only briefly scratched the surface. Therefore I have included a few resources that I found useful. In addition, there are plenty of online sites and other books to assist you.

Slide 34           

And of course I am always available for questions.

Thank you very much for your time.

End Transcript