|
Transcript for Survey Design Overview Presentation
This is a full transcript of the online presentation. For the
presentation itself, go here.
Begin Transcript:
Transcript for the presentation: Survey Design
Overview
Slide 1
Today I will be talking about survey design. I will be
presenting a general overview of the topic as it relates to PACE EH process.
Slide 2
The goal of using surveys in the PACE EH process is to
evaluate the communitys perception of the intervention (i.e installation of
street lights) and its effectiveness. In addition, we would also like to measure
the communitys physical and mental health. The best way to do this would be
through biomonitoring the collection of medical information on each
individual. However, this method is costly and invasive. So, instead we have
respondents fill out surveys, through which we can evaluate the communitys
physical and mental health. We hope to see a change in the communitys physical
and mental health associated with the intervention.
Slide 3
I will speak on four different sections of survey design:
types of surveys, along with their benefits and limitations; sample frames;
question design and word usage; and survey layout and design.
Slide 4
There are four general survey types: personal interviews,
telephone surveys, mail surveys, and I have combined email and internet surveys
into one category.
Slide 5
Personal interviews are when an interviewer speaks face to
face with a respondent. The interviewer asks the respondent specific questions
from the questionnaire according to set guidelines. Many times this interview
occurs in the respondents home.
The benefits of this interview time include the ability to
obtain in-depth and comprehensive information. For example, if a respondent
indicates that the air quality in the home is not good and smells funny the
interviewer may be able to ask the respondent to elaborate on smells funny.
Potentially asking the respondent to provide an example of the smell or asking
the respondent to name a similar smell.
The interviewer is able to use props, pictures, or cards
for interactive questions. The use of these items can help to keep the
respondents interest in the survey process. In addition, the use of these items
may also assist the respondent in understanding the question and/or jogging
their memory.
The interviewer is also able to utilize the respondents
non-verbal cues. He or she may be able to provide the needed clarification to a
question without being asked. This is especially helpful if the respondent
appears to be confused about a question but may feel unsure about asking for
clarification.
A limitation to personal interviews is that they are quite
costly. In addition, to the usual costs such as design and data entry, money
needs to be available for travel and salaries for trained interviewers.
This survey type is also time intensive. It requires that
interviewers spend time interviewing each individual plus time spent traveling
to and from each interview.
A survey type which uses an interviewer is subject to
interviewer bias. No matter how specific the questions and guidelines are each
interviewer conducts personal interviews differently. Interviewers provide
verbal and, more importantly, non-verbal cues to the respondent. How an
interviewer appears when asking the questions and receiving the answers may
influence the responses the respondent provides. In addition, how the
interviewer probes for further information or provides clarification to
questions may also influence the respondents answer. Trained interviewers can
reduce, but not eliminate, the amount of interviewer bias.
Finally, there is a lack of anonymity. Respondents may be
worried about confidentiality and therefore may be less than forthcoming with
their answers, especially when dealing with sensitive issues such as income, or
socially undesirable behaviors and ideas. In addition, the respondent may also
wish to appear to the interviewer in a favorable light. He or she may over
report socially acceptable behaviors or thoughts and underreport socially
unacceptable behaviors or thoughts. For instance, in a PACE-EH community the
respondents may over report their satisfaction with the community improvements
just because they admire the PACE-EH Project Coordinator who is conducting the
survey.
Slide 6
Telephone interviews are when the interviewer asks
respondents questions over the phone from a set questionnaire. Respondents
provide answers according to the instructions given by the interviewer.
One of the benefits of talking directly with the respondent
is the ability to probe for detailed information. It also allows the respondent
to ask for clarification when they do not understand a question or the
instructions provided by the interviewer.
For telephone interviews, the turn around time from
interview to collection is quick. Especially, if the responses are entered into
a computer system during the interview process.
Again, with all survey types that involve an interviewer we
are faced with the limitation of interviewer bias. Remember the bias can be
reduced, but not eliminated, with the use of trained interviewers.
Moving from personal interviews to telephone interviews we
lose the ability to utilize props, pictures or cards to assist the respondent in
jogging their memory or understanding a question.
Telephone interviews are a costly endeavor. While they do
not include the cost of travel time this type of survey still includes one or
more interviewers salary.
Finally, while telephone interviews provide more anonymity
than personal interviews (i.e. the interviewer can not see the respondents
house, identify them on the street, etc.) the method is still not completely
confidential. The interviewer was able to get in touch with the respondent and
the respondents responses can be linked to a particular phone number. Because
there is not the promise of absolute confidentiality the respondent may still
not want to answer sensitive questions or provide completely accurate
information.
Slide 7
In mail surveys, respondents fill out the surveys on their
own by reading the questions and following the provided instructions. Typically
the survey is mailed to the individual; however, there are other methods of
delivery, such as handing the survey to the respondent. The important idea with
this survey type is that the respondent fills out the survey on their own.
Lets go over some of the benefits.
Mail surveys provide the anonymity that the previous survey
types did not. Respondents may be more willing to complete the survey and
provide accurate responses to all the questions since they perceive complete
confidentiality.
Since, mail surveys are presented to respondents on sheets
of paper it is very easy to include pictures or images to help the respondent
understand the questions or jog their memory.
Another benefit of this survey type is the lack of an
interviewer. Since there is no interviewer there is no interviewer bias. In
addition, the costs of conducting the survey are greatly reduced since we no
longer have to pay an interviewers salary.
One of the limitations of mail surveys is a lengthy turn
around time. Researchers must mail out surveys to respondents and then wait for
the respondents to mail the surveys back. This can take many weeks and does not
include the time required for follow-up for those respondents who do not return
the survey.
There is also the inability to probe for detailed
information or to provide clarification when needed. The only data that you will
receive is from the questions written on the survey nothing else. So it is
imperative that the researcher thinks carefully about the goals of the survey,
the data required to meet these goals, and if the questions asked will gather
this data.
Slide 8
Finally email and internet surveys. Respondents are either
emailed the survey or provided with the location of the survey online. They
provide responses to the questions according to the given instructions.
With email and internet surveys the researcher is able to
customize the survey to each respondent. The respondent is only allowed to
answer the questions that apply to them. For example, a respondent is asked if
they have school age children. If the answer is yes, the respondent would then
be asked about the method of transportation their child uses to get to and from
school. The respondent would also be asked about the safety of their child
traveling to and from school. If the respondent has said no they do not have
school age children then they would not have been presented with any questions
pertaining to school age children.
Email and internet surveys have the fastest distribution
method of all the survey types. As soon as the respondent is contacted (whether
through email, a website, or a flyer with a web address) they can be directed to
a website to fill out the survey.
One of the greatest benefits of this survey type is
automatic data entry. Since, the interview method is electronic the responses
are entered electronically directly into a database.
There are two major limitations to email and internet
surveys.
These surveys are computer programming intensive. The
branching features which allow for individual survey customization require a
knowledgeable programmer or an expensive computer program to create the survey
so that it runs smoothly and works properly.
In addition, they may not represent your target population.
Not all individuals have access (or easy access) to the internet or the
respondent may not be computer savvy. So, when choosing to use email or internet
surveys or any survey type the researcher must be aware of the
characteristics population they wish to interview. These characteristics may
lead the researcher to choose one survey method over another or simply print
their survey in Spanish or provide a Korean speaking interviewer.
The rest of this presentation leans toward creating paper
surveys.
Slide 9
Sampling frames.
Lets say this box with all the little yellow diamonds
represents a population, how about all the residents of Flagler county. All the
residents of Flagler county would be our general population. Now lets say that
we are curious about parents perception of childhood lead poisoning. Our target
population or the people we want to survey would be all the parents in Flagler
county. Of course we will not be able to survey all of the parents in this
county we dont have the funds or the man power. So, instead we take a
representative sample of the population, say 2,000 parents.
Slide 10
There are two main issues that we have to be aware of when
creating the sampling frame.
One: everyone in the target population must have a know
chance to participate. So, if there are one hundred thousand parents in Flagler
county then each parent has a one in one hundred thousand chance of being
selected to participate in our survey.
Two: no one outside the target population should be
included in the survey. If our target population is parents in Flagler county we
do not want individuals who are not parents participating or parents who reside
outside Flagler county.
Problem two is fairly easy to overcome, it can be as simple
as asking respondents if they are parent and if they reside in Flagler County.
If the answer is no to either question then survey is not administered.
Slide 11
Addressing problem one is more difficult. In an ideal world
we our sampling frame would consist of a list of all the parents in Flagler
County. Our sample of 2,000 parents would be randomly selected from this list,
giving every parent in Flagler county a known chance to participate.
However, it is unlikely that a list of all the parents in
Flagler county exists. It is more likely that the sampling frame is created from
a list of parents whose children are enrolled in public school. We would then
randomly select 2,000 parents from this list. In this sampling frame each parent
in Flagler County would not have a known chance to participate. Instead only
those who have children in public school would have a know chance to
participate.
Slide 12
When creating the sample frame one of the first things a
researcher needs to do is define the population. What group of people do you
want to gather information from? It is imperative that the sample frame
represent the population, (i.e the same proportion of males and females, black
and whites, etc). If the population in the sampling frame is not representative
of the target population then the results of the survey and the conclusions made
therein can not be generalized to the target population. Meaning that if we
interviewed only parents in Flagler County with children in public schools, we
could not say that our results and conclusions are the opinions of all the
parents in Flagler County. Because parents of home school children or parents of
toddlers may have a different opinion. Second, it is important to define the
sample area. This can be census blockgroups, school district boundaries,
counties, etc. A defined sample area will assist the researcher in identifying
the physical location of their target population. It may also provide
demographic information on the target population which will help in creating a
representative sample, in other words creating the sampling frame. The
researcher will also need to figure out how to contact the respondents and if
the method of contacting the respondent will obtain a representative sample of
the population. For instance, if using phone numbers obtained from a telephone
listing service it is important to find out if cell phone numbers are included
in the list. If cell phone numbers are not included in the list, the sample will
not represent the population because a number of individuals, typically the
younger generation, are now using cell phones exclusively instead of traditional
land lines. Another example is contacting the respondent during business hours
(approximately 8 to 5.) It has been shown that women are more likely than men to
be at home during this time period. The survey would then have a
disproportionate number of females and not represent the true proportion of
females in the population.
Here is a real life example of a sampling frame. In Indian
River County a survey was conducted in a PACE EH community. The general
population was Indian River County. The target population was all the adults in
West Wabasso, the PACE EH community. We wanted to know how the adults in the
community perceived the changes made over the past two years.
Slide 13
The west Wabasso community was defined by census blocks as
highlighted in the blue-gray area. In this case the target population and the
sampling frame were the same. We attempted to survey all the adults in the
community. And managed to capture approximately 93 percent. The proportion of
males and females responding to our survey was approximately the same as the
target population.
Slide 14
Alright, lets move on to survey questions. In general, we
want each question to be concise and to the point. However, it may not always be
possible to keep each question short, especially for written or internet surveys
where the respondent does not have a chance to ask for clarification. Sometimes
a researcher needs to provide that clarification in the question. For instance:
About how long has it been since your last visited a doctor for a routine
checkup? A routine checkup is a general physical exam, not an exam for a
specific injury, illness, or condition. In this question the researcher has
defined the word checkup to make sure each respondent interprets the word the
same way.
When writing survey questions or choosing questions for
your survey, first figure out what the objectives or purpose of the survey is.
Then assign one of these three categories to each question must know, useful to
know, and nice to know. The data obtained from questions in the must know
category are essential to meeting the objective of the survey. The data obtained
from questions in the useful to know category are helpful in meeting the survey
objectives and provide a more accurate picture. The data obtained from the
questions in the third category is exactly what is sounds like nice to know
but not necessary to meet the survey objectives. After each question is
categorized, immediately throw out any question categorized nice to know.
Questions categorized useful to know can be included only if the survey is not
too long or complicated.
Finally, when writing questions figure out how the data
obtained from the question will be analyzed. If you can not figure out a way to
analyze the data then go back to the drawing board and either rephrase the
question or start from the begin and reassess what type of information you would
like to get out of the question.
Slide 15
Here is an example from the west Wabasso survey. We asked
respondents to agree or disagree with the following statement: My children miss
fewer school days due to illness than 2 years ago. We also allowed respondents
to mark not applicable. As you can see in the pie chart 36 percent of
respondents marked not applicable. It turns out that older respondents were more
likely to answer not applicable they younger respondents. Probably because they
did not have children in school. If we had thought about why an individual might
have marked not applicable ahead time we would have added a filter question
asking respondents if they had school age children. Only those who answered yes
would be instructed to answer the question about children missing school.
Slide 16
There are three general types of survey questions.
Behavioral or factual questions ask a respondent various facts about themselves
or how they behave. For example: Have you used public bus transportation within
the past 30 days? or Are you limited in any way in any activities because of
physical, mental, or emotional problems?
Knowledge questions deals with a respondents knowledge of
a particular topic or their cognitive skills. An example of this question is Do
you know firsthand of a hazardous waste site located in your community?
Finally, there questions that deal with the respondents
psychological state or their attitude about a particular subject. For instance:
Thinking about your community as a whole, how safe do you feel your community
is from crime? or Do you believe there is a problem in your community with
roaming dogs?
Slide 17
There are two different types of responses to survey
questions open ended question and closed ended questions. Lets discuss closed
ended questions first. Closed ended questions have pre-determined categories
that respondents can choose from. The first example provides multiple choices to
the question What is the main source of drinking water used in your home? When
using multiple choice responses the researcher has to make sure they include all
possible responses and that each category is mutually exclusive. Here there are
three choices however, if this questions had only two responses; a city, county
or town water system, or a small water system operated by a home association,
then individuals who obtained their water from a private well would feel
frustrated and may not continue filling out the survey, they may not answer the
question or they may choose one of the two answers even though neither one
describes their situation.
The second example has interval responses. It is important
to make sure when using interval responses that the intervals do not over lap.
An overlapping interval would be 5 to 10 years with last category being 10 years
or more.
Closed ended questions also include agreement or response
scales. In the third example we have a response scale. The respondent can circle
a number between very bad and very good.
Slide 18
The previous questions could also have been done with
labels instead of numbers; very good, good, neither good nor bad, bad, and very
bad.
When using response or agreement scales the researcher
wants to obtain results that have variability within the response categories. If
all the responses were in one category there would be nothing to analyze. For
instance if our question only had three categories - very good, neither good nor
bad, and very bad - the majority of people would answer neither good nor bad. In
this case people are forced to choose an extreme value or an average value. If
we eliminated the extreme categories by changing very good to good and very bad
to bad we would be more likely to obtain variability in the results.
Slide 19
Open ended questions. These are questions where the
respondent writes in their answer. The first example requires the respondent to
write in a particular number. In this case how many days per week they use the
communitys public recreation areas. Using this response format the researcher
is able to create response categories in the analysis phase dependent on the
type of results they receive.
Our second example allows the respondent to provide an
answer in their own words. Allowing the respondent to answer in their own words
can potential provide the researcher with a lot of valuable information they
would not otherwise receive. However, this type of open ended question is
difficult to analyze. The researcher has to read each response and then
determine how the results will be summarized and used in the analysis.
Open ended questions are especially useful if the
researcher is uncertain how individuals will respond. As an illustration, lets
say we had a community that we did not know very well and we wanted to know what
environmental issue was the most important to the residents. On our survey we
asked the closed ended question: Which environmental issue do you think needs
the most attention. We provided four choices; water, septic, air quality, and
other. However, in our community the major problem was animal control issues and
fears about radon in their homes. Therefore the majority of respondents from
this community would mark other or choose one of the three listed choices even
though this was not the environmental issue that was the most important to them.
The data from our survey would provide unusable results or incorrectly access
the concerns of the community. In this case, it would have been better if we had
utilized an open ended question.
Slide 20
Certain adjectives and adverbs have been shown through
research to be interpreted differently dependent on the person. When writing
questions you want to avoid using these words. Instead use adjectives that have
been shown to have consistent meanings for different individuals.
Slide 21
When writing questions be aware of the word or and the
word and. This use of these words could potentially lead to a single question
with two different topics. In other-words a double barreled questions. For
example: Do you feel the air in your home and work environment is adversely
affecting your health? If an individual answers yes then are saying the air in
their home is affecting their health, the air at work is affecting their health,
or the air at home and at work is affecting their health. Because we are unsure
of what the respondent meant, we are shooting in the dark when trying to target
intervention methods to improve air quality. It would have been more effective
if we had split the question into two different questions.
Occasionally it is ok to use the word or and the word
and but the researcher must examine all aspects of the question and be sure
that the data received supports the survey objectives.
When writing survey questions we want to avoid creating
double negatives and potential question confusion. For example, lets say we
asked a respondent to answer yes or no to the following statement I do not
think there is noise pollution in my community This translates to No, I do not
think there is noise pollution in my community meaning the person believes
there is noise pollution in their community or Yes, I do not think there is
noise pollution in my community meaning the person does not think there is
noise pollution in their community. It would be easier to have asked the
question without the word not.
Slide 22
Unless you are giving a survey to a homogenous population
avoid using slang, jargon, or acronyms. A homogenous population may be a group
of doctors. For this population feel free to use technical medical terms in your
questions. Or if surveying health department employees it probably ok to use CHD
instead of County Health department.
When wording questions use specifically defined time
periods. For example; How many times in the last week did you use your
communitys public recreational areas? Do you, as the researcher, mean: a work
week, which is 5 days; the past 7 days starting with yesterday; or the past 7
days starting with the previous Sunday. It would have been better if the
question had asked How many times during the past 7 days did you use your
communitys public recreational areas?
Make sure you as the researcher clearly understand the
meaning of all the words you use. When using the word government, is it
important to the survey objectives for the respondent to provide information
related only to federal government or all government in general?
Slide 23
When designing the layout of the survey, place general
questions before more specific related questions. i.e. Ask the questions related
to the respondents opinions about their local police before asking questions
about the level of crime in the community.
Instructions on how and which questions to answer in a
written survey should be in a different font and/or format then the font and
format used for the question. This will help the respondent to differentiate the
survey instructions from the questions and be more likely to follow the
instructions.
Number all the survey questions. This will alert the
respondent to having skipped a question (i.e. answering question 10 after
question 7.) Question numbering also makes it easier for the data entry person
to correctly enter the responses into the database.
Slide 24
Do not try to condense your survey to fit on one page.
Putting a lot of information onto a single sheet makes it appear harder and more
complicated to complete, especially if the researcher reduced the font size to
make all the information fit. It is better to create your survey to allow for
enough room to write in responses to open ended questions (always remembering
that hand written responses require more room than computer type) and use a
readable font, which would be in general a 12 point font or a 10 point sans
serif font. If you want the survey to fit on one sheet of paper cut questions.
This is where your question categories must know and need to know come in
handy.
Slide 25
For closed ended questions a vertical layout is easier to
read than a horizontal layout. However, to save space it is sometimes useful to
use a horizontal layout. If this is the case make sure that question choices are
clearly labeled and there is substantial space between the choices. Notice in
this box that this is difficult to determine where the mark for a good response
should be. On the line before good or the line after good. Instead, a horizontal
layout should look like this.
Slide 26
I hope you have picked up from the previous two slides that
the appearance of your survey is very important. Respondents are more likely to
fill out an attractive, well laid-out survey, which is easy to read and
understand than one that is not.
Slide 27
It is very important to introduce yourself and your
organization to the respondent. Then explain to them why you are conducting a
survey and how the results will be beneficial to them or their community. This
way the respondent knows who you are and has a reason to fill out the survey.
Slide 28
Finally, make sure to thank the respondent for their
participation.
Slide 29
Ok lets talk about response bias.
There are four main categories of response bias; memory,
motivation, communication and knowledge.
Knowledge individuals may not know the answer to the
question they are presented with but may hazard a guess anyways.
Communication The respondent may also not understand the
question or may interpret the question differently than the researcher intended.
Motivation - Respondents may not be motivated to tell the
truth. This may be because of potential social repercussions or wanting the
researcher to view them in a better light.
Finally, as we all know people forget things whether it be
the time an event occurred, how an event transpired, how they felt last week,
where they placed their keys, or what they had for breakfast. A respondents
memory greatly affects the outcome of the survey. Two important types of memory
bias are, forward telescoping and backward telescoping. Forward telescoping is
the inclusion of events from a previous period. Forward telescoping leads to an
over-reporting of events or activities. Backward telescoping, pushing the event
back into a previous time period leads to the under-reporting of events or
activities. The important thing to understand about forward and backward
telescoping is that it refers to over-reporting and under-reporting of events.
For example, a individual may indicated that they had trouble sleeping 15 nights
during the past 30 days when in actuality they had trouble sleeping only 10
nights during the past 30 days. This would be the over-reporting of an event or
forward telescoping. Research has shown that forward telescoping occurs more
often than backward telescoping.
In addition, people may have trouble reconstructing past
events. Basically, what this means is that people examine the present, their
feelings, thoughts, activates, and project backwards to fill in pieces of
incomplete memory. People also remember that there has been change over time and
tend to exaggerate the amount of change. For this reason when evaluating PACE EH
projects, in which interventions can occur over many years, we recommend using a
pre and a post survey. The results of the pre intervention survey will be
compared to the results of the post intervention survey to determine if the
community views the intervention as effective.
Slide 30
Now lets continue looking at our real world example. In
the community of West Wabasso located in Indian River County. A survey was
conducted after the interventions. The respondents were asked seven community
improvement questions in which they were instructed to answer if the situation
now compared to two years ago had improved, not changed, or worsened. The
respondents were also provided with the fourth choice, unsure
Slide 31
There were also five quality of life question used in the
analysis. Respondents were asked to agree, disagree or they could mark not
applicable.
Slide 32
In general, respondents felt their quality of life had
improved over the past two years due to the changes in their community. Four
community change issues showed a statistically significant association with
improved quality of life.
Removal and/or demolishing of abandoned houses and cars
Provision of high-quality police service
The installation of street lights
The creation of safe places to walk/exercise outdoors
Now there are several limitations to the survey and the
methods used to conduct it. However, as you can by the results the information
obtained through the use of a survey was invaluable and provided data which can
be used to back up the anecdotal evidence provided by the community.
Slide 33
Survey methodology is a very large field of statistics with
years of research behind it. In this presentation I have only briefly scratched
the surface. Therefore I have included a few resources that I found useful. In
addition, there are plenty of online sites and other books to assist you.
Slide 34
And of course I am always available for questions.
Thank you very much for your time.
End Transcript
| 
