Thread: Good Survey/Bad Survey

What constitutes a good survey? How many variables and choices would it need to have? Or, is a survey not a good form of finding ones opinion? For example, if I made a survey asking the public, "Who is the best actor?" Then, showed pictures to choose from, how am I supposed to know that they weren't picking the most attractive one? What would you consider a good survey and what would you consider a bad survey? I just need some opinions on this matter, thanks in advance.

A survey is only as good as the clarity of your terms, the directness of your questions, and the relevance of your answer choices.

Your question "best" fails right there... Best, how? We all define it differently, and in different contexts. I might consider "best" the prettiest, whereas you might consider "best" the most highly paid.

You need to spend considerable effort asking smart questions. Instead of "who is the best actor," you would need to ask something like, "which of the actors below has consistently demonstrated their skill in character portrayal?" It's still opinion, but you're at least asking their opinion about a very specific thing.

Also, your answer choices are huge, and you need to list options which are meaningful. If you list a bunch of actors and ask people to choose their favorite, you're invariably going to get folks who don't see their favorite actor, or who don't see any differences in your list.

One way around that is to ask one question per actor and then have them rank them on some sort of Likert scale (like 1 to 7, where 1 is poor and 7 is excellent). You then ask the exact same question about some other actor and compare the average and standard deviation. This allows you to make a more meaningful comparison across different actors (the ones you're interested in), and also provides you with numbers (you can see how large that difference is... you can quantize it... and it's much less qualitative).

Example - "On a scale of 1 to 7, where 1 is "poor" and 7 is "excellent," how would you rate Brad Pitt's skill in character portrayal?" Then, you do it again, but this time with a different actor. "On a scale of 1 to 7, where 1 is "poor" and 7 is "excellent," how would you rate George Clooney's skill in character portrayal?" You can also ask specific questions in this way... "On a scale of 1 to 7, where 1 is "extremely ugly" and 7 is "extremely attractive," how would you rate the physical appearance of Scarlett Johanson ?"?" Then, you do it again, like this for comparison: "On a scale of 1 to 7, where 1 is "extremely ugly" and 7 is "extremely attractive," how would you rate the physical appearance of Paz Vega?"


Overall though... surveys can be VERY powerful tools if you have designed them well, and if you gather enough responses from a large enough population. The more people you get responses from, the more confident you can be that the averages you are sharing are stable (given that you've taken efforts to get a representative sample... a sample of respondents that is a close approximation of society as a whole).


I hope that helps. I've got some experience in this area, and the above are just some quick thoughts off the top of my head.

Thanks, a lot of good information for developing an accurate survey.

inow, there are doubtless a plethora of books on this topic. Could you recommend any in particular?

Unfortunately, no. I've never read any books on the subject. I learned everything I know from experience (collecting data of my own for research in school, working with others in their research and dissertations, and starting my career in this type of research for a national organization). Sorry.

Originally Posted by inow

Unfortunately, no. I've never read any books on the subject. I learned everything I know from experience ...

isn't that always the way?

I know books* have been written about survey techniques and designs,
but I've never read one -
what little I've learned has been ad hoc in study or work ...

*and/or chapters within larger texts -
these are often tailored to the broader subject (market analysis, psychology, government administration, etc)

there are booklet style or powerpoint guidelines or tutorials available online -
some of which might be suitably adapted for a given purpose ...

eg: Survey Questionnaire Design -
a pdf booklet from Fairfax County Dept of Systems Management for Human Services

Anastasi, A. & Urbina, S. (1997). Psychological testing. [7th edition] Upper Saddle River, NJ: Prentice-Hall.
Cronbach, L. J. (1990). Essentials of psychological testing. [5th edition] New York, NY: Harper Collins Publishers.
Gregory, R. J. (2004). Psychological testing: History, principles, and applications. [4th edition] Boston, MA: Allyn & Bacon.
Hogan, T. P. (2003). Psychological testing: A practical introduction. New York: John Wiley & Sons.

These are fairly good books on testing theory within psychology. They should include between them, test construction, various kinds of validity, reliability and how they can be measured and improved upon.
I suggest browsing them before purchase so you are sure that this is the sort of information you are looking for.

Holy crap you're alive!