Notices
Results 1 to 8 of 8

Thread: Good Survey/Bad Survey

  1. #1 Good Survey/Bad Survey 
    Suspended
    Join Date
    Jan 2010
    Posts
    101
    What constitutes a good survey? How many variables and choices would it need to have? Or, is a survey not a good form of finding ones opinion? For example, if I made a survey asking the public, "Who is the best actor?" Then, showed pictures to choose from, how am I supposed to know that they weren't picking the most attractive one? What would you consider a good survey and what would you consider a bad survey? I just need some opinions on this matter, thanks in advance.


    Reply With Quote  
     

  2.  
     

  3. #2  
    Veracity Vigilante inow's Avatar
    Join Date
    Oct 2009
    Location
    Austin, TX
    Posts
    3,500
    A survey is only as good as the clarity of your terms, the directness of your questions, and the relevance of your answer choices.

    Your question "best" fails right there... Best, how? We all define it differently, and in different contexts. I might consider "best" the prettiest, whereas you might consider "best" the most highly paid.

    You need to spend considerable effort asking smart questions. Instead of "who is the best actor," you would need to ask something like, "which of the actors below has consistently demonstrated their skill in character portrayal?" It's still opinion, but you're at least asking their opinion about a very specific thing.

    Also, your answer choices are huge, and you need to list options which are meaningful. If you list a bunch of actors and ask people to choose their favorite, you're invariably going to get folks who don't see their favorite actor, or who don't see any differences in your list.

    One way around that is to ask one question per actor and then have them rank them on some sort of Likert scale (like 1 to 7, where 1 is poor and 7 is excellent). You then ask the exact same question about some other actor and compare the average and standard deviation. This allows you to make a more meaningful comparison across different actors (the ones you're interested in), and also provides you with numbers (you can see how large that difference is... you can quantize it... and it's much less qualitative).

    Example - "On a scale of 1 to 7, where 1 is "poor" and 7 is "excellent," how would you rate Brad Pitt's skill in character portrayal?" Then, you do it again, but this time with a different actor. "On a scale of 1 to 7, where 1 is "poor" and 7 is "excellent," how would you rate George Clooney's skill in character portrayal?" You can also ask specific questions in this way... "On a scale of 1 to 7, where 1 is "extremely ugly" and 7 is "extremely attractive," how would you rate the physical appearance of Scarlett Johanson ?"?" Then, you do it again, like this for comparison: "On a scale of 1 to 7, where 1 is "extremely ugly" and 7 is "extremely attractive," how would you rate the physical appearance of Paz Vega?"


    Overall though... surveys can be VERY powerful tools if you have designed them well, and if you gather enough responses from a large enough population. The more people you get responses from, the more confident you can be that the averages you are sharing are stable (given that you've taken efforts to get a representative sample... a sample of respondents that is a close approximation of society as a whole).


    I hope that helps. I've got some experience in this area, and the above are just some quick thoughts off the top of my head.


    Reply With Quote  
     

  4. #3  
    Suspended
    Join Date
    Jan 2010
    Posts
    101
    Thanks, a lot of good information for developing an accurate survey.
    Reply With Quote  
     

  5. #4  
    Universal Mind John Galt's Avatar
    Join Date
    Jul 2005
    Posts
    14,169
    inow, there are doubtless a plethora of books on this topic. Could you recommend any in particular?
    Reply With Quote  
     

  6. #5  
    Veracity Vigilante inow's Avatar
    Join Date
    Oct 2009
    Location
    Austin, TX
    Posts
    3,500
    Unfortunately, no. I've never read any books on the subject. I learned everything I know from experience (collecting data of my own for research in school, working with others in their research and dissertations, and starting my career in this type of research for a national organization). Sorry.
    Reply With Quote  
     

  7. #6  
    Forum Sophomore
    Join Date
    Sep 2009
    Location
    Goomalling, Western Australia
    Posts
    178
    Quote Originally Posted by inow
    Unfortunately, no. I've never read any books on the subject. I learned everything I know from experience ...
    isn't that always the way?

    I know books* have been written about survey techniques and designs,
    but I've never read one -
    what little I've learned has been ad hoc in study or work ...

    *and/or chapters within larger texts -
    these are often tailored to the broader subject (market analysis, psychology, government administration, etc)

    there are booklet style or powerpoint guidelines or tutorials available online -
    some of which might be suitably adapted for a given purpose ...

    eg: Survey Questionnaire Design -
    a pdf booklet from Fairfax County Dept of Systems Management for Human Services
    Nature abhors perfection; cats abhor a vacuum.

    "I don't know; I'm making it up as I go ..." Dr H Jones (Jr).
    Reply With Quote  
     

  8. #7  
    Forum Ph.D.
    Join Date
    Oct 2004
    Location
    Maastricht, Netherlands
    Posts
    861
    Anastasi, A. & Urbina, S. (1997). Psychological testing. [7th edition] Upper Saddle River, NJ: Prentice-Hall.
    Cronbach, L. J. (1990). Essentials of psychological testing. [5th edition] New York, NY: Harper Collins Publishers.
    Gregory, R. J. (2004). Psychological testing: History, principles, and applications. [4th edition] Boston, MA: Allyn & Bacon.
    Hogan, T. P. (2003). Psychological testing: A practical introduction. New York: John Wiley & Sons.

    These are fairly good books on testing theory within psychology. They should include between them, test construction, various kinds of validity, reliability and how they can be measured and improved upon.
    I suggest browsing them before purchase so you are sure that this is the sort of information you are looking for.
    Reply With Quote  
     

  9. #8  
    Forum Isotope
    Join Date
    Feb 2009
    Location
    Transient
    Posts
    2,914
    Holy crap you're alive!
    Wise men speak because they have something to say; Fools, because they have to say something.
    -Plato

    Reply With Quote  
     

Bookmarks
Bookmarks
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •