Brainstorming
It’s obvious we need to see how real people rate others. What kinds of questions would get quality results while also providing value to the user being analyzed?
Given some real pictures of ourselves and friends on social media, ask them:
- kinder/nicer
- set you up
- expert in field
- homework help
- sociable
- knows a lot of people
- set up on date
- competitive
- loan $100
- loan cell phone
- which seems like a kinder person?
- which would you let set you up on a date?
- which is an expert in their field?
- which has the most friends on Facebook?
- which would you let borrow your cell phone to make a call?
Hard data – verifiable
Clustered data – voluntarily given
Derived
Facts
Posts
Assumptions
Iterate
Testing and data collection
In-Person Interviews
We interviewed a diverse selection of people around NYU in-person. Some combination of the following questions (with specific pictures associated with each) were used with yes/no answers and why they answered that way.
- Hire to plan an event?
- Hire this guy?
- “he looks smarmy–no way!”
- “it’s clear it’s a Halloween party and I love Halloween, so he’d get points for this elaborate costume”
- “is capt. morgan his alter-ego? not good”
- Is he interested in guys or girls?
- Would you invest with this guy?
- “yes, bc thinks all money managers are kind of sleezy anyway”
- Invest in hybrid annuities?
- Would you loan him money?
- Would you let him be your banker?
- “Only because he looks too young. The drinking doesn’t bother me. But he’s too young to be my banker….”
- Take care of your pets?
- Take care of your parrot?
- “NO WAY! He’s got a CAT!”
- Watch your dog?
- Would you hire this person as a babysitter?
- Would you have this couple set you up on a date?
- Would you have coffee with this man?
- Would you hang out with him? Does he look like an interesting person?
- What’s this guy feeling? scared or scary?
- Does he look focused or scared here?
- What reaction to him do you have with this pic?
- Which, if any, of these pictures should be removed from their Facebook profile?
Mechanical Turk
I programmed several batches of Mechanical Turk surveys to broaden the scope of our test group; however, this also comes at the cost of potentially lower quality answers and no ability to probe deeper for ambiguous/multi-faceted responses.
For the first batch, we had respondents rate pictures remove or don’t remove
For another batch, based on multiple pictures of same person, we asked:
- would you hire them?
- what kind of job do they have?
Another batch was done A/B style, where they choose one of two pictures based on questions, and optionally say ‘why’ in a freeform field
- Which seems like a nicer person?
- “She is being friendly with the rooster, so she would be friendly with people.”
- Which would you let set you up on a date?
- Which is an expert in their field?
- Which has the most friends on Facebook?
- Which would you let borrow your cell phone to make a call?
We asked some questions about pictures of couples
- What is their relationship? Would you let them set you up on a date?
- “Lover because her looking is like romance. Yes, she is hot”
- “They look like they are dating because they are holding each other. No, because they probably don’t have anyone to set me up with.”
- “They look like co-workers and the guy with the pen in the mouth is clowning around with her to get her attention. Yes, the blond has to have some hot looking friends”
- Which person would you rather invest with? Why?
And pictures of singles
- Would you hire this person?
- Would you let this person take care of your pets?
- Would you invest with this person?
- Would you hang out with them?
- Does (s)he look like an interesting person?
- What do you think this person’s job is?
- Should this picture be removed from their Facebook profile?
