What They Say vs. What They Do

Researchers and media love their surveys. They are relatively easy to set up, and once the data are collected, they have a certain air of authority in them. However, while surveys can be useful, even if performed correctly, they are limited by a number of factors, not the least of which is the survey responders' lack of self-awareness that is required for gathering true insight into their own motivations. It's far more useful -- and often difficult -- to watch their behavior.

Fire the bums!
For instance, recently Rasmussen conducted a poll that concluded that 56% of Americans think we should throw all of Congress out on their ears and bring in new representatives. With an overwhelming incumbency retention rate around 95%, though, we are left to wonder why there is such a disparity.

Coke is it. No, really.
Back in the 80s Coca-Cola panicked when Pepsi started winning blind taste tests, so they tweaked their recipe until their new formula won in the blind taste tests they conducted. They introduced their new formula with much fanfare and a chipper Bill Cosby. Public outcry over this unnecessary change was so great, Coca-Cola had to backtrack -- offering both versions for awhile until they got rid of "New Coke" altogether.

What can we conclude from all of this?

While most people probably don't intentionally lie when they're answering survey questions or participating in a poll, the fact remains that surveys don't necessarily accurately measure the root causes of anything, and they certainly can't accurately predict what is going to happen or what can happen if a company (or a politician, for that matter) provided some actual leadership in its respective category.

It hurts none to ask people what their preferences are, but gaining insight from the answers requires more than reporting the results. First, you don't know that you're asking the right questions. And second, even if you were, the participants of the survey may not be able to think about the questions in the terms that conclusive action requires.

What people do is far more telling than what they say. But whether we conduct a survey, perform a usability test or conduct an ethnographic study, we mustn't be afraid to see what we see. - Cam Beck

Comments

[Stephen Denny]

Quick! What do you think about the customer service you get from your second favorite credit card?

Response: "... uh... well... it's fine, I guess..."

As recorded by survey taker: [excellent]

Surveys and other quant studies are only projectable if the respondant cares about what you're asking. There's little value in doing deep statistical projections based on purely subjective fluff.

[Cam Beck]

Stephen - Awhile back I got a call from a pollster. Not sure why they thought I would be a good candidate for that type of call, but they called nonetheless, and asked if I thought it was imperative to "rally behind" a certain politician.

I wanted a follow-up question, but immediately in my mind I was running through all the situations where it would be wrong to rally behind this politician. After all, what if he's wrong? Are we still expected to rally?

The answer they were looking for was "yes," but I disappointed them with an emphatic "no."

"Thank you for your time. Goodbye."

Drat. I wish I knew what they did with that information.

[Lewis Green]

Cam,

You are right on target. Surveys can be useful as a benchmark but to tell us what those surveyed really think and how they act, we must gather data in real-time through observation or via internal marketing techniques using software that both gathers data when customers call, write or otherwise initiate communications and then analyzes the data to predict behavior.