Newsletter: Can 1,500 Canadians tell us what the entire country is thinking?

In the survey research days of yore, data was collected by knocking on doors and dialing landlines. With a fixed address tied to both, it was easier for researchers to make sure they were collecting a group of people that mirrored the overall population, in terms of where they lived. And they had a way to access (almost) everyone in the country.

But door knocking is expensive and, increasingly, Canadians are doing away with landlines. People like me, who wander around the country and don’t see a point in changing cell numbers, can make it harder to get a good picture of the country. (My cell number is still “in the six” even though I’m living in Vancouver … “the four”?)

Scott Edward Bennett has worked in the field of survey research for almost half a century and now teaches at Carleton University. He’s seen how changes in technology have affected data collection with the shift to online surveys in the last ten or 20 years. “It’s very difficult and very expensive to do them well,” he says.

While online surveys can be faster and cheaper, it can be hard to get a representative sample — a small group of people who accurately reflect the entire population. There isn’t a directory of emails in the same way there’s a list of phone numbers so companies doing online surveys pull from a list of email addresses of people who’ve said they’re interested in participating.

You can still get good online surveys, says Bennett, but it’s “a very tricky business” to make sure that you get some kind of reliable results because online surveying “provides a picture of a certain part of the population but not the whole population.”

So, when The Discourse partnered with Environics Institute for Survey Research to gain early access to findings of the Canada’s World Survey 2018, I wanted to know more about how they got the data.

The 2018 survey is an update to a survey they conducted ten years ago measuring how Canadian attitudes have shifted and what issues emerged. For Keith Neuman, Executive Director at the Environics Institute for Survey Research, it was important for the 2018 version to be conducted over the phone — just like it had been ten years ago. “You can’t really make good comparisons between surveys when you’ve conducted them different ways,” Neuman points out.

So how do researchers get people’s numbers? It’s likely through something called random digit dialling (RDD). There are companies that specialize in creating lists of active phone numbers for survey researchers and pollsters. The goal is to try to maximize the odds that anyone in Canada could be contacted, even though only a small group will be reached.

Another advantage of RDD is it provides a random sample of people to make sure you aren’t getting results from the same group of people (something that could happen in an online survey.)

Once you’ve picked up the phone and agree to take part in a survey, you could get asked other questions about your demographics like your gender, age, income or education. Researchers will look at the census or other statistics and try to mirror the population as much as possible by trying to get certain quotas for each demographic and adjusting the differences through a process called “weighting.”  

Still confused? Getting a random sample and weighting is kind of like stirring your soup and taking a taste test. Stir it well and you can get a small taste of what the whole pot tastes like. Check out this video by the Pew Research Center about the science behind getting a balanced representation of people.

You can see here how Environics weighted their survey results for age after conducting phone interviews with 1,501 Canadians. Neuman says while weighting is an important step, it rarely changes results by more than about one per cent.

So knowing all of this, what should you look for as a reader or journalist to make sure a survey is reliable?

1. The sample: How many people answered the survey and does it represent the population? Generally, the more people that answer the better, but it also depends on how those people were reached. Phone? Email? Was it random?
2. The wording of the questions: Is there any bias in the wording or structure of the questions? Can you take a look at the exact questions and the order they were asked? For example, a leading question might be: How bad is your relationship with your boss? Rather than asking: What is your relationship with your boss? Check out this discussion on Quora for other ways survey wording can bias the responses.
3. The company behind the research: Are they reputable and how are they funded? If you haven’t heard of the survey company, do a quick online search and see if they share how they’re funded and what other work they have done. McGill’s library lists a few reliable survey companies that look at public opinions here.

What we’re wrangling right now.

I delved into some of the findings from the Environics survey in this article that explores how Canadians increasingly believe multiculturalism, diversity and inclusion is one of the country’s most notable contributions to the world — a shift away from peacekeeping and foreign aid.

The survey also estimated that close to 2 million people had a direct role in resettling refugees in Canada over the last two years.

The findings surrounding Canadian attitudes towards refugees led us to more questions and so over the next couple of months, leading up to World Refugee Day on June 20, 2018, The Discourse is investigating Canada’s refugee system and how it responds in times of crisis.

Do you have a connection to the refugee system — either as a newcomer or someone who works in refugee resettlement? What do you think of Canada’s contribution so far? Should we be doing more? Less? Tell us what you think by filling in this short survey.

Know your sources

The Discourse is committed to being transparent and accountable to our readers, so we’ve listed all our sources for this newsletter, here:

• 2014 Statistics Canada’s Survey of Household Spending, table 2.0.5. “Canadians are moving away from landlines and switching to mobile phones only.”
• 2018 Canada’s World Survey, Environics Institute for Survey Research in partnership with the Canadian International Council, SFU Public Square and The Bill Graham Centre for Contemporary International History.[end]