EXCERPTS:
Canada's National Household Survey on incomes produced flawed data with harmful implications for public policy, according to a range of researchers and statistics experts who have sifted through the numbers.
Consultants, urban planners and health policy experts say the data quality is worse than they'd expected, masking key shifts in income inequality and poverty in the country. The blurred picture has also left them unable to track trends over time. And they question whether the higher cost to produce data of worse quality - at $22-million extra for the survey for a total census cost of $652-million - amounts to wasted taxpayers' dollars.
Statistics Canada developed the new voluntary survey after the federal government cancelled the mandatory long-form census in 2010. Some call the data from the survey "dangerous" because lower response rates obscure who is faring better and who is worse off - which could lead to misguided policy decisions in the years ahead.
Voluntary surveys typically garner weaker response rates from those at the higher and lower ends of the income spectrum. And response rates to the National Household Survey show much lower responses in many mid-sized cities, smaller communities and rural areas.
The 2006 mandatory long form, which required everyone who received a questionnaire to fill it out, had a response rate of 93.5 per cent. The NHS was voluntary and had a much lower response rate of 68.6 per cent. A less representative sample makes it much more difficult to draw unbiased conclusions.
Many who rely on the data plan to discard using it altogether.
"The Auditor-General should get on this one," said Thomas Lemieux, professor at the University of British Columbia and president of the Canadian Economics Association.
He says he will stay clear of the NHS data in assessing long-term trends on income inequality. "I had a secret hope that because Statscan has lots of information to fill the hole that they would be able to create still a workable, good-quality product. My heart fell when I saw that Statscan said they're not going to compare to 2006...I didn't think it would be that bad. "
Munir Sheikh, Statistics Canada's former chief statistician who resigned in 2010 over the decision to scrap the census, said the effort has been a waste of money.
"The irony is, we've spent more money compared to a census to get data which is largely useless. Why anyone would want to do this is beyond me. Why would you spend $600-million for this?" said Mr. Sheikh, who is now an adjunct professor at Queen's University.
In Winnipeg, Dr. Patricia Martens, director of the Manitoba Centre for Health Policy, says flawed or missing data - that under-represents, for example, the number of poor children in her province - could lead to misaligned funding for early childhood education programs or immunization campaigns.
"I worry we've left out the most vulnerable people, so we might make erroneous statement like the income gap is shrinking when it's not," said Dr. Martens, who is also a professor at the University of Manitoba's faculty of medicine and member of the Order of Canada.
Her mistrust of the data is such that her team plans to ditch the NHS altogether, and instead rely on the older, mandatory 2006 census.
Statscan has long been regarded as one of the top statistical agencies in the world and many held out hope its respected analysts would find a way to make the data comparable with previous years. But the agency bluntly said in its Sept. 11 release people should "use caution" in comparing the NHS income data with other surveys or censuses due to the methodology change.
David Hulchanski and a team at the University of Toronto have tested the NHS income data. They have found it doesn't align with annual tax-filer data, with "wonky" results in many areas - in particular showing rising incomes among poor people and falling ones among the rich.
"We're concluding it pretty much is garbage," said Prof. Hulchanski, who co-directs data analysis of the Neighbourhood Change Research Partnership based at U of T.
They are concerned the data will be used for partisan purposes and make it seem like Canada is becoming a more equal society "when the opposite is happening."
In Calgary, consultant Robert Gerst is urging his clients - municipalities, provinces and companies in the oil-and-gas sector - not to use the new data. "It's dangerous," he said.
Planners in his city have already started using the survey to help determine where to place new bus routes - though faulty data could mean empty seats and millions of dollars wasted, he said. "It's a mess because it's not that the data is wrong. It's unreliable - we don't know whether it's right or wrong, we have no clue. And there's no scientific way to tell."
Not everyone is ditching the survey. Doug Norris, chief demographer at Environics Analytics in Ottawa, says though he won't use it for long-term comparisons, it still provides a useful snapshot of how incomes in some cities or provinces compare with others. The problem may also be less acute for the private sector because many businesses rely on their own market surveys to make decisions.
Statscan says each user should determine for themselves if the NHS income information meets their particular needs.
"The fit for use depends on the particular analysis done," Statscan said in an e-mailed statement to The Globe. The new survey "is a very rich source of information and meets the requirements of most users."
The federal government hasn't yet said what it plans to do for the 2016 census. Mr. Sheikh recommends making Statistics Canada an independent agency from the government, similar in structure to the Canada Revenue Agency. Dr. Martens in Winnipeg wants to see the return of the mandatory long-form census. And the University of Toronto professors recommend the next census be restored to the "non-politicized, non-partisan scientific methodology that existed before" the NHS.
Industry Canada, which oversees Statscan, told The Globe a review is still under way for the next census. "We continue to look at ways to improve the next census cycle while protecting Canadians' privacy."
-reprinted from the Globe and Mail