One of the key reasons to have a citizens’ panel is to allow Councils to poll the members of their communities easily on local issues. This raises an important question – How reliable are the results from an online citizens’ panel? Or to put it another way, is a citizens’ panel survey scientific?
The best way to tackle this question is by reviewing what used to be the gold standard of surveys – the telephone poll. I then look at how it is possible to build a citizens’ panel which is more efficient and cost effective than telephone surveys for community polling.
The Decline of the Telephone Survey
Since the 1930’s randomised, telephone surveys have been perceived as the gold standard when it comes to opinion research. Telephone surveys provided a cheap (when compared to face-to-face interviews) and scientifically random way of polling communities on a broad range of topics. Market researchers cottoned onto this pretty quickly and soon people were becoming inundated with telephone calls asking their opinion on subjects from motor oil to mortgages. The result is that people grew pretty fed up with having their evenings and weekends interrupted by unsolicited research calls. This frustration has increased the number of people refusing to take part in market research calls. The result has been an increase in non-response bias for telephone surveys in New Zealand and internationally.
At the same time, access to landline telephones has decreased in New Zealand. Statistics New Zealand reported that in 2013, 85.5 percent of households had access to a landline phone at home, down from 91.6 percent in 2006. The phenomenon of falling landline ownerships is due in large part to two factors. Mobile access has increased (83.7 percent of households in 2013 having access to a cellphone at home, compared with 74.2 percent in 2006) and the availability of naked broadband (internet connection without a landline) packages.
While the development of the internet in the 1990’s has helped to contribute to the death of the telephone survey, it has also brought with it an even more economical and efficient way of capturing opinions – the online survey. And with online surveys came the research panel.
What makes a survey scientific?
There are two key features of a scientific survey. They are:
- The Sampling Methodology
- The Representativeness of the Sample
The gold standard that most researchers are looking for is a random sample which is broadly representative of the survey population. This is exactly what we do when we are recruiting for our panels.
A scientific survey must have a representative sample.
What makes a survey unscientific?
An unscientific survey can be spotted a mile away by its sampling technique. The most common unscientific survey you’ll come across is a self-selecting poll. There are loads of these on the internet and are most common on news websites and the like.
While they are fun to complete they only provide insights into the thoughts and opinions of the type of people that visit the particular website they are hosted on. Not the general population.
So is a Citizens’ Panel scientific?
Yes – I believe it is. In a world of declining telephone survey participation rates and increased internet access, we believe a citizens’ panel is the best thing going. Telephone surveys are suffering from ever decreasing response rates and fewer and fewer people now own landlines. The result is it is now becoming harder and harder to achieve a representative sample when using the telephone.
Since a citizens’ panel is able to provide both a randomised and representative sample it satisfies both requirements of a scientific survey.
The key thing to pay attention to when developing your citizens’ panel is the recruitment process. It is essential that a degree of randomness is applied to the recruitment process, and effort is made to ensure that the panel is representative of the total population you are seeking to represent.
How to recruit a scientifically representative panel?
Recruitment, recruitment and recruitment. Panel recruitment is the key element which will determine whether or not your panel is capable of generating statistically significant results. The key thing with the recruitment process is we spend a lot of time and effort in ensuring that the bulk of our panels are randomly selected (either by telephone or street interception).
We pay particular attention to the makeup of our panels to ensure they have a good mixture of the following demographics and geographic variables:
- Age
- Gender
- Ethnicity
- Geographic Location
The Statistics New Zealand table builder is an invaluable tool when trying to figure out the makeup of your local community.