Citizens’ panels are an easy and cost-effective way for local government to consult with their community. A lot of thought and planning goes into a survey even before the first question is drafted. Surveys take a lot of time and resources to develop, analyse and report on and you do not want to get to the drafting of the final report and find that the survey data does not actually supply you with the information you require or that the data is unusable. As they say, prior planning and preparation prevent poor performance. This article will go over some of the best practices when designing surveys for panels.
When planning a survey there are a number of factors to consider:
- What do you want to learn from the survey? This normally takes the form of 1-5 learning objectives
- Who is the target audience of your survey?
- What is the purpose of the final report? including who is the end reader?
- What are the potential actions that could be taken based on the results of the survey?
Citizens’ panel surveys often stem from a piece of work or a project being carried out by local government. It is essential to keep in mind what the objective of the overall project is and to align the survey objective with this. It is a good idea to include the objective of the survey in the introduction like in the example below. This allows the respondent to be informed not only the purpose of the survey but also how the survey results will be used.
Your target audience will have an impact on how your survey is designed along with the style and type of questions you ask. One of the biggest barriers to survey participation is the belief from respondents that they don’t know anything about a topic so they don’t even look at the survey. A classic example is urban planning. It sounds scary and very technical yet you would struggle to find someone who doesn’t have an opinion on car parking, traffic flow, where pedestrian crossings are needed and how to improve the main street. All of these items fall under urban planning. The trick is to pitch your survey so your target audience is motivated and inspired enough to provide feedback.
During the planning stage, you need to consider the final research report that will be produced as a result of the survey. Issues like what type of data will be included in the report and how the data should be presented needs to be considered. The type of data you need or want in the final report will shape the style of questions you choose.
You also need to consider the report audience which will have a huge impact on how the final report is written. Is the report going to be read by technical experts such as water engineers, will it be read by Councilors or is it being prepared for a general audience? The audience will shape the length and depth of the report and the questions you ask as well. If the audience is experts in a field you can go into depth on issues and into to technical aspects and fine detail whereas if a report is being prepared for Councilors who may not have in-depth knowledge of an issue, just need enough information to make an informed decision and often have dozens of papers and reports to read a non-technical and shallower report would be more appropriate. This means the questions asked don’t have to go into intricate detail.
Once the survey is done and the final report is present what are the potential actions that can be taken by decision makers. Surveys are often used to collect data so decision-makers can make decisions based on evidence. During the planning stage, it is good practice to identify the various actions or decisions that could occur as this will assist in question design. A good example is proposed changes to local liquor bans. The potential actions are the liquor ban areas remain the same or they change, from this one can design a question on whether there is support for a change in liquor bans and follow it up with a question on proposed areas any new ban may apply to.
Once the planning stage has been completed then you can move onto survey design.
Keep Surveys Short
In terms of survey design, it is best to keep surveys short. A rough guide is that a survey should take less than five minutes to complete. The longer the survey is, the greater the potential of survey fatigue setting in. Survey fatigue can result in a respondent clicking any old answer to get the survey completed, thus providing noisy data. Long surveys can also increase survey drop out rates leading to partial responses which then have to be cleaned out of your data. It’s useful to keep in mind that an average respondent can answer five quantitative questions a minute and two open-ended (text questions) a minute.
When designing the questions for your survey it is important to keep in mind the objective of the bigger project but also the objective of the survey and the elements identified during the planning stage. It is recommended that questions are grouped around the learning objectives and that the questions are designed with the audience in mind.
Think of the survey as a conversation
While designing and laying out questions think of the survey as a conversation between you and the respondent. Citizens’ panel surveys can deal with a range of issues from the complex such as representation in Council, long-term plans through to the basic day to day issues such as changes to local bylaws to allow dogs to walk in a local park. It is important to use language that panel members will understand. It is best practice to avoid the use of jargon or abbreviations; if abbreviations have to be used it is recommended that what the abbreviation stands for is spelt out. The questions themselves should be short, concise statements which are meaningful on their own. The questions can use a mix of positive, negative and natural language however it is important to avoid leading questions as this will result in bad data.
Provide a response for everyone
One common design error is not providing a response option for all respondents, this can be as simple as providing a ‘Don’t Know’ or ‘Not Applicable’ option. If the response options do not provide for everyone the respondent may choose not to complete the survey meaning you miss out on data or they will choose the response which closest fits them which can lead to noisy data and have an impact on the survey results.
When using rating scales such as a Likert scale which is a common rating tool it is important to keep the scales consistent throughout the survey. This not only applies to the scores i.e. a sourcing scale of 1-5 but also the values of each score. If you change the scale part way through the survey it may confuse the respondent, they could answer a question incorrectly or get frustrated and not complete the survey. All of these factors impact on the data collected and the results of the survey.
As discussed earlier it is recommended that questions are grouped together, one way of doing this is by using a grid. It is good practice that a grid question has no more than five rows.
It is a requirement under the Local Government Act 2002 that local governments have an engagement policy on how it will engage with communities on issues, proposals, assets and other activities or matters that are significant or may have significant consequences; a number of Councils have established citizens’ panels as part of their engagement policy. Citizens’ panels are easy to set up and maintain, are cost-effective and the data collected enables decisions and policy to be informed and backed by research.