Stats NZ has a new website.

For new releases go to

www.stats.govt.nz

As we transition to our new site, you'll still find some Stats NZ information here on this archive site.

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Non-Sampling Error and Questionnaire Design (QD)

Questionnaire design 

Questionnaire testing

Questionnaire testing specifically relates to the detailed and structured techniques that ensure a questionnaire will obtain the information that is actually required. Testing will identify potential problems with the questionnaire that may lead to inaccurate information.

Inadequate questionnaire testing leads to poor questionnaires being constructed. For example, if the questions are difficult to answer or the questionnaire is too long, the respondent is less likely to complete the questionnaire which leads to a lower response rate. Poor data quality is another problem that can occur if testing is inadequate. For example, if a large number of respondents interpret a question differently, we could get a large bias, and perhaps significantly overestimate, or underestimate, a particular value.

Testing ensures that non-sampling error is tightly controlled, and testing will continue to have a high priority to ensure we maintain a high quality standard. Statistics New Zealand's questionnaire design standards require three main testing methods to assess quality: cognitive interviewing between the respondent and designer before the questionnaire is finalised; technical review; and pilot tests conducted by mailing the survey out to a small group of respondents in the survey population. These methods allow us to check if the responses are matching expectations, and also address any
problems that are identified.

Statistics New Zealand keeps up to date with the best and latest ways of testing questionnaires from local and overseas research and experience. Different testing methods are investigated and used if appropriate. For example, we have recently used focus groups and behaviour coding in economic survey testing. Methods used for social survey testing, such as the 'embedded interviewer questions' technique, could also be used for economic survey testing when appropriate. We are continuing to investigate innovative ways of incorporating new testing techniques into our survey processes, to maximise their benefit.

top

Concepts in questionnaire inadequately defined

The concepts and terms used in a survey need to be well defined before a questionnaire can be developed. This implicitly requires that information needs and data outputs are clearly specified. As different respondents may understand concepts in different ways, we need to make sure that the terms and language style used to collect the survey's information are clear to respondents. Otherwise they may interpret them in a different way than that intended, or not be able to answer the question(s) at all. For example, a respondent from a large business may be more likely to understand and correctly interpret financial accounting concepts and the associated language than a respondent from a smaller business. The ultimate impact that this source of error could have is to bias final survey outputs, such as having large businesses showing one trend and smaller businesses showing another.

The detection of poorly understood concepts occurs in the questionnaire testing phase, at which point changes are made in consultation with subject matter specialists within Statistics New Zealand. However, note that this only identifies poorly understood concepts at that point in time. It does not resolve the issue of the rapidly evolving modern economic world where concepts and practices may change in a relatively short time frame. Regular maintenance of questionnaires is required to allow for real world changes.

top

Questionnaire updating in each new survey period

Each survey period, a new questionnaire is required. Modification is usually minor, such as changes to dates, logos and/or signatures. In general, the actual questions remain the same as for the previous period. The impact from this source of error is likely to be quite small since in general only minor changes are required.

As it is important to have consistency over time, there is usually little need for detailed re-testing. However, as the results of survey must remain relevant, major reviews are scheduled periodically. Concepts and wording are periodically reviewed, while balancing the need to avoid introducing discontinuities in time series data (could occur if businesses start reporting differently). In addition to changes from these reviews, informal feedback from our respondent liaison division is considered by the design team for more immediate adjustments to questionnaires.

top

Questionnaire not aligned with issues in the economy

Changes in the economy can affect the appropriateness of survey questions. These can be the result of intentional changes, such as changes in the legislation that governs the running of businesses or the way they are taxed, or situational shifts in the economy, such as the way information is sourced (greater reliance on computer systems) or perceptions held by employers. The potential impact from this source of error could be that questions are more difficult to answer or misinterpreted, leading to poor data quality or even non-response. This means that the responses may not provide a true indication or measure of the economic variables of interest.

Most major changes to survey questions are generally kept until the next redevelopment is due. However, if feedback and investigation suggests a survey is becoming different from the economic situation that it is being used in, the survey will gain an increased priority for redevelopment. Questions can also be modified between reviews as it is important that they remain relevant for the life of the survey.

top

Incorporation of new technologies

There are a range of technologies (new and existing) available for the transmission and capture of data, including the traditional postal method, scanning and recognition engines, personal interviewing, fax, email and the Internet. Statistics New Zealand is also developing 'StatsGate', which will be a secure web based data transfer system. There are benefits to be realised by incorporating new technologies, for example, if a respondent can choose the way they respond, it is much more likely that we will get a response.

Introducing new technology is a potential source of error, and if it is not controlled well, the negative impact could exceed the benefits of incorporating new technology. For example, data that is emailed may be less likely (than that provided on paper form) to indicate whether the value includes GST. The issues of confidentiality and security must also be thoroughly worked through, as they have been for paper forms, to ensure that users and respondents can be confident that results remain secure. However, if it is controlled well, the introduction of new technology can result in significant improvements to processes, and even a reduction of the non-sampling error (compared with the existing technology).

We continue to trial new technologies with specific surveys and evaluate the success of those trials for use in other surveys. Statistics New Zealand is currently setting up an internal imaging centre, where we will be able to electronically scan and recognise forms.
A project investigating the creation, and ultimate use, of true electronic forms is now underway. Research projects looking into adapting the current questionnaires to other forms of electronic spreadsheets, as well as looking at the relationship between questionnaire design and the emerging technologies being planned for introduction to Statistics New Zealand, are also underway.

top

Information not readily available

Sometimes it is clear what is required but not so easy to get that information. This could happen if the internal reporting period for a business is different to the survey requirements or if the information is just not available at the time of the survey. The questions can also be difficult to answer if, for example, the questionnaire requires answers from different areas of a business. The impact of this will be increased respondent load which, if it becomes too high, could lead to lower responses rates to the survey.

The range of businesses surveyed is diverse, which limits our ability to make it easy for all businesses to provide the required information. However, there are other ways we deal with this problem, for example, by asking respondents to tell us what period they are reporting for. They give us the information they can, and then we try to match that data to the required period. In this way this source of error does not normally cause significant problems. We will continue to better understand the requirements of respondents, and make improvements where possible.

top

Unable to individually tailor questionnaires

This source of error can lead to some respondents not being able to understand what is required in certain questions. Currently however, this is not considered to be a significant problem. When the information needs and the type of information for surveys are similar, the questionnaire format and the wording of the questions are generally kept as similar as possible. For some surveys it is possible to tailor the questions for different types and sizes of businesses. An example of tailored questionnaires being used is in the Consumer Price Index Survey which is customised for each respondent, and to a lesser extent the Annual Enterprise Survey, which has questionnaires customised for industry groups.

While it would be ideal to undertake this for all surveys, it is not always practical for cost and logistical reasons. In the future we will continue to consider this source of error in relation to each specific survey, and if it is decided that it could be a significant problem, the tailoring of format and questions will be investigated and addressed in questionnaire testing.

top

Language used

While multilingual forms have been used with success in our household surveys, they have not been utilised in our economic surveys. The primary reason for this is that the New Zealand IRD requires records in English. Since it is compulsory for a business to report to the IRD, it should have the information we require available in English. However, if the person actually filling in the questionnaire can not understand it due to the language, this could lead to us getting incorrect data, and ultimately is likely to lead to a lower response rate.

Currently very little is done at the questionnaire design stage since this is not considered to be a significant source of error. However, while Cantonese or Mandarin questionnaires may be useful for some businesses, a greater overall reduction of non-sampling error would be made by having Cantonese or Mandarin speakers available to the processing sections when required. These people could be available to contact once the survey has been sent out, and could also call back respondents who may have provided only some information. This is a good example of how we are targeting our resources to the areas that will give the greatest benefit. Having people available for all surveys is far more efficient than making all surveys multilingual. In the future we will monitor this source of error but we will continue to target resources in the place that gives the greatest benefit.

top

Dress rehearsal (test) conditions different from reality

A dress rehearsal is a very valuable exercise, but there is potential for the testing conditions to be different to those at the time of the actual survey. Some of these conditions are more obvious and measurable, such as the timing of questionnaires with tax returns for annual surveys. However others are not, such as the fact that the level of contact with a business during testing will be much higher than when the final questionnaire is sent out. If problems are not identified this can lead to unexpected and unwanted results in the main survey. This may also lead to lower response rates if the load is actually greater for a lot of businesses than testing showed.

We try to make the testing conditions as close to reality as possible, using our experience to identify potential areas that could otherwise be overlooked. A range of business sizes and types are included in the testing to ensure comments are representative. There is often a trade-off though, between the number of each type of businesses we would like to go to and the time and resources available to complete the work. In the future we will continue to incorporate the knowledge of this potential source of error into our overall testing procedures to ensure that we simulate real conditions as closely as possible.  

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Top
  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+