Stats NZ has a new website.

For new releases go to

www.stats.govt.nz

As we transition to our new site, you'll still find some Stats NZ information here on this archive site.

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+

Introduction

At Statistics New Zealand (SNZ) an important aim of our ongoing work is to understand, manage, control and report on all known sources of error that impact on the quality of our statistics. Some of these sources simply reflect the inherent variability that exists among the units we are seeking to measure and summarise. This variability manifests itself in sampling error when we use samples for cost-effectiveness reasons to estimate characteristics about a population. Sampling errors are relatively easy to measure and are usually reported for all of our sample based statistics. Other sources reflect process, measurement and inference errors, and are referred to as non-sampling error.

It is not possible to eliminate all sources of error. However, our continued efforts at understanding and managing variability and error ensure we are exercising a high level of control on all known sources of error through the targeting of resources to control the errors efficiently and effectively. The design of our collections is based on the resulting statistics being reliable enough to support the main uses of the data. More demanding uses of the data need to take into account the various likely sources of error which could impact on that use. This document aims to provide information about the various sources of non-sampling error to allow users to make some informed judgement about the use of statistics produced from surveys of businesses.

There are seven key aspects to our survey practices that contribute to the successful management and control of non-sampling error:

  • We keep surveys focussed on simple measurable objectives.
  • We invest a lot of resources in our survey frames so that they cover a very high percentage of the population and are as up to date as economically possible.
  • We thoroughly test questionnaires before they are used in a survey.
  • We follow up non-respondents and consistently achieve high response rates.
  • We have high quality edit and imputation procedures in the processing systems for our surveys.
  • We are transparent and open about our methods.
  • We use standard procedures and processes at all stages of the survey process.

These aspects of our practices are referred to many times in the following pages. This reflects that, not only are these well-established parts of our operating culture, but also that their ongoing development attracts much of our attention and resources.

While there are some sources of error present in both economic and social surveys, each type of survey has a range of unique types of non-sampling error. The following pages focus exclusively on the sources of non-sampling error and their management in our economic surveys. In the near future a similar set of pages focusing on our social surveys will be placed on this website.

The general concept of non-sampling error and the associated issues are discussed below. This is followed by links to pages providing the answers to frequently asked questions on more detailed aspects of non-sampling error at Statistics New Zealand, relating to the design, measurement and inference phases of the survey process. These pages discuss the specific sources of non-sampling error, why they can be a problem, what our current procedures are for controlling them and what we will be focusing on for improvement in the future. In doing so, we point to those key aspects users need to know in terms of possible impact on particular uses of the statistics.

top

What is meant by 'error'?

All surveys which are conducted to measure one or more variables of interest, such as total sales or total expenditure, are subject to some form of error. This error can be broadly classified into two major types, sampling error and non-sampling error.

Sampling error arises due to the variability that occurs by chance because a random sample, rather than an entire population, is surveyed. Consider a sample of units selected randomly from the population and then used to make an estimate for a variable of interest. If we repeated the process (random selection) we would almost certainly select different units into the sample, and hence get a slightly different estimate for the variable - this is the concept of sampling error. The level of sampling error can be estimated numerically and used directly when analysing the results for a variable in the survey.

Non-sampling error is all error that is not sampling error. It is present in both sample surveys and censuses, and can occur at any stage of the survey process. There are many potential sources of non-sampling error, for example, businesses not responding to a survey, processing errors, or respondents unintentionally reporting incorrect values. The greater the impact these sources of error have, the greater the difference will be between our survey (or census) estimate and the true value. The difficulty with non-sampling error is that usually we can not directly calculate a numerical measure of its effect, and this can make it challenging to incorporate when analysing results. This is a problem since
the size of this error can be large, potentially even larger than the sampling error.

top

Why does non-sampling error occur?

Non-sampling error can occur for many reasons, such as the complexity of survey processes, inconsistencies in procedures, lack of understanding of issues by staff, poor questionnaire design leading to respondent misinterpretation, poor systems leading to processing errors and data adjustments, or even user misunderstanding. These problems can occur at one or more places in the survey process.

Statistics New Zealand devotes substantial resources into improving its systems and procedures to control the potential sources of non-sampling error. Statistics New Zealand is committed to providing the highest quality data and results commensurate with the requirements of the key uses of the data. We will continue to focus on understanding the issues surrounding non-sampling error and developing new initiatives to further enhance our control of this type of error.

top

Why should I be concerned by non-sampling error?

Precisely because it is difficult to measure numerically the effect of non-sampling error on an estimated value. A better understanding of the reasons the errors occur, the types of errors that exist and the level of these errors, will allow you to assess some of the limitations encountered when using our data and how your work may be affected.

This knowledge will allow you to make more informed judgements when analysing our data. For example, if you were analysing manufacturing sales in New Zealand, you would want to know how manufacturing is defined, what businesses are included and what businesses are excluded. You may also wish to know the sampling error for the sales variable to learn the level of confidence that this variable is estimated with. This knowledge would assist you when making conclusions about manufacturing sales in New Zealand.

In a less tangible way (but just as significantly), you will be more informed when making conclusions about manufacturing sales if you understand any limitations that exist in the data. This does not mean doubting the estimated value, but rather knowing which sources of error have been controlled, and where there are still limitations, why they occurred and what their effects are. You can then assess for yourself the possible impact, or indeed lack of impact, that non-sampling error may have for your project or analysis.

top

Is this just a problem with Statistics New Zealand economic surveys?

No. Non-sampling error is a problem that will be encountered by any person or organisation running a survey (or census), whether it is economic or social. The degree to which the data from a survey (or census) will be affected by this type of error depends on the complexity of the processes, how well the causes of error are understood and the level of resources that has been put into controlling these errors and minimising their effect. At Statistics New Zealand we recognise the potential problem of non-sampling error and have implemented a number of initiatives to reduce the impact that it will have on our surveys and census.

top

More detailed aspects of non-sampling error at Statistics New Zealand

Use these links to find answers to frequently asked questions on the specific sources of non-sampling error at Statistics New Zealand, and learn about how these sources relate to the design, measurement and inference phases of the survey process.

First steps toward controlling non-sampling error 
- an introduction to how we control this type of error, what the two major types of this error are, and how the subsequent non-sampling error web pages and discussion are structured.

Non-sampling error identified in the initial design phase of the survey process 
- controlling errors occurring in the sampling frame and questionnaire design.

Non-sampling error identified in the measurement phase of the survey process 
- controlling errors occurring in interviewing, responding and processing.

Non-sampling error identified in the inference phase of the survey process 
- controlling errors occurring due to non-response and imputation, data adjustments, and misinterpretation.

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Top
  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+