Stats NZ has a new website.

For new releases go to

www.stats.govt.nz

As we transition to our new site, you'll still find some Stats NZ information here on this archive site.

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Non-Sampling Error and Non-Response (NR)

Non-response

Complete non-response

Complete (or unit) non-response occurs when the respondent does not return the questionnaire or it is returned too late. The non-response for most of our economic surveys is currently around 10 to 15 percent of the original sample size. The main impact of this error is its potential to cause a bias in the results since non-respondents may have different characteristics to the respondents. Imputing values for the businesses who do not respond incorporates information from those who do respond. We do not know if these two groups would have given different answers, but the larger the non-response is, the larger the bias could potentially be.

Our current procedures have two main focuses. The first is to reduce the level of non-response and the second is to treat it once it has occurred. To reduce the level of non-response we have introduced a key firm strategy that ensures all of the most influential businesses (for a given survey) are followed-up to complete the survey each period. We have response rate targets set overall, and also set at the industry level to reduce differential non-response between industries. We have also used pre-notification letters and phone calls to ensure that the survey, when sent, goes to the right person in an organisation. This is especially important for the larger organisations.

When non-response has occurred, we use imputation to construct a response for the businesses who have not responded. The imputation uses auxiliary data such as responses from other similar respondents, and historical data such as the businesses' responses in the last period, as a base from which to construct a response for a business. In the future we will be able to exploit further the use of IRD data and other information sources, to obtain values directly rather than through surveying, for some business types. The questionnaire design, as already mentioned, is another area of focus where future improvements will contribute to reduced levels of non-response. This includes, for example, incorporating multi-modal response methods, such as a special secure website 'Statsgate', where registered users can report responses online.

top

Partial non-response

Partial (or item) non-response occurs when the respondent returns the questionnaire but has not responded to all of the questions in the survey. This could introduce bias if the imputed response is very different to the (unknown) actual value. This may happen if, for example, there was non-response by the respondent because the item/variable actually has no value. Conversely, values may be assumed to be zero when in fact they are not, for example, if an item was left blank because the respondent missed the question, or did not know the answer. Although imputation can also potentially introduce bias and volatility, survey estimates are generally better as a result of our use of imputation methodologies than they would be if we had simply ignored the non-response.

Current Statistics New Zealand procedures and systems are very much the same as those noted above for complete non-response. We follow up key businesses for all of the required responses, and try to phone all of the other businesses who have only responded partially to the questionnaire. Unlike the procedures for dealing with complete non-response we do not send a reminder letter since the business has responded. We instead focus on contacting them by telephone. If we still do not get a response, imputation would be carried out using auxiliary data such as the values provided by other similar respondents, and historical data such as the business response last period, as a base from which to construct a response for a business.

top

Non-responding births

These are businesses that are birthed on our Business Frame and into a survey sample but do not respond for some time, if ever, to the survey. These businesses are more difficult to impute for as there is no historical information available. The main impact these units can potentially have is where they are inaccurately imputed from the time they are birthed and do not respond for many periods. When they do provide a response it could be very different from the imputed value. This will cause a discontinuity in the value for that business and would mean that we had been overestimating or underestimating the value in our imputation. However, at the published/aggregated levels this is unlikely to cause a significant problem. Currently, these businesses are followed up through reminders and telephone calls. But for the non-responding births we have to impute a value, and the imputation strategies for non-responding births use mean imputation, since there is no historical information.

top

Imputation methodology

Where there is non-response in a survey we use imputation methods to impute a value, either implicitly by rating up or explicitly by unit-record imputation. The majority of economic surveys use broadly the same imputation methodology, although different surveys require slightly different approaches for the detailed parts of the process. The source of error we are focussed on here is the use of imputation, regardless of the specific method used.

Imputation can potentially introduce bias depending on the data used, and assumptions
made. If the non-response for a survey is high, then the imputed values can form a large component of the estimates, and could potentially mask true movements. This would diminish the confidence we have in the totals or other estimates. There is no specific percentage cut-off, but generally the greater the amount of imputation, the greater the amount of care that needs to be taken when using the results from that industry or survey.

A lot of research has been carried out by the wider statistical community (including educational institutions, companies and international statistical agencies) on imputation methodologies. We are always looking at ways to improve the methodology, either as a result of our own research and experience or that of the wider statistical community. All new or improved methods (and their adaption to our environment) are tested thoroughly before the decision is made to implement them (or not). We try to ensure imputed values are as close to the (unknown) true value as possible, such that we can achieve our aim of having our aggregate estimates (real and imputed data) as close to the true value as possible. For each period of every survey monitoring work is completed by the Survey Methodology Division to identify outliers, and other unusual responses or factors which may unduly impact on imputed values. Over time the methodology is reviewed and improved as necessary, although major changes usually only happen when a survey is redesigned. 

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Top
  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+