Stats NZ has a new website.

For new releases go to

www.stats.govt.nz

As we transition to our new site, you'll still find some Stats NZ information here on this archive site.

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Non-Sampling Error - Data Adjustment (DA) and Misinterpretation Error (MI)

Data adjustment

Adjustment due to incorrect time period

In certain situations, the respondent cannot provide the information for the time period required, for example, if they were not trading for the entire time period or simply cannot break down the total for the period in a way that we might want to report it. We make the adjustment to the reported value(s) rather than the respondent making it. There is little we can do to change a reporting time period, but we can address the problem either in how we ask for the information, or how we deal with it once it has occurred. This adjustment may cause some bias if an adjustment has been made under an incorrect assumption. For example, if we assume the business behaviour is the same in the period they respond for, and in the period they do not (can not) respond to.

Currently, the adjustments are made in specific ways that are appropriate for each survey. In the monthly Retail Trade Survey a business responds as best it can for the period, but the data is adjusted to calendar months. We only adjust the data if we know they have been operating for the whole calendar month, but for some reason can only give us data for a period that is different to the calendar month. For example, the business may only be able to provide some combination of weekly sales, so each month they give us data for a four or five week period, or in other cases, the business can only give us two months worth of data every second month (since this ties in with their GST reporting). In all cases any adjustment methods to be used are subject to peer review (and possible refinement) before implementation.

top

GST adjustments

All of the data that is published by Statistics New Zealand is GST exclusive (unless otherwise stated), so we generally require responses to also be GST exclusive. Respondents sometimes still provide GST inclusive data, although often this is indicated on the form. The adjustment we make is therefore to remove the GST component from the reported value, which is a generally a simple task. However, we are directly introducing an error if an adjustment is made under an assumption that is incorrect. For example, if the value is a combination of GST exclusive and inclusive components, we may actually be adjusting an amount by too much.

Adjustments are made automatically whenever a respondent has indicated that GST is included in their data. In all other cases where there is evidence to suggest the data is not GST exclusive, the adjustment is made by staff on a case by case basis. Very little can be done in this area other than to periodically review the procedures for making adjustments to ensure they are still relevant.

top

Seasonal adjustment

The majority of economic survey time series are checked for the existence of a stable seasonal pattern by the statisticians in our Analytical Support Division. Monthly series are also checked for stable trading day patterns, and if such patterns are found they are removed. This process is termed seasonal adjustment. For more detailed information see the seasonal adjustment page on our website.

Any errors introduced due to seasonal adjustment would more strictly be classified as 'model errors', which are outside the scope of this page. Nevertheless, there is a potential to introduce non-sampling error into our results if the assumptions we make when applying the seasonal adjustment model are incorrect. This is always checked when a series is first assessed for seasonality. However, it is possible for a series to radically change its behaviour and no longer fit the seasonal model we use. This occurs very rarely, but there is always potential for unexpected phenomenon to occur, though we have procedures intended to recognise this occurrence. The other potential impact of this adjustment is that short-term economic trends can be smoothed out to some extent, although longer-term trends will be unaffected.

Currently, we control for this source of error in several ways. When the series is first assessed for seasonal adjustment there are criteria that must be met before it can adjusted and released to the public. All staff dealing with seasonal adjustment are trained in the interpretation of seasonally adjusted series and trends, as well as the understanding of the quality measures produced. Further, when a new figure is added to the series, the seasonal adjustment results are examined by user sections for inconsistencies. Any found are brought to the attention of the statisticians and the series will not be released until the inconsistency is resolved.

Our Analytical Support Division also ensures each survey time series that is seasonally adjusted is reviewed annually to determine if the quality of the seasonal adjustment has deteriorated. To ensure we continue to use current best practice in the area of seasonal adjustment, we research and investigate methods used by overseas statistical agencies for application into our environment.

top

Misinterpretation Error

Users not aware of limitations of results

If the limitations of a survey, including non-sampling error, are not understood, this can impact on the conclusions made based on the survey results. For example, if the two types of error (sampling error and non-sampling error) are not considered, the values for two survey output groups may be assumed to be different when in fact they may not be or causation may be assumed to exist between two items when in fact there could be some other reason. Not being aware of limitations can also mean users are not aware that results are estimates and not true population totals; users are not aware of assumptions made in results; and/or users are not aware of the limited scope of some results, often applying the results incorrectly to areas outside the scope of the findings.

Currently, the commentary provided by Statistics New Zealand is usually of 'statistically significant' occurrences only. Statistically significant in this case means that, having considered the limitations of the data, the results are still noteworthy. For example, that there really is a difference between the values of two groups. The sampling errors are included for some variables, otherwise they are generally available (or able to be calculated) for users if requested. Our primary means of dissemination are the website and the 'Hot Off The Press' publications. In the future, we must continue our ongoing effort to better inform users of the limitations of the survey in these and other publications. Not only must we provide the information but we must do it in a clear and concise way, giving users exactly the information they need to make informed decisions.

top

Users not provided with sufficient explanations

Statistics New Zealand needs to provide users with clear explanations of the results.

There are aspects to the results that are not errors, but simply items to consider when making conclusions. For example, if certain industry types were out of scope of the survey, how the sample was designed and selected, or how and where the data is seasonally adjusted. In addition, the presentation of survey limitations will only be useful if the limitations are clearly explained. For example, the 'Hot Off The Press' should provide users not just with measures of sampling error, but an explanation of their meaning too. Without understanding the methodology or techniques used, users could still incorrectly interpret the results.

Currently some methodology is included with our published results and the explanations include non-technical descriptions. A lot of the explanations are standard, in that they are always included, while others including some of the methodology explanations vary from survey to survey in terms of the amount of detail required and presented. While our explanations are acceptable, this source of error has a high priority as we recognise the importance of maintaining our control of this error, and also the need for continual improvement.

In the future, we will continue our ongoing effort to clearly explain the methodology and results in these and other publications. We will also periodically review the amount and content of more constant explanations, such as the methodology for surveys. We may also investigate from users whether the explanations provided are sufficient. The latter needs to be done in a sound and methodical way though, since different users require different levels of explanation.

top

Users provided with misleading graphs

Poorly graphed results can be very misleading, and we must take care to ensure that published graphs show correct trends and results, and are not confusing. Quite simply, a poor graph can lead to misinterpretation of results, and ultimately poor decision making. Currently, we have in place a set of graphics guidelines, that are used when graphs are prepared and published. This includes requirements such as that all graphs produced with broken axes are clearly labelled. The majority of graphs used in our published results are line or bar graphs, which makes it easier to set and follow standards.

This is not currently a significant source of error, but it has a high priority as we recognise the importance and need to maintain our control of this error. In the future, we will continue our ongoing effort to present graphical information to the highest possible standard.

top

Format and readability of publications

Users may have difficulty following publications if they are difficult to read or set out poorly. The true impact of this is unknown though, since it is difficult to measure. Statistics New Zealand has established standards for the publication of results, including for example, the 'Hot Off The Press' in paper form, website publications and other official publications. These ensure that we present results clearly and consistently across surveys and time periods.

As part of our ongoing quality control, we consult regularly with users to ensure our statistics continue to be presented to users in the most readable and understandable format. We also monitor standard practice in other statistical agencies to ensure our publications meet accepted international standards.  

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Top
  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+