Handbook of Statistical Data Editing and Imputation (Wiley Handbooks in Survey Methodology) - Hardcover

De Waal, Ton; Pannekoek, Jeroen; Scholtus, Sander

Hardcover

ISBN 10: 0470542802 ISBN 13: 9780470542804

Verlag: Wiley, 2011

Inhaltsangabe

A practical, one-stop reference on the theory and applications of statistical data editing and imputation techniques

Collected survey data are vulnerable to error. In particular, the data collection stage is a potential source of errors and missing values. As a result, the important role of statistical data editing, and the amount of resources involved, has motivated considerable research efforts to enhance the efficiency and effectiveness of this process. Handbook of Statistical Data Editing and Imputation equips readers with the essential statistical procedures for detecting and correcting inconsistencies and filling in missing values with estimates. The authors supply an easily accessible treatment of the existing methodology in this field, featuring an overview of common errors encountered in practice and techniques for resolving these issues.

The book begins with an overview of methods and strategies for statistical data editing and imputation. Subsequent chapters provide detailed treatment of the central theoretical methods and modern applications, with topics of coverage including:

• Localization of errors in continuous data, with an outline of selective editing strategies, automatic editing for systematic and random errors, and other relevant state-of-the-art methods

• Extensions of automatic editing to categorical data and integer data

• The basic framework for imputation, with a breakdown of key methods and models and a comparison of imputation with the weighting approach to correct for missing values

• More advanced imputation methods, including imputation under edit restraints

Throughout the book, the treatment of each topic is presented in a uniform fashion. Following an introduction, each chapter presents the key theories and formulas underlying the topic and then illustrates common applications. The discussion concludes with a summary of the main concepts and a real-world example that incorporates realistic data along with professional insight into common challenges and best practices.

Handbook of Statistical Data Editing and Imputation is an essential reference for survey researchers working in the fields of business, economics, government, and the social sciences who gather, analyze, and draw results from data. It is also a suitable supplement for courses on survey methods at the upper-undergraduate and graduate levels.

Über diesen Titel

Über die Autorin bzw. den Autor

Ton De Waal, PhD, is Head of the Department of Methodology at Statistics Netherlands, where he has also worked at the Division of Business Statistics. Dr. de Waal has written numerous papers in his areas of research interest, which include statistical data editing and imputation for business surveys and statistical disclosure control.

Jeroen Pannekoek, PhD, is Senior Researcher in the Department of Methodology at Statistics Netherlands, where he currently leads the research program on data processing methodologies. He has published several papers on discrete data models, measurement errors, interviewer effects, and disclosure control methods.

Sander Scholtus, MSc, is Researcher in the Department of Methodology at Statistics Netherlands. He has conducted extensive research on heuristic methods and algorithms for detecting and correcting errors in survey data.

Collected survey data are vulnerable to error. In particular, the data collection stage is a potential source of errors and missing values. As a result, the important role of statistical data editing, and the amount of resources involved, has motivated considerable research efforts to enhance the efficiency and effectiveness of this process. Handbook of Statistical Data Editing and Imputation equips readers with the essential statistical procedures for detecting and correcting inconsistencies and filling in missing values with estimates. The authors supply an easily accessible treatment of the existing methodology in this field, featuring an overview of common errors encountered in practice and techniques for resolving these issues.

The book begins with an overview of methods and strategies for statistical data editing and imputation. Subsequent chapters provide detailed treatment of the central theoretical methods and modern applications, with topics of coverage including:

• Localization of errors in continuous data, with an outline of selective editing strategies, automatic editing for systematic and random errors, and other relevant state-of-the-art methods

• Extensions of automatic editing to categorical data and integer data

• The basic framework for imputation, with a breakdown of key methods and models and a comparison of imputation with the weighting approach to correct for missing values

• More advanced imputation methods, including imputation under edit restraints

Throughout the book, the treatment of each topic is presented in a uniform fashion. Following an introduction, each chapter presents the key theories and formulas underlying the topic and then illustrates common applications. The discussion concludes with a summary of the main concepts and a real-world example that incorporates realistic data along with professional insight into common challenges and best practices.

Handbook of Statistical Data Editing and Imputation is an essential reference for survey researchers working in the fields of business, economics, government, and the social sciences who gather, analyze, and draw results from data. It is also a suitable supplement for courses on survey methods at the upper-undergraduate and graduate levels.

Collected survey data are vulnerable to error. In particular, the data collection stage is a potential source of errors and missing values. As a result, the important role of statistical data editing, and the amount of resources involved, has motivated considerable research efforts to enhance the efficiency and effectiveness of this process. Handbook of Statistical Data Editing and Imputation equips readers with the essential statistical procedures for detecting and correcting inconsistencies and filling in missing values with estimates. The authors supply an easily accessible treatment of the existing methodology in this field, featuring an overview of common errors encountered in practice and techniques for resolving these issues.

The book begins with an overview of methods and strategies for statistical data editing and imputation. Subsequent chapters provide detailed treatment of the central theoretical methods and modern applications, with topics of coverage including:

• Localization of errors in continuous data, with an outline of selective editing strategies, automatic editing for systematic and random errors, and other relevant state-of-the-art methods

• Extensions of automatic editing to categorical data and integer data

• The basic framework for imputation, with a breakdown of key methods and models and a comparison of imputation with the weighting approach to correct for missing values

• More advanced imputation methods, including imputation under edit restraints

Throughout the book, the treatment of each topic is presented in a uniform fashion. Following an introduction, each chapter presents the key theories and formulas underlying the topic and then illustrates common applications. The discussion concludes with a summary of the main concepts and a real-world example that incorporates realistic data along with professional insight into common challenges and best practices.

Handbook of Statistical Data Editing and Imputation is an essential reference for survey researchers working in the fields of business, economics, government, and the social sciences who gather, analyze, and draw results from data. It is also a suitable supplement for courses on survey methods at the upper-undergraduate and graduate levels.

Auszug. © Genehmigter Nachdruck. Alle Rechte vorbehalten.

Handbook of Statistical Data Editing and Imputation

By Ton de Waal Jeroen Pannekoek Sander Scholtus

John Wiley & Sons

Copyright © 2011 John Wiley & Sons, Inc
All right reserved.
ISBN: 978-0-470-54280-4

Chapter One

Introduction to Statistical Data Editing and Imputation

1.1 Introduction

It is the task of National Statistical Institutes (NSIs) and other official statistical institutes to provide high-quality statistical information on many aspects of society, as up-to-date and as accurately as possible. One of the difficulties in performing this task arises from the fact that the data sources that are used for the production of statistical output, both traditional surveys as well as administrative data, inevitably contain errors that may influence the estimates of publication figures. In order to prevent substantial bias and inconsistencies in publication figures, NSIs therefore carry out an extensive process of checking the collected data and correcting them if necessary. This process of improving the data quality by detecting and correcting errors encompasses a variety of procedures, both manual and automatic, that are referred to as statistical data editing. The effects of statistical data editing on the errors have been examined since the mid-1950s [see Nordbotten (1955)].

Besides errors in the data, another complicating factor in order to fulfill the task of NSIs and other statistical institutes successfully is that data are often missing. This can be seen as a simple form of erroneous data, simple in the sense that missing values are easy to identify; estimating good values for these missing values may, however, be hard.

Errors and missing data can arise during the measurement process. Errors arise during the measurement process when the reported values differ from the true values. A possible reason for a measurement error can be that the true value is unknown to the respondent or difficult to obtain. Another reason could be that questions are misinterpreted or misread by the respondent. An example is the so-called unity measurement error that occurs if the respondent reports in euros when it was required to report in thousands of euros. Another example is a respondent reporting his own income when asked for the household income. For business surveys, errors also occur due to differences in definitions used by the statistical office and the accounting system of the responding unit. There may, for instance, be differences in the reference period used by the business and the requested period (financial year versus calender year is an example). After the data have been collected, they will pass through several other processes, such as keying, coding, editing, and imputation. Errors that arise during this further processing are referred to as processing errors. Note that although the purpose of editing is to correct errors, it is mentioned here also as a process that may occasionally introduce errors. This undesirable situation arises if an item value is adjusted because it appeared to be in error but it is actually correct. Missing data can arise when a respondent does not know the answer to a question or refuses to give the answer to a certain question.

Traditionally, NSIs have always put a lot of effort and resources into statistical data editing, because they considered it a prerequisite for publishing accurate statistics. In traditional survey processing, statistical data editing was mainly an interactive activity intended to correct all data in every detail. Detected errors or inconsistencies were reported and explained on a computer screen and corrected after consulting the questionnaire or contacting respondents, which are time- and labor-intensive procedures. In this book we examine more efficient statistical data editing methods.

It has long been recognized that it is not necessary to correct all data in every detail. Several studies [see, for example, Granquist (1984, 1997) and Granquist and Kovar (1997)] have shown that in general it is not necessary to remove all errors from a data set in order to obtain reliable publication figures. The main products of statistical offices are tables containing aggregate data, which are often based on samples of the population. This implies that small errors in individual records are acceptable. First, because small errors in individual records tend to cancel out when aggregated. Second, because if the data are obtained from a sample of the population, there will always be a sampling error in the published figures, even when all collected data are completely correct. In this case an error in the results caused by incorrect data is acceptable as long as it is small in comparison to the sampling error. In order to obtain data of sufficiently high quality, it is usually enough to remove only the most influential errors. The above-mentioned studies have been confirmed by many years of practical experience at several statistical offices.

In the past, and often even in the present, too much effort was spent on correcting errors that did not have a noticeable impact on the ultimately published figures. This has been referred to as "over-editing." Over-editing not only costs money, but also takes a considerable amount of time, making the period between data collection and publication unnecessarily long. Sometimes over-editing even becomes "creative editing"; the editing process is then continued for such a length of time that unlikely, but correct, data are "corrected." Such unjustified alterations can be detrimental for data quality. For more about the danger of over-editing and creative editing, see, for example, Granquist (1995, 1997) and Granquist and Kovar (1997).

It has been argued that the role of statistical data editing should be broader than only error localization and correction. Granquist (1995) identifies the following main objectives:

1. Identify error sources in order to provide feedback on the entire survey process.

2. Provide information about the quality of the incoming and outgoing data.

3. Identify and treat influential errors and outliers in individual data.

4. When needed, provide complete and consistent individual data.

During the last few years, the first two goals—providing feedback on the other survey phases, such as the data collection phase, and providing information on the quality of the final results—have gained in importance. The feedback on other survey phases can be used to improve those phases and reduce the amount of errors arising in these phases. In the next few years the first two goals of data editing are likely to become even more important. The main focus in this book is, however, on the latter, more traditional, two goals of statistical data editing. Statistical data editing is examined in Chapters 2 to 6.

Missing data is a well-known problem that has to be faced by basically all institutes that collect data on persons or enterprises. In the statistical literature, ample attention is hence paid to missing data. The most common solution to handle missing data in data sets is imputation, where missing values are estimated and filled in. An important problem of imputation is to preserve the statistical distribution of the data set. This is a complicated problem, especially for high-dimensional data. Chapters 7 and 8 examine this aspect of imputation.

At NSIs the imputation problem is further complicated owing to the existence of constraints in the form of edit restrictions, or edits for short, that have to be satisfied by the data. Examples of such edits are...

Details zur Ausgabe

• Verlag: Wiley
• Erscheinungsdatum: 2011
• Sprache: Englisch
• ISBN 10: 0470542802
• ISBN 13: 9780470542804
• Einband: Gebundene Ausgabe
• Auflage: 1
• Anzahl der Seiten: 464
• Kontakt zum Hersteller: Wiley-VCH GmbH; product_safety@wiley.com; ; Boschstrasse 12; Weinheim; 69469; Deutschland