A Review of Methods to Combine Probability Surveys and/or Calibrate One Probability Survey with Official Federal Statistical Survey Data

Stas Kolenikov, NORC at the University of Chicago
Paul Scanlon, National Center for Health Statistics
James Dahlhamer, National Center for Health Statistics
Katherine Irimata, National Center for Health Statistics
Michael Yang, NORC at the University of Chicago

NCHS is endeavoring to better understand the viability of collecting survey data from multi-client probability- based survey panels to produce timely national estimates of population characteristics such as health outcomes and policy-relevant information on emerging topics. The NCHS Rapid Surveys System project is designed to continue the statistical advances in calibration weighting made by the Research and Development Survey (RANDS) program through combining, calibrating, or modeling probability panel data to benchmarks from NCHS flagship surveys, as well as to explore other techniques to develop weighted estimates from multi- client consumer probability panels that are fit-for-purpose for reporting health and health-related population characteristics. To commence this effort, NCHS commissioned NORC and Ipsos to prepare a detailed literature review serving two panel-relevant topic areas. This panel presentation reports on NORC’s research regarding methods to combine probability surveys, leveraging federal gold standard data to adjust other sources for estimation. We focus particularly on the situation where at least one data source is a probability-based panel.

Assessing and Improving Calibration Weighting of Web Surveys Using the R-indicator

Rong Wei, National Center for Health Statistics
Van L Parsons, National Center for Health Statistics
Yulei He, National Center for Health Statistics

Recently, commercial panel-based web surveys have been developed to complement the ability of the federal statistical system in providing health information about the U.S. population. For example, the Research and Development Survey (RANDS) is a series of primarily probability-sampled, commercial panel surveys conducted by the National Center for Health Statistics. During the COVID-19 pandemic, a special series of RANDS was used to publicly release experimental estimates on the impact of the pandemic. Despite their great potential, statistical inferences based on these web surveys might be subject to potential bias compared with NCHS’ traditional, high-quality household surveys. For example, web-based panel surveys may have additional errors due to larger nonresponse and potential coverage bias. To mitigate these biases, calibrating the weights from the web survey to a benchmark survey may be useful. We propose to use the R-indicator, originally suggested as a measure of quantifying “representativeness” of surveys with nonresponse, to assess and improve the quality of calibration weighting. In the development of target calibration weights for web surveys, this metric can be effectively used to identify possible calibration variables and compare alternative weighting strategies. Several examples are provided using RANDS surveys.

A Test of a “Federal” Approach to Increase Survey Response and Fit-for-purpose of Probability Panels for Federal Data Collection

David Dutwin, NORC at the University of Chicago
Ipek Bilgen, NORC at the University of Chicago
J. Michael Dennis, NORC at the University of Chicago

The AmeriSpeak Probability Panel was originally designed to, in part, service Federal research, given its unique (among panels) design that includes intensive nonresponse follow-up (NRFU) to recruit households to the panel via Federal Express mailings and in-person recruiting. Nevertheless, NORC at the University of Chicago has developed additional protocols, dubbed “AmeriSpeak Federal,” that include sampling with double the rate of households receiving the NRFU during panel recruitment and survey-specific protocols after the recruitment over and above the standard AmeriSpeak approach, including addition of prenotification and nonresponse reminder letters, additional call attempts, and a higher survey incentive. AmeriSpeak tested the impact of these efforts in a recent omnibus survey to assess response rate improvement, reduction in weight variation, and impact of survey estimates. We report these findings in this panel and illustrate the relative value of AmeriSpeak Federal for Federal data collection.

Combining and Standardizing Panel Surveys as Part of a Government Survey System - an Investigation

Van L Parsons, National Center for Health Statistics
Yulei He, National Center for Health Statistics
Katherine Irimata, National Center for Health Statistics
Bill Cai, National Center for Health Statistics

Commercial web-based panel surveys can be used to complement existing large scale government surveys to provide health information about U.S. populations. In particular, the National Center for Health Statistics has started utilizing commercially provided, probability-sampled, panel surveys to create auxiliary, and timely, health information systems. This new platform, the NCHS Rapid Surveys System, will consist of periodic surveys fielded by two different survey vendors. While the panel surveys are of adequate quality for many purposes, their use as part of government official statistics may require additional post-data-delivery standardization. Firstly, commonly used health estimators produced from a panel survey should reflect estimates calculated from a high-quality government reference survey. To meet this requirement, some calibration fine-tuning of the panel survey to the reference survey may be required. Secondly, multiple panel surveys may be combined to increase required sample sizes and representativeness. Estimation methods based on the combination of more than one data source must be easily automated for timely release. Our study considers some robust methods used in meta-analysis for data combination. Furthermore, any combined panel survey must still be “representative” of the reference survey. Evaluations of the proposed methods using data from the Research and Development Survey (RANDS) are presented.

Using the Household Pulse Survey to Identify Potential External Benchmarks of Economic, Social, and Health Well-Being

Priyam Patel, National Center for Health Statistics
Lauren Rossen, National Center for Health Statistics
Katherine Irimata, National Center for Health Statistics
Morgan Earp, National Center for Health Statistics

With the increased need to produce early estimates, the National Center for Health Statistics (NCHS) is turning towards the use of web-based panels for faster data collection and model based early estimation models. NCHS currently uses probability-sampled panel surveys to create auxiliary and timely, health information systems, including the Rapid Surveys System. One challenge is identifying benchmarks to improve the accuracy of NCHS estimates. This analysis examines the potential for using the Household Pulse Survey (HPS) to identify social and economic well-being indicators that not only trend with health but are also predictive of the prevalence of future health outcomes. The HPS is a web-based survey conducted by the U.S. Census Bureau, and it contains questions that were developed collaboratively with 13 federal agencies to collect data on a broad range of topics, such as childcare, employment, and housing security. We explored several different methods including regularization and conditional forests to identify indicators that are highly correlated or tend to trend with health outcomes. These findings can help guide the selection of potential predictors for use in small domain and temporal models.