Open Access
Fitting z-curves to estimate the size of the UESD file drawer and the replicability of published findings
Loading...
Files
Fitting_zcurves_2024.pdf (1.16 MB)
Full-text in Open Access, Published version
License
Attribution-NonCommercial 4.0 International
Cadmus Permanent Link
Full-text via DOI
ISBN
ISSN
2053-1680
Issue Date
Type of Publication
Keyword(s)
LC Subject Heading
Other Topic(s)
EUI Research Cluster(s)
Initial version
Published version
Succeeding version
Preceding version
Published version part
Earlier different version
Initial format
Author(s)
Citation
Research & politics, 2024, Vol. 11, No. 3, OnlineOnly
Cite
FRESE, Joris, Fitting z-curves to estimate the size of the UESD file drawer and the replicability of published findings, Research & politics, 2024, Vol. 11, No. 3, OnlineOnly - https://hdl.handle.net/1814/77191
Abstract
The unexpected event during survey design (UESD) established itself as a viable causal inference design across multiple social science disciplines in the past few years. The distribution of UESD test statistics has not yet been scrutinized for potential anomalies to the same degree as those from other causal inference methods, such as DiD, RDD, or IV. In this article, I leverage recent advances in meta-analytical methodology to estimate the replicability of statistically significant UESD findings and quantify the size of the file drawer of non-significant findings. Precisely, I aggregate 1095 ITT coefficients and standard errors from UESD studies published between 2019 and 2023 to fit their z-curve and estimate their observed discovery rate, expected discovery rate, and expected replication rate. While most statistically significant UESD findings are predicted to be replicable, the distribution of z-values also indicates publication bias toward marginally significant findings and a large file drawer of non-significant findings. The innovative z-curve methodology, which has not seen much (or any) use in political science yet, provides promising new insights beyond established tools for the assessment of publication bias, such as funnel plots or caliper tests, and can readily be applied to entire subfields of quantitative political science.
Table of Contents
Additional Information
Published online: 28 August 2024