Fitting z-curves to estimate the size of the UESD file drawer and the replicability of published findings

FRESE, Joris

doi:10.1177/20531680241277634

Fitting z-curves to estimate the size of the UESD file drawer and the replicability of published findings

dc.contributor.author	FRESE, Joris
dc.date.accessioned	2024-09-05T10:09:58Z
dc.date.available	2024-09-05T10:09:58Z
dc.date.issued	2024
dc.description	Published online: 28 August 2024	en
dc.description.abstract	The unexpected event during survey design (UESD) established itself as a viable causal inference design across multiple social science disciplines in the past few years. The distribution of UESD test statistics has not yet been scrutinized for potential anomalies to the same degree as those from other causal inference methods, such as DiD, RDD, or IV. In this article, I leverage recent advances in meta-analytical methodology to estimate the replicability of statistically significant UESD findings and quantify the size of the file drawer of non-significant findings. Precisely, I aggregate 1095 ITT coefficients and standard errors from UESD studies published between 2019 and 2023 to fit their z-curve and estimate their observed discovery rate, expected discovery rate, and expected replication rate. While most statistically significant UESD findings are predicted to be replicable, the distribution of z-values also indicates publication bias toward marginally significant findings and a large file drawer of non-significant findings. The innovative z-curve methodology, which has not seen much (or any) use in political science yet, provides promising new insights beyond established tools for the assessment of publication bias, such as funnel plots or caliper tests, and can readily be applied to entire subfields of quantitative political science.	en
dc.format.mimetype	application/pdf	en
dc.identifier.citation	Research & politics, 2024, Vol. 11, No. 3, OnlineOnly	en
dc.identifier.doi	10.1177/20531680241277634
dc.identifier.issn	2053-1680
dc.identifier.issue	3	en
dc.identifier.uri	https://hdl.handle.net/1814/77191
dc.identifier.volume	11	en
dc.language.iso	en	en
dc.orcid.putcode	1814/80491:166865351
dc.orcid.upload	true	*
dc.publisher	Sage	en
dc.relation.ispartof	Research & politics	en
dc.rights	info:eu-repo/semantics/openAccess	en
dc.rights.license	Attribution-NonCommercial 4.0 International	*
dc.rights.uri	http://creativecommons.org/licenses/by-nc/4.0/	*
dc.title	Fitting z-curves to estimate the size of the UESD file drawer and the replicability of published findings	en
dc.type	Article	en
dspace.entity.type	Publication
eui.subscribe.skip	true
person.identifier.orcid	0000-0002-5871-997X
person.identifier.other	47874
relation.isAuthorOfPublication	e544c3f3-6684-4bbd-9fc6-565eaaa069a6
relation.isAuthorOfPublication.latestForDiscovery	e544c3f3-6684-4bbd-9fc6-565eaaa069a6