Master - Lab - Status Quo. How is effect size reported in CHI publications?
Supervisor: Anna-Marie (ortloff@cs.uni-bonn.de)
In quantitative analysis, researchers investigate the relationship between one or more independent, or predictor variables, and one or more dependent or outcome variable(s) [1]. Effect sizes measure the strength of the relationship of independent variables with dependent variables [1] or, when comparing two or more groups, the size of the difference between the groups [2]. However, effect sizes are often not or not sufficiently reported in scientific publications [3,4]. In this lab, you should determine the status quo of effect size reporting in CHI publications.
Your tasks
- Choose a sample of quantitative CHI-Papers, i.e. papers with hypothesis tests in them. Sampling possibilities could be a random sample, or only papers with an award.
- For each hypothesis test in the paper, determine and note:
- Was an effect size reported (Which type? Standardized or unstandardized?)
- Was a judgment of the size of the effect size reported (e.g. small, medium, large…)
- Was the effect size interpreted further?
- Use qualitative analysis methods to identify strategies for effect size reporting and interpretation.
This lab works well for a group. Each group member would analyze their own papers.
Literature to start with:
-
[1] Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge university press.
-
[2] Coe, R. (2002, September). It’s the effect size, stupid, https://f.hubspotusercontent30.net/hubfs/5191137/attachments/ebe/ESguide.pdf
-
[3] Groß, T. (2021). Fidelity of statistical reporting in 10 years of cyber security user studies, https://arxiv.org/pdf/2004.06672.pdf
-
[4] Ortloff, A. M., Tiefenau, C., & Smith, M. (2023). SoK: I Have the (Developer) Power! Sample Size Estimation for Fisher’s Exact,Chi-Squared,McNemar’s, Wilcoxon Rank-Sum, Wilcoxon Signed-Rank and t-tests in Developer-Centered Usable Security, https://www.usenix.org/system/files/soups2023-ortloff.pdf
Papers to analyze:
- CHI ‘23 proceedings: https://dl.acm.org/doi/proceedings/10.1145/3544548 (not all may be available in open access)
Requirements:
It is helpful to have some basic knowledge of inferential statistical testing / null hypothesis statistical testing. English slides and German videos for an introduction, as well as an introduction to qualitative analysis can be provided.