You Can’T Publish Replication Studies (And How To Anyways) : Position Paper

Reproducibility has been increasingly encouraged by communities of science in order to validate experimental conclusions, and replication studies represent a significant opportunity to vision scientists wishing contribute new perceptual models, methods, or insights to the visualization community. Unfortunately, the notion of replication of previous studies does not lend itself to how we communicate research findings. Simple put, studies that re-conduct and confirm earlier results do not hold any novelty, a key element to the modern research publication system. Nevertheless, savvy researchers have discovered ways to produce replication studies by embedding them into other sufficiently novel studies. In this position paper, we define three methods — re-evaluation, expansion, and specialization — for embedding a replication study into a novel published work. Within this context, we provide a non-exhaustive case study on replications of Cleveland and McGill’s seminal work on graphical perception. As it turns out, numerous replication studies have been carried out based on that work, which have both confirmed prior findings and shined new light on our understanding of human perception. Finally, we discuss how publishing a true replication study should be avoided, while providing suggestions for how vision scientists and others can still use replication studies as a vehicle to producing visualization research publications.

G. Quadri, and Paul Rosen. You Can’T Publish Replication Studies (And How To Anyways) : Position Paper. Vis X Vision Workshop at IEEE VIS, 2019.

@article{quadri2019you,
  title = {You Can’t Publish Replication Studies (and How to Anyways) : Position Paper},
  author = {Quadri, G. and Rosen, Paul},
  journal = {Vis X Vision Workshop at IEEE VIS},
  year = {2019},
  abstract = {Reproducibility has been increasingly encouraged by communities of science
    in order to validate experimental conclusions, and replication studies represent a
    significant opportunity to vision scientists wishing contribute new perceptual models,
    methods, or insights to the visualization community. Unfortunately, the notion of
    replication of previous studies does not lend itself to how we communicate research
    findings. Simple put, studies that re-conduct and confirm earlier results do not hold
    any novelty, a key element to the modern research publication system. Nevertheless,
    savvy researchers have discovered ways to produce replication studies by embedding them
    into other sufficiently novel studies. In this position paper, we define three methods
    -- re-evaluation, expansion, and specialization -- for embedding a replication study
    into a novel published work. Within this context, we provide a non-exhaustive case study
    on replications of Cleveland and McGill's seminal work on graphical perception. As it
    turns out, numerous replication studies have been carried out based on that work, which
    have both confirmed prior findings and shined new light on our understanding of human
    perception. Finally, we discuss how publishing a true replication study should be
    avoided, while providing suggestions for how vision scientists and others can still use
    replication studies as a vehicle to producing visualization research publications.}
}