On reproducibility of research findings with generalizability methods: An example from translational medicine

The question of research reproducibility and its outcomes is increasingly discussed with great apprehension, especially in biomedical research [Goodman et al., 2016]. From a statistical perspective, several methodological advances have been proposed. Kenett and Shmueli (2015), review the terminology used in this debate and refer to generalizability, as a dimension that can clarify what are research claims that should be scrutinized as reproducible. In this talk, we expand on the idea of generalizability of research findings by referring to Type S errors proposed in Gelman and Carlin (2014). The talk will first discuss methods for setting up a boundary of meaning (BOM) and then show how Type S errors fit with this generalizability approach. An example from the discipline of targeted drug delivery, namely of specific identification of colon cancer, will be used to demonstrate the approach. The technology comprises a composite fluorescent polymeric system developed by Bloch et al (2015) for targeting cancerous tissues in the colon for improved diagnostics during invasive medical procedures. The composite vehicle was composed of fluorescent polymer, decorated with a recognition peptide, encapsulated in echogenic microbubbles. The biological effect (affinity to cancer cells or malignant colon epithelium from a rat model for colon cancer, as quantified by fluorescence intensity at the target) is screened by statistically designed experiments. The Design of Experiment (DoE) approach is used to identify possible intrusive interactions among the various variables. DoE is the basic approach used in quality by design (QbD) methods to map design spaces. The study design here consisted of 44 experimental factor level combinations in a mostly balanced array of 4 factors, two at 2 levels and three at 3 levels.

Bloch, M., Kenett, R.S., Jablonowski, L., Wheatley, M., Yavin, E. and Rubinstein, A. (2015), Polym Adv Technol 26, pp. 898-905.

Gelman, A. and Carlin, J. (2014), Perspectives on Psychological Science, Vol. 9(6), pp. 641–651.

Goodman, S., Fanelli, D. and Ioannidis, J. (2016), , 8(241), pp. 341-347

Kenett, R.S. and Shmueli, G. (2015), Nature Methods, 12(8), p 699.