On the pitfalls and promises of using mixed methods in literacy research: perceptions of reviewers

Mixed methodology holds considerable promise for the field of literacy research, yet it continues to be underrepresented in published research. The purpose of this study was to identify potential problems in conducting and publishing reports of mixed methods research in literacy. Predicated on a view of research as principled argument, we qualitatively analysed the content of 79 reviews of 22 manuscripts reporting mixed methods studies submitted over a four-and-a-half year period to a leading journal devoted to research in reading and literacy. We sought to identify the major pitfalls that might weaken authors’ arguments for the claims made. Our analysis revealed eight major problems in the use or reporting of mixed methods research: qualitative component underdeveloped, either in terms of data/analysis or sampling; quantitative component underdeveloped, either in terms of data/analysis or design; lack of focus; flawed logic of inquiry; weak articulation between the qualitative and quantitative components; and methodological handwringing. We argue that the ways authors sometimes use or report mixed methods research can undermine their ability to present cogent and persuasive arguments for their knowledge claims. Based on the analysis, we provide guidance for literacy scholars who choose to conduct mixed methods research.