Assessing the Extent that the Gender and STEM Practice- Oriented Literature is Evidence-Based

Abstract

One principal way to judge if the activity or program described in a practice-oriented publication is worth replicating is to consider the strength of its foundational component. A strong foundational component supports the credibility of an initiative by demonstrating that it was designed with an understanding of other similar initiatives and with evidence that there was a reasonable promise that the desired outcomes were realistically achievable. This paper reports on the findings of a mixed methods content analysis of 142 practice-oriented articles published in three different refereed venues between 1995 and 2009. A scale to measure the quality of the foundational component of the practice-oriented publications was used in the quantitative phases to test if there had been a change in the relative number and quality of practice-oriented publications over time. The validity of the scale was tested in subsequent qualitative phases of the project by an inductive examination of the characteristics of top-scoring articles. Results of the quantitative analysis indicated that the average quality rating of the foundational element of the articles was relatively low (3.37 out of a possible 6 points). The scale proved effective in distinguishing articles that were exemplary in other ways as well. The low overall quality score on the foundational component of practice-oriented publications either means that the authors of these publications are failing to provide ample documentation that these initiatives were evidence-based or the content analysis reveals a long-standing trend toward the adoption of non-evidence based practices.