An empirical evaluation of the adequacy of self- assessed knowledge competency in a certified population of women's health care nurse practitioners

Abstract

Background: From a multitude of recognized sources which include the PEW Commission, the Institute of Medicine, and the Citizen Advocacy Center as well as from healthcare consumers in general, there has been a persistent call for change in the processes used to demonstrate ongoing competency of health care professionals after initial licensure or certification in their specialty. In response to such long-standing demands for accountability, the Board of Directors of the National Certification Corporation (NCC), a nationally accredited, not-for-profit voluntary certification organization, approved development of a pilot project to address these concerns. The pilot project’s major objective was to determine if NCC-
certified nurses could appropriately self-assess their areas of strength and weakness related to the knowledge competencies for the active certification examination in their specialty. And further, to determine if the accuracy of their self-assessment of competency varied across subgroups for that specialty.

Methods: Study participants were selected in three separate random samples from NCC-certified Women’s Health Care Nurse Practitioners (WHNP). Common criteria used across all three sample groups included a certification maintenance cycle of 2009, stratification of the number of years of specialty experience based on NCC maintenance cycles and geographic distribution. The final sample sizes were 487, 485 and 476. Participants across all sample groups completed a survey instrument developed to obtain each individual’s self-assessment of their knowledge competency in Gynecology, Obstetrics and Primary Care as subgroups covered in their core certification examination. Once the survey tool was completed, one of three forms of a 100 item test instrument was administered. Content in each of the forms was distributed as 50% gynecology, 30% obstetrics and 20% primary care.  Sample One received a form in which all items represented entry into practice level; Sample Two was presented with items that reflected new information in the specialty field and Sample Three completed a form comprised of a combination of basic and new information.

Results: The Pearson product moment correlation coefficients were calculated among the three separate self-assessed competency ratings and the four test scores (total, gynecology, obstetrics and primary care). Psychometric consideration of the results as a whole raised concerns about the substantive interpretation of correlation coefficient as an effect size and led to summary findings that did not support self-ratings as an accurate method of knowledge assessment in this population. A mechanism to directly examine knowledge competencies was recommended. In regard to the question of constituent accuracy of assessment across specialty subgroups, participants across all samples were found to be more accurate in self-assessing their weaknesses in gynecology and obstetrics than they were in primary care.

Conclusions: Based on the overall findings of the pilot project, the NCC Board of Directors approved development of a Continuing Competency Initiative to transition certification maintenance from a constituent-assessed and selected continuing education process to a new format. As recommended by the study findings, the new Professional Development Maintenance Program is based on a third party administered evaluation tool to provide individualized direction for certification maintenance.