Testing an objective structured assessment technical skills tool: A pilot study

Abstract

Purpose: The purpose of this prospective pilot study was to test the feasibility, reliability and validity of an adapted version of an objective structured assessment technical skills (OSATS) tool using a criterion-reference rating scale while evaluating student registered nurse anesthetists (SRNAs) performing mask ventilation, laryngeal mask airway (LMA) insertion, and endotracheal intubation.

Methods: Prior to testing, three OSATS evaluation tools were developed with nurse anesthesia faculty input. Six faculty-evaluators pilot tested an adapted version of three OSATS tools for feasibility and validity with 23 SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation with an airway management trainer. SRNAs had 15 minutes; five-minute each to perform mask ventilation, insert an LMA, and perform endotracheal intubation. Paired observations from six faculty-evaluators determined inter-rater reliability. Separate survey questionnaires were created for faculty-evaluators and SRNAs; using three four-point Likert-type rating scales to measure feasibility and validity.

Results: Descriptive statistics revealed that faculty-evaluators’ found the OSATs tools to be feasible based on six out of seven somewhat favorable to strongly favorable responses. The Kendall’s Coefficient of Concordance W showed statistical significance for mask ventilation and endotracheal intubation (p-value of .041 and .036). Six faculty-evaluators found the OSATs tools to have face and content validity based on six out of seven somewhat favorable responses. Twenty-three SRNAs found the OSATS tools to have face validity based on somewhat strongly favorable responses.

Conclusions: An adapted version of OSATS tool for evaluating SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation demonstrated feasibility, reliability, and validity. The inter-rater values obtained from Kendall’s Coefficient of Concordance W in this project correlated with the Cook and Beckman report.