A Johns Hopkins study shows that the abilities of orthopedic surgeons in training are more accurately ascertained when detailed checklists are coupled with rigorous error tracking.
Tracking surgical trainees' performance on cadavers can be greatly enhanced by coupling step-by-step checklists and measures of general surgical skills works with a rigorous system for tracking errors, according to researchers at Johns Hopkins University.
"The takeaway message is that checklists of procedural steps are a good way to assess the technical skills of these surgical residents. But they don't measure quality, highlighting a need to measure and give feedback on errors as part of the training," says Dawn LaPorte, MD, professor of orthopedic surgery at the Johns Hopkins University School of Medicine and an author of a report on the study in the Journal of Surgical Education.
Most training models rely on case numbers that serve as a proxy for a resident's mastery of a procedure. In reality, that only measures volume, not skill level, she says, adding that learning opportunities have been reduced with caps on resident work hours that are designed to reduce fatigue-related errors.
Residents rarely get formal feedback on their motor skills or errors under the current system, LaPorte says, which allows mistakes to perpetuate.
Evaluating residents as they work on cadavers identifies areas in need of improvement and allows them to learn from mistakes. "They can practice repeatedly in a safe environment before they're holding the knife to operate on real people," LaPorte says.
Johns Hopkins researchers created the study as a means to address shortcomings in the assessment of motor skills by using the Objective Structured Assessment of Technical Skills checklist that was modified for surgical procedures of the shoulder.
The researchers tested the checklist in residents practicing three different approaches to accessing a shoulder in need of repair: from the front, back, or side. The procedures are common first steps for shoulder surgeries to address injuries to the bicep, shoulder bones, or rotator cuffs.
Three orthopedic specialists from Johns Hopkins used the grading systems to evaluate the work of 23 Johns Hopkins medical residents ranging from their first to fifth year of residency as they performed each of the procedures on cadavers.
Under OSATS, the residents received a point for the successful completion of each step in the checklist and a zero for failed or incomplete steps. They also received zeros if they performed the steps out of order. The crucial steps in this case involved making the incision in the right place and avoiding important structures, such as nerves and arteries.
Besides the OSATS checklists, the faculty surgeons rated residents using the so-called Global Rating Scale and a simple pass/fail system.
The Global Rating Scale was developed by Richard Reznick in 2006 and is applicable to all surgical procedures, taking a holistic view of the surgical process and offering residents feedback on such things such as whether or not they used the surgical instruments correctly and if they display a broad understanding of the procedure.
Under the pass/fail system, residents receive a failing grade if they commit an egregious error — in the case of shoulder surgery, severing a nerve or blood vessel.
The Johns Hopkins team found that OSATS and the Global Rating Scale provided good, objective ways of measuring resident performance, while the pass/fail system gave residents unambiguous feedback.
More advanced residents received higher OSATS and Global Rating Scale scores than those just beginning their residencies. Scores on all three procedures ranged from the low single to double digits for first-year residents to the high double digits for fifth-year students.
Across all three procedures, examiners observed 11 incidents in which residents damaged the nerves or veins. First- and second-year residents were responsible for nine of those mistakes.
The researchers found that none of the three evaluations adequately captured those mistakes. The pass/fail scheme came closest, in that residents received a failing score for severing a nerve or major blood vessel, but it is not set up to subsequently inform residents of the precise nature of their mistakes.
And residents could theoretically perform well on the OSATS checklist even if they made an egregious error because points are not deducted, only earned in the grading system. To mitigate this, the researchers propose adding safety steps, such as identifying and protecting important structures, to the checklist.
LaPorte wants to implement these training protocols for other common surgical procedures.
Next up is developing an OSATS checklist for treating compartment syndrome, an injury in which bleeding or swelling prevents blood from flowing to affected tissues.
The residents under review said they like an objective evaluation system like the checklist because it gives them the confidence they need before entering an actual operating theater. "The goal is to make objective evaluations standard for all procedures expected of residents," LaPorte says.