by Elizabeth Hofheinz, M.P.H., M.Ed.
A uniform residency training curriculum for the entire United States…it’s time, say the American Academy of Orthopaedic Surgeons (AAOS), the American Board of Orthopaedic Surgery (ABOS), and the American Orthopedic Association/Council of Orthopedic Residency Directors (AOA/CORD).
The ABOS was founded in 1934 to establish “educational standards for orthopaedic residents and by evaluating the initial and continuing qualifications and competence of orthopaedic surgeons.”1
Formed in 1982, the Accreditation Council for Graduate Medical Education (ACGME) later defined six core competencies for residents—medical knowledge, patient care, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice.2
Before Covid-19 struck, says recent research, orthopedic residents and fellows were primarily using digital content for their education in the form of electronic textbooks, journal articles, online presentations, and surgical videos on a variety of websites.3
And now that digital learning is ubiquitous, it is time to put a fine point on resident competency.
Ann Van Heest, M.D. is chair of the GME committee at the ABOS. She told OSN, “During Covid-19 the number of surgical cases available to residents has been vastly curtailed. This pandemic has highlighted the issue of resident competency, with the fundamental question being, ‘How many cases does a resident need in order to be proficient with a given procedure?’”
The answer, says Dr. Van Heest, is that it depends on the person and it depends on the procedure. To minimize variability, Dr. Van Heest and her colleagues have divided competency into three domains: knowledge, skills and behavior.
Assessing knowledge
The cornerstone of the knowledge domain is a blueprint that has been crafted by the AAOS, the AOA, and CORD. “We are working with Dr. Paul Tornetta,” said Dr. Van Heest, “who heads up the AAOS residency curriculum. Together, we are trying to make the Orthopaedic In-Training Exam (OITE) and residency curriculum that AAOS is developing in line with the ABOS Part I examination.”
The plan, says Dr. Van Heest, is for the OITE to follow the ABOS blueprint, thus linking the two exams so as to achieve greater continuity and organization. “Some of the questions on the OITE are also on the ABOS Part I exam, so there is some question sharing. We are trying to coordinate subareas as well; for example, if spine is 10% of Part I, then it will be 10% of the OITE.”
Assessing skills
Traditionally, residents’ skill levels were determined by the program directors’ opinion, says Dr. Van Heest. “What we have always lacked is real-time concrete evidence of a resident’s surgical competency for independent practice.”
“Several of my colleagues and I published an article last year in JBJS where the ABOS collaborated with CORD to study the use of two web-based surgical skills assessment tools by 294 residents at 16 residency programs. The programs, selected by the CORD, were representative of varying program sizes, geographic locations, and degrees of academic affiliation.”
“One of the tools was an O-score, a validated assessment of surgical skills that can be used for performance in any procedure. It is comprised of a 9-question evaluation of 8 steps of the surgical procedure using a 5-point scale. The other tool was a P-score, a single-question evaluation using a 5-level assessment. Our hypothesis was that both scores would provide similar results when used by faculty to evaluate residents’ surgical competency.”
Residents were asked to electronically request an evaluation from attendings when performing any of the 25 procedures assessed in the study. The attending rated the residents on their level of competency during the procedure and those numbers went up on resident’s dashboard. Dr. Van Heest: “We found that both scoring tools reflected a progression of competency and have thus decided to combine the two and make them available to all residency programs for real-time web-based faculty assessments of resident performance.”
The real-time nature of use of the OP tool is particularly helpful, says Dr. Van Heest, as it allows for rapid course correction. “If someone is not making progress then faculty members are alerted so they can determine if spatial or other issues are cropping up. In the event that someone needs remedial practice then that person can take advantage of a simulation lab.”
Assessing behavior
“In a 2019 AOA pilot study, we looked at five different areas of behavioral skills and found that we’re basically looking for extreme outliers. The majority of residents undergo so much screening during their undergraduate years, medical school, and residency that it is unlikely that they will make it through with behavioral issues.”
Indeed, says Dr. Van Heest, nearly 98% of resident evaluations indicate no behavioral issues, while 2-3% have deficiencies in certain areas such as ethics, communication, interactions and teamwork, reliability and work ethic, and lifelong learning. “This 2-3% does correlate to individuals whom the program director had already identified as having behavioral issues, so it is externally validated.”
It is also helpful that the ABOS and the CORD are developing a behavioral assessment tool. “No matter where someone trains in the U.S., with this tool, everyone will be on the same page as far as what is acceptable behavior,” says Dr. Van Heest.
Paul Tornetta, III, M.D. is Chair of the Council on Education for the AAOS. He told OSN, “For many years some of us have tried to facilitate the adoption of a uniform orthopaedic resident curriculum. The current climate, helped by increasingly positive relationships between the ABOS and AAOS, is ideal for this to move forward. Fundamentally, we are all focused on having the residency curriculum reflect the knowledge base that ABOS is testing.”
The only entities with the potential to do this in a truly unified fashion are the AAOS and the ABOS, says Dr. Tornetta. “Along with other AAOS members such as Steve Haddad, M.D., I was invited to take part in the revision of the ABOS Part I blueprint. This document will be soon be divided into subspecialty regions, and that will inform the blueprint for the OITE. That way, we are testing the same concepts and things that the ABOS is testing. We are developing the residency curriculum around the blueprint.”
Dr. Tornetta: “It is important to recognize the vital role of CORD as they have diligently worked to identify areas of need as well as solutions. We developed concepts around the logistics of how the curriculum will work, using critical input from CORD (how program directors should assign topics, etc.) as well as from the AAOS Resident Assembly. It has been a monumental team effort.”
In the future, competent in Seattle will mean competent in Nashville…and that’s an idea everyone can get behind.
References
