Quality Measurement Needs Quality Improvement: A National Leader’s Thoughts on Moving the Needle

by Elizabeth Hofheinz, M.P.H., M.Ed., January 17, 2020

It is most likely that everyone in the medical field is on board with measuring quality. But what, alas, should be measured?

One physician in Chicago has reflected profoundly on this question, so much so that he was asked to advise CMS on the matter. Bala Hota, M.D. is the Chief Analytics Officer at Rush University Medical Center and is a national leader in the field of quality measurement. Dr. Hota, also the Associate Chief Medical Officer and Associate Chief Information Officer at Rush, told OSN, “The jury is out as to what extent our current quality measures are valuable. My team, which includes Rush CEO Omar Lateef, D.O., has worked with U.S. News & World Report and CMS on this subject. When you closely examine the major quality ratings groups, it is evident that we are not ‘there’ yet in terms of accurately measuring quality. We are getting closer but there are considerable gaps in our knowledge.”

“Patients generally trust that the healthcare system they enter is one of high quality…one where they will receive the correct diagnosis, be treated well, and have the appropriate care delivered. But it’s unclear if the U.S. News & World Report or the Medicare Star Rating systems are on point.”

Thus, they took to the data.

“In 2017 we published a paper showing that the U.S. News & World Report best hospital rating system had been flawed for the past seven years. And because we are among the first researchers to get our own data back (U.S. News & World Report purchases data from Medicare, combines it, then run algorithms) we attempted to reproduce what they had produced. A surprise awaited us, however, as our estimates of, for example, being above average on safety, were vastly different than the estimates put forth by U.S. News & World Report. This greatly impacted the orthopedics group and it ended up ranked much lower because of overall hospital safety measures.”

“About one-third of the U.S. News & World Report is specialty-specific,” states Dr. Hota. “Another third is based on reputation (a survey where doctors rate other doctors, which has its own flaws). The remaining third are general hospital-based ratings that are applied to specialists.”

To play the game, know the rules…

“It is important for hospitals and providers to understand the rules of their system as each system has a set of rules as to how you decide what quality looks like. They often are technical and have quirks that will affect how you perform in the rules. For example, most rules evolve from billing data so the codes you use establish the rates you as providers and hospitals have to work with. That rating might be something obvious such as deaths or something nuanced like bad outcomes following a procedure, infections post procedure or patients with excessive bleeding after a procedure.”

Apples to apples, please…

“Most of these measures are adjusted for how sick your patients are, i.e., risk adjustment. And we know that there is true quality and that there are providers and hospitals that provide better care than others. However, there is also variability in the accuracy and completeness of what is documented. Many times, the difference in ratings is due to variation in documentation – what a doctor wrote – and not the care given, so you are not getting an apples to apples comparison.”

But over the last ten years, says Dr. Hota, things have improved incrementally, with increasingly sophisticated ratings systems.

“Quality measures are useful in determining areas for improvement, so we take data and use it at the Rush Center for Quality, Safety, and Value analytics. The more hospitals are data-driven in performance improvement the more they can actually improve over time by comparing their performance year after year.”

“There are over 40 measures that are potentially usable that get combined into the Star ratings. In practice, however, it is only four or five measures that drive most of the scoring. The first is hospital mortality…heart attack and stroke for example. The second is the Patient Safety Indicator (PSI 90), which is a composite of billing data related indicators of safety events like pressure sores, falls, and clots. The third measure is readmissions and the fourth is the Press Ganey patient satisfaction scores. Those four domains comprise 88% of the overall Star ratings.”

But three of those four roses have a thorn, however.

“The issue with PSI 90 it’s that it lacks accuracy; it is imprecise because it is based on billing data. Thus, there is a move away from using it (U.S. News & World Report doesn’t use PSI scores anymore). Unfortunately, we don’t know what to replace it with.”

“The problem with the mortality measurement is that covers only a couple of conditions. If I am just coming in for an infusion is the hospital’s acute mortality an indicator of how safe it is for me? As for readmission, is that actually a good measure of quality? There is some data saying that there are conditions in hospitals where there are lower readmission rates there is higher mortality. Some providers have sicker patients and the hospital is the right place for them.”

“Our team believes that readmission as it is currently measured is probably not what we want to use. Certain patients are very ill and the only place to properly manage them is in the hospital—but then physicians are penalized. We are now working with Mayo Clinic to develop a measure for avoidable versus unavoidable readmissions. As of now the readmission criteria is a blunt instrument, but we hope to change that.”

Asked what advice CMS is considering, Dr. Hota told OSN, “CMS wants a system that patients and doctors can understand, and that people actually use. I was impressed that they seem truly interested in improving the way things are measured. They have been using a statistical model, and while it is difficult to understand, they are attempting to simplify it. And CMS has published results comparing hospitals and are moving to provider-level ratings. As of now there isn’t a lot of such data, so it will likely take five years. A novel effort that is underway is the CMS Meaningful Measures Program, an effort to simplify the number of measures used and ensure that we have more impactful measures that are clinically relevant and robust. The program, is meant to further the goal of aligning measures across healthcare facilities.”