I have written, both here, at The Incidental Economist, and even the New York Times about my skepticism about pay for performance measures. One of my main concerns is that the measures picked often don’t translate into actual benefits seen by patients themselves. But that’s just my opinion? What do the people in charge of running the hospitals and programs to improve quality think?
A recent study in JAMA Internal Medicine helps to answer that question. “Attitudes of Hospital Leaders Toward Publicly Reported Measures of Health Care Quality”:
Importance: Public reporting of quality is considered a key strategy for stimulating improvement efforts at US hospitals; however, little is known about the attitudes of hospital leaders toward existing quality measures.Objectives: To describe US hospital leaders’ attitudes toward hospital quality measures found on the Centers for Medicare & Medicaid Services’ Hospital Compare website, assess use of these measures for quality improvement, and examine the association between leaders’ attitudes and hospital quality performance.
Design, Setting, and Participants: We mailed a 21-item questionnaire from January 1 through September 31, 2012, to senior hospital leaders from a stratified random sample of 630 US hospitals, including equal numbers with better-than-expected, as-expected, and worse-than-expected performance on mortality and readmission measures.
Main Outcomes and Measures: We assessed levels of agreement with statements concerning quality measures, examined use of measures for improvement activities, and analyzed the association between leaders’ attitudes and hospital performance.
The researchers in this study mailed out questionnaires to the senior hospital leaders of more than 600 hospitals with a range of performance on CMS quality measures of mortality and readmission metrics. They achieved a 60% response rate, which is pretty impressive.
They found that hospital leaders take the performance measures seriously. More than 87% of them used them in setting annual goals for their hospitals. More than 90% of them reviewed their results for these measures with their boards of trustees. More than 94% of them reviewed the results with senior clinical and administrative leaders.
However, that doesn’t mean that they thought that these measures accurately measured patient experiences at their hospitals. Less than half of them thought that measures of mortality or readmission were a good assessment of actual quality at the hospital.
Around half of these leaders felt that the focus on these reporting measures could lead to a reduced focus on other factors that were more important. They also expressed concerns that hospitals might “game the system” by improving documentation and coding rather than actually improving their performance. Many also thought that random variation might affect the measures as much as actual policies. This isn’t a new concern either.
In fact, those directly in charge of quality – like chief quality officers or vice presidents of quality – were more skeptical of the value of publicly reported performance measures than CEOs or CMOs. It seems that those closest to the measures have the least amount of faith in them.
It’s not that the idea of paying for performance is bad, or that it can’t work. The problem is that the goals and needs of different hospitals may be, well, different. Additionally, it’s often the case that we continue to measure what’s easy rather than what matters. It’s not just me saying this. It’s the people in charge of quality all over the country.