Paying for quality, not quantity is getting to be almost cliche. Recently, the Obama administration doubled down, however, announcing ambitious targets for Medicare payments to be "tied to" quality in the future. Unfortunately, the evidence behind such programs working continues to elude us. This week, two studies were published in JAMA that add to my skepticism. The first, "Association of Hospital Participation in a Quality Reporting Program With Surgical Outcomes and Expenditures for Medicare Beneficiaries":
Importance The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides feedback to hospitals on risk-adjusted outcomes. It is not known if participation in the program improves outcomes and reduces costs relative to nonparticipating hospitals.Objective To evaluate the association of enrollment and participation in the ACS NSQIP with outcomes and Medicare payments compared with control hospitals that did not participate in the program.
Design, Setting, and Participants Quasi-experimental study using national Medicare data (2003-2012) for a total of 1?226?479 patients undergoing general and vascular surgery at 263 hospitals participating in ACS NSQIP and 526 nonparticipating hospitals. A difference-in-differences analytic approach was used to evaluate whether participation in ACS NSQIP was associated with improved outcomes and reduced Medicare payments compared with nonparticipating hospitals that were otherwise similar. Control hospitals were selected using propensity score matching (2 control hospitals for each ACS NSQIP hospital).
Main Outcomes and Measures Thirty-day mortality, serious complications (eg, pneumonia, myocardial infarction, or acute renal failure and a length of stay >75th percentile), reoperation, and readmission within 30 days. Hospital costs were assessed using price-standardized Medicare payments during hospitalization and 30 days after discharge.
Researchers looked at a decade of data for hundreds of hospitals which did (and did not) participate in the American College of Surgeons National Surgical Quality Improvement Program. Specifically, they looked at whether 30-day mortality, complications, and readmission changed in participating hospitals. They looked at costs as well.
After controlling for patient factors and secular trends, they found no differences at all in outcomes before enrollment, and 1, 2, and 3 years after enrollment. No difference in 30-day mortality (4.5% before, 4.3% after. No difference in complications (11.0% before, 11.1% after). No difference in readmissions (12.8% before, 13.3% after). Mean total Medicare payments didn't change either.
Some surgical outcomes did improve over the study, but these occurred in all hospitals, even those not in the program. It appears that the quality-based feedback didn't make a difference.
The second study, "Association of Hospital Participation in a Surgical Outcomes Monitoring Program With Inpatient Complications and Mortality":
Importance Programs that analyze and report rates of surgical complications are an increasing focus of quality improvement efforts. The most comprehensive tool currently used for outcomes monitoring in the United States is the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP).Objective To compare surgical outcomes experienced by patients treated at hospitals that did vs did not participate in the NSQIP.
Design, Setting, and Participants Data from the University HealthSystem Consortium from January 2009 to July 2013 were used to identify elective hospitalizations representing a broad spectrum of elective general/vascular operations in the United States. Data on hospital participation in the NSQIP were obtained through review of semiannual reports published by the ACS. Hospitalizations at any hospital that discontinued or initiated participation in the NSQIP during the study period were excluded after the date on which that hospital’s status changed. A difference-in-differences approach was used to model the association between hospital-based participation in NSQIP and changes in rates of postoperative outcomes over time.
Exposure Hospital participation in the NSQIP.
Main Outcomes and Measures Risk-adjusted rates of any complications, serious complications, and mortality during a hospitalization for elective general/vascular surgery.
This study looked at surgical outcomes at hospitals also participating, and not participating, in this program. Specifically, they looked at elective surgery to look at rates of complications and mortality from 2009 to 2013. More than 100 hospitals took part, with more than 345,000 hospitalizations. The most common procedures were hernia repairs, bariatric surgery, mastectomy, and cholecystectomy. After controlling for other factors, once again there were no differences between hospitals in the quality programs and those out of them with respect to complications, serious complications, or mortality.
From the accompanying editorial:
The most likely explanation for the findings of these 2 studies is that end-results information, although necessary for improvement, is not sufficient, and that the skills necessary to make effective changes in processes and cultures do not yet pervade US hospitals, to say the least. Both research groups speculate about that as a reason for their results. As Osborne et al suggest, “Changing physician practice requires complex, sustained, multifaceted interventions, and most hospitals may not have the expertise or resources to launch these effective quality improvement interventions.”
I agree. I've said so before, and I'll say it again. I think the problem with our quality metrics is that we use the ones that are easy to measure, not the ones that "matter". Getting information is also just the first step. Using it to change behavior and practice requires infrastructure, time, and money. We routinely underestimate how much more than just "data" is necessary. As long as we continue to do so, I fear many of the announcements we see about "paying for quality, not quantity" will amount to sound and fury.