Doi:10.1016/j.jamcollsurg.2003.11.017

Measuring the Quality of Surgical Care:Structure, Process, or Outcomes? John D Birkmeyer, MD, FACS, Justin B Dimick, MD, Nancy JO Birkmeyer, PhD With widespread recognition that surgical outcomes standard in cardiac surgery and in hospitals of the De- vary by providersurgeons and hospitals are increas- ingly being asked to provide evidence of the quality of In this article, we consider the relative merits of these care that they deliver. Patients and their families are different approaches to measuring and ultimately im- turning to the Internet and other sources to make better proving the quality of surgical care. Adopting the Dona- informed decisions about where and by whom to un- bedian we consider quality measurement in dergo surgery. Both public and private payers are look- three domains: structure, process, and outcomes. Al- ing to steer selected populations of surgical patients to though each of these three approaches has unique ad- high-quality providers—so-called value-based purchas- vantages, each has its own conceptual and practical lim- To meet these interests, policy makers, health ser- vices researchers, and a variety of related organizationshave redoubled their efforts to develop and implement quality indicators germane to surgery.
Structural measures include a broad list of variables re- There remains considerable debate about which mea- flecting the setting or system in which care is delivered.
sures should be used to reflect surgical quality. Structural These may describe hospital’s physical plant and re- measures—a very broad group of variables that reflect sources. They also include measures that relate directly the setting in which care is delivered—have received or indirectly to staff expertise or staff coordination and considerable attention lately. For example, the Leapfrog organization. Of these variables, procedure volume, Group, a large coalition of health-care purchasers, is en- measured at either the surgeon or hospital level, is most couraging patients to seek care at hospitals with high commonly used as a surrogate for surgical quality. Al- procedure volumes for several Process mea- though the magnitude of volume-outcomes associations sures, which reflect the particulars of care that patients with various procedures is debated, there is little doubt actually receive, have long served as quality indicators in that high-volume providers have lower operative mortal- primary care and other specialties (eg, use of ␤-blockers ity, fewer complications, or better longterm survival after myocardial infarction). There is evidence that fo- with some operations than their lower-volume cusing on process measures may be equally useful in surgeryFinally, and most obviously, the quality of sur- Among other structural variables, subspecialty train- gical care can be assessed by direct outcomes measure- ing by the operating surgeon is often cited as a predictor ment. Quality improvement programs focusing on risk- of improved surgical outcomes. For example, patients adjusted morbidity and mortality rates have long been undergoing resection for rectal cancer had lower recur-rence rates and improved survival when treated by sur- geons board certified in colorectal surgeryStructural Supported by a grant from the National Cancer Institute (1 RO1 CA098481- variables more broadly related to staff organization and 01A1). Dr John Birkmeyer is a consultant for the Leapfrog Group and chairs resource availability may also influence surgical out- its expert panel on Evidence-Based Hospital Referral. Dr Dimick is also aconsultant for the Leapfrog Group.
comes. For example, a considerable body of evidence has Received September 17, 2003; Accepted November 26, 2003.
accrued suggesting that critically ill surgical patients From the Center for Surgical Evaluation and Policy, Department of Surgery have lower mortality in “closed” intensive care units— (Dimick), University of Michigan, Ann Arbor, MI, and VA Outcomes those in which patients are managed primarily by dedi- Group, VA Medical Center (Birkmeyer, Dimick, Birkmeyer), White RiverJunction, VT.
cated, board-certified intensivists.Similarly, hospitals Correspondence address: John D Birkmeyer, Section of General Surgery, with high nurse-to-bed ratios seem to have lower mor- University of Michigan, 2920 Taubman Center, 1500 East Medical CenterDrive, Ann Arbor, MI 48109-0331.
tality rates for some operations.Finally, resource avail- 2004 by the American College of Surgeons can be assessed easily and inexpensively, often with ad- CABG ϭ coronary artery bypass graftingNSQIP ϭ National Surgical Quality Improvement Program Disadvantages of structural variablesAmong the downsides, the literature assessing structuralmeasures is incomplete. It focuses on a small number of ability may be an important determinant of surgical out- variables (eg, volume) and outcomes measures (eg, op- comes. For example, one study from the Department of erative mortality). Little is known about the importance Veterans Affairs found that hospitals with lower than of structural variables that are more difficult to measure expected mortality rates tended to have more up-to-date or about relationships between structure and nonfatal technology and equipment in their intensive care outcomes. Unlike process measures, which can often be evaluated in randomized clinical trials, most structuralmeasures can only be assessed in observational studies. It Advantages of structural variables is often difficult to rule out confounding as an explana- From a measurement perspective, structural measures tion for observed associations between structure and have several attractive features as indicators of surgical outcomes. Second, in contrast to process measures, quality. As already described, many of these variables are many structural measures are not readily actionable, strongly related to surgical outcomes. For example, with which limit their ultimate effectiveness as a means to- esophagectomy and pancreatic resection, operative mor- ward quality improvement. For example, a small hospi- tality rates at very high volume hospitals are on average tal can increase how many of its high-risk patients re- 10% lower, in absolute terms, than at lower-volume cen- ceive perioperative ␤-blockers, but it cannot readily ters. The primary advantage of structural variables is make itself a high-volume center for a given procedure expediencyCompared with direct outcomes assess- or, unless it has sufficient staff, convert to a closed-model ment, structural variables, including procedure volume, Table 1. Using Structure, Process, and Outcomes to Measure Surgical Quality, with Examples, Advantages, and Disadvan-tages of Each Buy-in from surgeons—the “bottom line” outcomes for most hospitals andprocedures Table 2. Examples of Process Measures Associated with Surgical Outcomes, According to Strength of Scientific EvidenceSupporting Them and the Cost and Complexity of Implementing Them Strength of the evidence/magnitude of potential benefit Tunneling short-term “Sign your site” Based on literature review and conclusions contained in “Making Health Care Safer: A Critical Analysis of Patient Safety Procedures,” from the Agency forHealthcare Quality and Resear Finally, and most importantly, structural variables are surgery, many processes of care are strongly associated very imperfect proxies for quality—they reflect average with improved patient outcomes. For example, the results for large groups of providers, not individuals. For Agency for Healthcare Research and Quality recently example, many low-volume hospitals have excellent per- commissioned a critical review of a large number of formance, but many high-volume centers are poor per- hospital-based practices related to patient safety formers. Even if all high-risk procedures were concen- A large number of practices related to perioperative trated in high-volume hospitals, there would remain care have high levels of evidence supporting their effec- substantial variation in quality across hospitals and this tiveness. These include practices related to central ve- nous line management, critical care, and minimizingrisks of postoperative cardiac events, venous thrombo- Process variables describe the care that patients actually Procedure-specific processes of care may sometimes receive and are routinely used as quality indicators in explain apparent associations between structural vari- nonsurgical specialties. For example, in large managed ables and outcomes. For example, Hannan and col- care organizations and Department of Veterans Affairs leagues performed a prospective clinical study of pa- hospitals, primary care physicians are regularly graded tients undergoing carotid endarterectomy at six according to the proportion of appropriate patients in hospitals in New York SIn that study, vascular their practices who receive screening mammography, surgeons had substantially lower 30-day rates of op- retinal examinations (in diabetics), or pneumococcus erative stroke or death than did general surgeons or vaccinations. Similarly, providers are assessed in terms of neurovascular surgeons. The investigators also found the proportion of patients surviving a myocardial infarc- that use of intraarterial shunting, eversion endarter- tion who are discharged on aspirin and ␤-blockers.
ectomy techniques, patching of the arteriotomy, and Although not yet used widely as quality indicators in protamine were associated with lower complication rates. Greater adoption of these four processes of care serious adverse events occurring after surgery are non- by vascular surgeons explained in large part their bet- medical in nature, arising from technical problems asso- ciated with the procedure itself—anastomotic leaks,bleeding, or wound complications. Although high- leverage technical processes have been elucidated for As potential quality indicators, process of care measures some procedures (notably CABG and carotid endarter- have several attractive features. In addition to the high level ectomy), few procedures have been as carefully studied, of evidence supporting their effectiveness (often random- ized clinical trials), some process measures have very largepotential benefits. For example, in one large trial, patientsreceiving ␤-blockers during and after major noncardiac surgery had much lower 1-year mortality than patients who Since surgeon Ernest A Codman began tracking the did not (3% versus 14%, p Ͻ Second, process “end results” of surgical procedures in the early 20th variables reflect the care that patients actually receive and centurydirect outcomes assessment has long been a may be perceived by providers as “fairer” measures of qual- staple in assessing the quality of surgical care. Although operative mortality is most commonly used, other out- Finally, and most importantly, process of care mea- comes measures that could be considered quality indica- sures are generally actionable and link directly to quality tors include complication rates, length of stay, readmis- improvement activities. For example, investigators and sion rates, patient satisfaction, functional health status, clinicians at six hospitals in northern New England have and other measures of health-related quality of life.
maintained a prospective clinical registry for coronary There are many ongoing, large-scale initiatives aimed artery bypass graft (CABG) and other cardiac proce- specifically at measuring and improving surgical out- dures since 1987. They identified numerous process of comes. Clinical outcomes registries in cardiac surgery, care measures linked to lower operative mortality, in- including those launched in New York, Pennsylvania, cluding use of an internal mammary graft, continuing and northern New England in the 1980s, were among aspirin through surgery, and maintaining a hematocrit the earliest and most More states and re- of 24% or higher when “on pump.” As a result of sys- gions and one national organization (the Society for tematic efforts to increase the use of these practices and Thoracic Surgeons) have since implemented similar data timely feedback of performance data to clinicians, oper- collection systems. Although these registries vary in ative mortality rates across the region fell by almost half many respects, all provide hospitals and cardiac surgeons during the 1990s, a decline significantly greater than with feedback on their risk-adjusted morbidity and mor- observed in regions of the United States without similar tality rates. Over the past decade, prospective outcomes registries have been implemented in numerous otherfields. Although most outcomes measurement efforts have been procedure-specific, the National Surgical Measurement systems focusing on process variables Quality Improvement Program (NSQIP) of the Depart- must be able to accurately identify eligible patient pop- ment of Veterans Affairs assesses hospital-specific mor- ulations (ie, the right denominator). Many processes bidity and mortality rates aggregated across a wide range known to be effective in general may not be appropriate of surgical specialties and procedures. Efforts to apply for all patients undergoing a given procedure (eg, the same measurement approach outside the Veterans ␤-blockers in patients with bradyarrhythmias or severe left ventricular dysfunction). Ensuring the right denom-inator implies the need for clinical data and may be labor intensive, a practical limitation of process measurement.
Direct outcomes measures have at least two major ad- A second major limitation of process measures is the vantages. First, because most consider patient outcomes relative lack of evidence about which processes are im- the “bottom line” of surgical practice, efforts assessing portant for specific procedures. Much of the existing quality with direct outcomes measures have obvious face literature on processes of care focus on the medical man- validity and are likely to get the greatest buy-in from agement of surgical patients Many of the most surgeons. Second, measurement alone may improve tially compounds problems with statistical power inmeasuring outcomes at the provider level To circumvent sample size limitations with procedure- specific measures, hospitals and surgeons could determinemorbidity and mortality rates after aggregating a widerange of procedures across different surgical specialties (eg,the NSQIP approach). Unfortunately, this approach is lesssatisfying from a quality improvement perspective. Under-standing and improving the delivery of a specific proceduremay require measures specific to that operation. Aggregatedmeasures of surgical morbidity and mortality may also besuboptimal for patients (and payers) interested in identify-ing excellence with individual operations. In other words,measures weighted heavily toward outcomes of common,low-risk operations (eg, hernia repairs, cholecystectomies)may not be very informative for patients deciding where to Figure 1. Minimum number of cases required at individual hospitals to identify a statistically significant doubling of baseline mortalityrisk.
Choosing the right measureAlthough structural, process, and outcomes measures all outcomes—the so-called Hawthorne effect. Surgical have unique strengths, these three measures have distinct morbidity and mortality rates in Veterans Affairs hospi- downsides, depending on how they are used. For these tals have fallen dramatically since implementation of reasons, both surgeons and policy makers should be flex- NSQIP in No doubt many surgical leaders at ible in their approach to measuring quality and develop individual hospitals made specific organizational or proc- strategies best suited to meeting specific needs.
ess improvements after they began receiving feedback on The procedure itself may be the most important fac- their hospitals’ performance. It is very unlikely that even tor in deciding about the most effective approach to a full inventory of these specific changes would explain quality measurement. Two attributes are particularly im- such broad-based and substantial improvements in mor- portant: 1) the baseline risks of the procedure and 2) how commonly it is performed at individual hospitals Measuring quality for procedures that are both The most important limitation of direct outcomes mea- low risk and uncommonly performed (Quadrant III) surement relates to sample size. For the large majority of should receive low priority. Many high-risk procedures, surgical procedures, very few hospitals (or surgeons) such as esophagectomy and pancreatic resection (Quad- have sufficient adverse events (numerators) and cases rant IV), are performed too infrequently at the vast ma- (denominators) for meaningful, procedure-specific jority of hospitals to support direct outcomes assess- measures of morbidity or mortality. Consider a hypo- ment. Procedure volume, a structural measure highly thetical hospital with an observed mortality rate of 10% correlated with mortality for many of these procedures, for a given procedure, twice the national average of 5%.
is likely the only practical quality indicator. Quality for That hospital’s rate would need to be based on at least procedures that are both common and relatively high 185 cases to be reasonably confident (95%) that its per- risk (eg, CABG, Quadrant II) is best assessed directly formance was significantly worse than the national aver- using risk-adjusted measures of morbidity and mortality.
age and not simply indicative of random variation Quality improvement consortia designed to accomplish Outside of cardiac surgery, very few procedures have this task are also ideal platforms for measuring process baseline mortality rates of 5% or higher and are per- variables and linking them to outcomes. Measuring formed this frequently at individual hospitals (particu- quality is perhaps most problematic for common but larly low-volume ones). Most common operations tend relatively low-risk procedures (eg, laparoscopic cholecys- to be associated with low baseline risks, which substan- tectomy, Quadrant I). For these procedures, volume and Figure 2. Recommendations for when to focus on structure, process, or outcomes.
other structural measures are not known to be major need for clinical data collection systems, which can be determinants of outcomes. Low baseline rates of mortal- expensive. For example, participation in NSQIP costs ity or other serious complications preclude measuring approximately $40 per operation.Hospitals, if not outcomes with sufficient precision. Quality for these payers, need to be prepared for this level of investment if procedures is best judged by process measures (where they are to pursue data-driven quality improvement available) or by outcomes measures other than morbid- ity and mortality (eg, functional health status).
The right measure also depends on the specific policy Improving the quality of quality measurement context and the ultimate goal of quality measurement.
As this article suggests, current tools for measuring sur- With public reporting initiatives, for example, the pri- gical quality are far from perfect. Opportunities for mary goal is to inform patients about where to undergo moving the field forward by focusing exclusively on elective surgical procedures. In this context, quality mea- structural measures (proxy variables like volume) are sures must have strong face validity for patients. Despite limited. Improving the quality of quality measurement its many limitations, procedure volume, a structural will require progress in other areas. Identifying high- measure, is considered important by many patients, who leverage processes of care is clearly one of them. As de- frequently ask their surgeons, “Do you do this procedure scribed earlier, most high-level evidence linking process often?” For purposes of quality improvement, quality to surgical outcomes pertains to the medical aspects of measures should be selected primarily on the basis of perioperative care, not the technical aspects of specific their validity as judged by providers and their relative procedures that determine their success. A better under- actionability. Quality improvement requires rigorous standing of such processes is essential if successes measurement of process and outcomes.
achieved with CABG are to be replicated in other areas.
A final, practical consideration is the potentially high Second, we should place a higher priority on measur- cost of quality assessment. Though information about ing patient-centered outcomes. To date, most large-scale structural variables can be obtained expediently using quality improvement initiatives in surgery have focused existing data, process and outcomes measures imply the on measures of morbidity and mortality. Though such outcomes may be central for many cardiovascular and risk-adjusted, and peer-controlled program for the measure- cancer procedures, they are considerably less useful for ment and enhancement of the quality of surgical care. NationalVA Surgical Quality Improvement Program. Ann Surg 1998; assessing the quality of low-risk operations, particularly those whose primary goal is improving health-related 8. O’Connor GT, Plume SK, Morton JR, et al. Results of a regional
quality of life. Valid and reliable instruments for assess- prospective study to improve the in-hospital mortality associatedwith coronary artery bypass grafting. JAMA 1996;275:841–846.
ing health-related quality of life are widely available.
9. Donabedian A. Evaluating the quality of medical care. Milbank
The more difficult task is finding ways to collect such 10. Dudley RA, Johansen KL, Brand R, et al. Selective referral to
Finally, we must be careful to avoid missing the big high volume hospitals: estimating potentially avoidable deaths.
JAMA 2000;283:1159–1166.
picture. Quality in health care can be described as “doing 11. Begg CB, Reidel ER, Bach PB, et al. Variations in morbidity after
the right things right.” This article (and debates about radical prostatectomy. N Engl J Med 2002;346:1138–1144.
surgical quality in general) focuses on only the latter 12. Finlayson EV, Birkmeyer JD. Effects of hospital volume on life
expectancy after selected cancer operations in older adults: a component of this aphorism—how well the procedure decision analysis. J Am Coll Surg 2003;196:410–417.
was performed (ie, doing things right). As suggested by 13. Porter GA, Soskolne CL, Yakimets WW, Newman SC.
wide geographic variation in the use of different proce- Surgeon-related factors and outcome in rectal cancer. Ann Surg1998;227:157–167.
dures in the United States, the quality of the decision to 14. Pronovost PJ, Angus DC, Dorman T, et al. Physician staffing
operate in the first place (ie, doing the right thing) may patterns and clinical outcomes in critically ill patients: a system- be an equally important issue in surgical care. A full atic review. JAMA 2002;288:2151–2162.
accounting of surgical quality will require measures of 15. Needleman J, Buerhaus P, Mattke S, et al. Nurse-staffing levels
and the quality of care in hospitals. N Engl J Med 2002;346: appropriateness and how well patient preferences are incorporated in clinical decisions, in addition to those 16. Daley J, Forbes MG, Young GJ, et al. Validating risk-adjusted sur-
assessing how well they do after surgery.
gical outcomes: site visit assessment of process and structure. Na-tional VA Surgical Risk Study. J Am Coll Surg 1997;185:341–351.
17. Birkmeyer JD, Siewers AE, Finlayson EVA, et al. Hospital vol-
ume and surgical mortality in the United States. N Engl J Med Study conception and design: J Birkmeyer, Dimick, N 18. Making health care safer: A critical analysis of patient safety
practices. Rockville, MD: Agency for Healthcare Research andQuality; 2001.
19. Hannan EL, Popp AJ, Feustel P, et al. Association of surgical
specialty and processes of care with patient outcomes for carotidendarterectomy. Stroke 2001;32:2890–2897.
20. Mangano DT, Layug EL, Wallace A, Tateo I. Effect of atenolol
on mortality and cardiovascular morbidity after noncardiac sur- gery. Multicenter Study of Perioperative Ischemia Research 1. Khuri SF, Daley J, Henderson W, et al. Risk adjustment of the
Group. N Engl J Med 1996;335:1713–1720.
postoperative mortality rate for the comparative assessment of 21. Peterson ED, DeLong ER, Jollis JG, et al. The effects of New York’s
the quality of surgical care: results of the National Veterans Af- bypass surgery provider profiling on access to care and patient out- fairs Surgical Risk Study. J Am Coll Surg 1997;185:315–327.
comes in the elderly. J Am Coll Cardiol 1998;32:993–999.
2. O’Connor GT, Plume SK, Olmstead EM, et al. A regional
22. Neuhauser D. Ernest Amory Codman MD. Qual Health Care
prospective study of in-hospital mortality associated with coro- nary artery bypass grafting. JAMA 1991;266:803–809.
23. Hannan EL, Kilburn H Jr, O’Donnell JF, et al. Adult open heart
3. Galvin R, Milstein A. Large employers’ new strategies in health
surgery in New York State. An analysis of risk factors and hos- care. N Engl J Med 2002;347:939–942.
pital mortality rates. JAMA 1990;264:2768–2774.
4. Birkmeyer JD, Wennberg DE, Young M, Birkmeyer CB. Leap-
24. Fink AS, Campbell DA Jr, Mentzer RM Jr, et al. The National
frog safety standards: potential benefits of universal adoption.
Surgical Quality Improvement Program in non-veterans admin- Washington, DC: The Business Roundtable; 2000.
istration hospitals: initial demonstration of feasibility. Ann Surg 5. Ferguson TB Jr, Peterson ED, Coombs LP, et al. Use of contin-
uous quality improvement to increase use of process measures in 25. Khuri SF, Daley J, Henderson WG. The comparative assess-
patients undergoing coronary artery bypass graft surgery: a ran- ment and improvement of quality of surgical care in the Depart- domized controlled trial. JAMA 2003;290:49–56.
ment of Veterans Affairs. Arch Surg 2002;137:20–27.
6. Hannan EL, Kilburn H Jr, Racz M, et al. Improving the out-
26. Khuri SF. Quality, advocacy, healthcare policy, and the surgeon.
comes of coronary artery bypass surgery in New York State.
27. McDowell I, Newell C. Measuring health: a guide to rating
7. Khuri SF, Daley J, Henderson W. The Department of Veterans
scales and questionnaires. New York: Oxford University Press; Affairs’ NSQIP: the first national, validated, outcome-based,

Source: http://www.surgicalcomplication.info/PDF/2004BJD.pdf