Journal for Quality in Health Care

1Associate Professor, Departments of Medicine, Health Policy & Management, and Epidemiology, and Director, Quality of Care Research, The Johns Hopkins Medical Institutions, Baltimore, MD 2Assistant Professor, Departments of Anesthesiology & Critical Care Medicine, Surgery, and Health Policy & Management, The Johns Hopkins Medical Institutions, Baltimore, MD and 3Assistant Professor, Departments of Medicine and Epidemiology, The Johns Hopkins Medical Institutions, Baltimore, MD, USA

Abstract

As consumers, payers, and regulatory agencies require evidence regarding health care quality, the demand for process of care measures will grow. Although outcome measures of quality represent the desired end results of health care, validated process of care measures provide an important additional element to quality improvement efforts, as they illuminate exactly which provider actions could be changed to improve patient outcomes. In this essay, we discuss the advantages and disadvantages of process measures of quality, and outline some practical strategies and issues in implementing them.

Keywords: outcomes, performance improvement, quality

Quality of health care has become a national and international six aims of being effective, safe, patient-centered, timely, efficient, and equitable [4]. Whereas the aims of effectivenesspolicy issue. Decades of study indicate that quality of care

needs improvement all over the world, and therefore sen- and safety of health care are nearly universal, societies and cultures around the world differ more in how much theytiment has grown that public disclosure of information about

quality of care should be one component of clinical gov- emphasize the additional aims of patient-centeredness, time- liness, efficiency, and equity. Process of care measures ofernance [1]. The United States government has developed

the Agency for Healthcare Research and Quality (AHRQ) quality assess the degree to which providers perform health care processes demonstrated to achieve the desired aims andand the National Quality Forum to promote the development

and reporting of quality measures [2]. the degree to which they avoid processes that avert the desired aims.Readers of this journal are well aware that more than thirty

years ago, Donabedian proposed that we can measure the Public agents and payers’ ultimate concern rests with providers’ impact on patient outcomes, and most of theirquality of health care by observing its structure, its processes,

and its outcomes [3]. The Institute of Medicine (IOM) in the measures of quality to date have focused on this. For example, the Center for Medicare and Medicaid Services (CMS formerlyUS has defined health care quality as ‘the degree to which

health services for individuals and populations increase the the Health Care Financing Administration, HCFA), which administers the Medicare entitlement program for the elderlylikelihood of desired health outcomes and are consistent with

current professional knowledge’ [4]. The IOM’s definition in the US, began by releasing mortality rates for hospitals [8]. The need for risk adjustment when comparing differentand framework thus incorporate two of Donabedian’s three

elements in a broad approach to measuring health care quality: providers became evident, and subsequent efforts attempted this. For example, several US state governments including(1) determining effects of health care on desired outcomes,

including a relative improvement in health, and in consumer New York, Pennsylvania, and California provided their cit- izens with publicly available report cards containing risk-evaluations or experience of health care and (2) assessing the

degree to which health care adheres to processes that are adjusted mortality rates for cardiac surgery by hospital and surgeon [9–12].proven by scientific evidence, professional consensus to affect

health, or that concur with patient preference [5–7]. The Measures of the process of care that affect outcomes were initially considered too technical for public or regulatory use.IOM has further suggested that health care should have the

Address reprint requests to H. R. Rubin, Quality of Care Research, The Johns Hopkins University, 1830 East Monument Street #8015, Baltimore, MD 21205, USA. E-mail: hrubin@jhmi.edu

© 2001 International Society for Quality in Health Care and Oxford University Press 469

H. R. Rubin et al.

Theoretically, providers would respond to publicly reported discuss practical strategies for using quality measures for improvement.outcome measurement by developing and implementing their

own internal report cards focused on evidence-based process measures. However, given the complexity and expense of developing clinical guidelines and process measures and keep- Advantages and disadvantages of process ing them updated, and the infancy of clinical information measures of quality of care systems that would track the clinical process routinely, in- dividual provider organizations have had difficulty in- There is considerable debate regarding whether quality meas- corporating process measurement into their operations. In ures should evaluate processes or outcomes of care. Before the last five years, joint efforts by providers, professional proceeding to develop process indicators, those considering societies, monitoring agencies, and quality-of-care experts this may find it useful to understand their strengths and have begun to assist in identifying and implementing ap- limitations. Within the categories of process and outcome, propriate process-of-care measures. In addition, government there are good and bad measures, but some specific ad- and payers have a better understanding of the difficulty of vantages and disadvantages apply broadly to all process risk-adjustment for outcome measures such as mortality rates, measures compared with outcomes (Table 1). and their own interest in process of care measures of quality On the plus side, process measures can be used to provide has grown. The National Committee on Quality Assurance feedback for quality improvement initiatives. Because many (NCQA) in the United States collects data on HEDIS® quality factors can influence patient outcomes, process measures measures and includes evidence-based measures of health have the potential to identify for clinicians exactly which plan processes of care [13]. In the US, these measures are processes they followed or didn’t follow that had the potential part of NCQA’s health plan accreditation program and are to affect patient outcomes. Process measures provide in- used by some employers, insurers, and government payers formation that is actionable; i.e. what is being done well to choose participating health plans. CMS (formerly HCFA) and what needs improvement. When process measures are in the US has also proceeded from reporting on Medicare developed well, so that they accurately reflect the care that beneficiary mortality rates to developing, measuring, and clinicians are delivering, clinicians feel accountable for them. reporting on evidence-based hospital and outpatient care In contrast, many other factors affect health care outcomes processes [14]. Although HCFA has released data on indicator that are beyond the provider’s control. When a clinician performance to the public only by state and not by provider, discovers that his patient had worse outcomes than another several US states have passed or are considering legislation clinician’s patients, it is unclear what he or she should be to collect and publish the same indicators for hospitals doing about it. When data collection about process of care and practices in those states (B. Miller, Maryland Hospital is generated unobtrusively simply by an electronic patient Association, personal communication). record when a provider performs the process, these measures

In the US, where employer-provided health insurance is become even more attractive and feasible because they elim- the norm, consortia of employers are also using quality inate burdensome additional data collection. measures to assess and select providers, and these are also Secondly, most process measures require less risk ad- beginning to incorporate some evidence-based measures of justment for patient illness than do most outcome measures. structure and process. Whereas the first effort of this type, The use of a process measure requires defining a population the Cleveland Health Quality Choice program [15], focused that is eligible to receive the process such as which patients on outcome measures, newer efforts are now including with asthma should receive anti-inflammatory medications or evidence-based, validated structure and process measures. For which patients should receive beta-blockers or aspirin after example, The Leapfrog group [16], a health care purchasing myocardial infarction. Once the eligible population is spe- consortium representing Fortune 500 companies, has de- cified, further risk adjustment is generally not required, al- veloped a national measure of quality of intensive care unit though it can be useful. In contrast, comparing mortality rates (ICU) care and provides incentives for their employees to or other outcomes for specific clinical conditions requires risk purchase health care from hospitals that meet this standard. adjustment. Risk adjustment requires definition and measure- In the area of validated structural and process measures, ment of many patient characteristics, including physiological, the standard includes whether intensive care specialists are anatomical, and health status data that are not part of routine monitoring the ICU, which has been demonstrated to improve administrative databases at present, that may not be part of patient outcomes [17]. medical records, and that may be expensive to collect. In

Health care quality measures, including process measures, addition, existing risk adjustment models often perform are developed for varied audiences who may wish to use poorly when applied to new data sets, limiting application of them for health care purchasing, utilization, or performance a common risk adjustment model to all providers [19,20]. improvement. For all these purposes it is imperative that The development of new risk adjustment models is analytically they are meaningful, scientifically sound, generalizable, and complex, requiring expert statistical analytical resources and interpretable [18]. To achieve this, quality measures must be a large sample of patients available for the development designed and implemented with scientific rigor. In this essay, process. Using process measures applied to an accurately we present an overview of the advantages and disadvantages defined population avoids some of the time and expense that

risk adjustment entails.of using process measures as quality of care measures, and

470

Process-based measures of health care quality

Table 1 Advantages and disadvantages: comparison of process and outcome measures

Process measures Outcome measures…………………………………………………………………………………………………………………………………………………………………………………………………………….. Resources

Need for updating and maintenance of Require updating and maintenance of Known risk factors and models may measures guidelines, review criteria, instruments require some updating, generally less

and software according to advances in often than updates to process measures treatment are needed

Need for development of risk Most measures do not require the use of Risk adjustment is difficult; need adjustment models and collection of extensive risk adjustment models; different models for each outcome risk data however, require good definition of

eligible patients Time needed for measurement Takes less time to accumulate, smaller Due to need for risk adjustment a larger

sample needed, less observation time sample is needed; also many outcomes needed for processes occurring during of interest are long-term such as five- or provider contact ten-year survival requiring long period of

observation Size of population needed for Can use a smaller sample size as all Due to need for risk adjustment a larger measurement included patients experience the process sample is usually needed for

and once eligible patients are defined, comparisons among providers or only descriptive statistics are needed treatments

Need for additional follow-up tracking Data collection can be done when Requires follow-up for measurement of of patients for later data collection clinical process is occurring short- and long-term outcomes at time

when routine data collection not occurring

Use of routinely collected data Has the potential to be abstracted from Often requires collection of data data already recorded for clinical and elements that are not being recorded for administrative use, and ultimately to be clinical or billing purposes such as long- completely integrated into such data term survival or patient-reported health collection and well-being on standardized scales

Need for advanced statistical Not needed in general Needed to create risk adjustment models consultation for development of and evaluate them when analyzing data measures and analysis of data

Validity

What patients care about Often inaccessible to patients who often The generic outcomes of survival, health do not understand the significance of a and well-being are what patients care specific component of care about and measures of these can be used

to validate process changes. More specific or proxy outcomes may not be accessible to patients

What providers care about Face validity with providers; relates Providers are wary of outcome measures directly to what the provider is doing that are influenced by many other things

besides what they do; must measure performance of risk adjustment models

Ease of use

Ease of specification and identification Difficult to specify population eligible Easy to define population for which of population at risk for a process; there can be many want to measure an outcome; many

exclusions, contraindications and special important outcomes are generic and can outcomes, and many important process be compared across several conditions measures are specific for a single disease

Creation of valid summary measures Difficult to summarize process measures Many important outcome measures, such in a valid way as they are rarely as survival, health and well-being, are comprehensive both global and generic and can be

compared across conditions and processes

Interpretability of feedback for quality Provides clear and interpretable feedback Most measures cannot be used to give improvement for quality improvement about what feedback to providers about how to

providers are actually doing; easy to improve what they are doing as rarely is benchmark an outcome always a consequence of a

particular faulty process; benchmarking is needed for comparisons among groups and adjusted outcomes can be difficult to understand

471

H. R. Rubin et al.

Thirdly, process measures can usually be collected more for self-management, and identification and avoidance of asthma triggers. While this comprehensive approach mayquickly than outcome data for two reasons, Firstly, outcome represent ideal care, it may be feasible to measure only whetherevents may be rare where only a small percentage of the certain medications have been dispensed by a pharmacy,patients have the outcome, requiring one to accrue a larger which provides a narrow perspective on overall asthmasample of patients. In addition, many outcomes that are care. Feasibility of certain measures may be limited by whatimportant to patients, such as quality of life and functional information is usually collected in a patient record. Forstatus, may require years after the illness to evaluate, increasing example, while many clinicians will document laboratorythe time for measurement. In contrast, care delivery occurs testing or use of medications, it is rare to document con-in a shorter period of time, and every eligible patient receives versations in which patients are educated about care of theirthe specific process of care being evaluated. illness. Thus, use of certain processes as indicators of careOn the other hand, there are several disadvantages to may be dictated by availability of data, rather than the relativeprocess measures. Firstly, to be valid, there must be a strong importance of the element of care. In addition, a providerrelationship between the process and outcome measures. may adhere well to one part of the process but not to another.These links between process and outcomes can come from Therefore, if process measures are not comprehensive andpreviously published evidence, or may be demonstrated for do not cover all the important parts of the process that canthe group whose quality of care is being evaluated. The prior affect outcomes, they may be misleading to users.evidence supporting the relationship may be weak or non-

existent for many processes even when they are truly linked to outcomes. Even when studies have been carried out, they may not demonstrate true process–outcome linkages. For Practical strategies for developing quality example, observational studies may show paradoxical as- measuressociations of good care with inferior outcomes because of confounding by indication. In confounding by indication,

As we have discussed previously, the audience for and usesicker patients (who subsequently have worse outcomes) of the quality measure will influence how the measure isreceive more or better care, setting up the paradoxical ob- developed and how the results are presented. Quality measuresservation that good care is linked to inferior outcomes. This are in general used by provider or care management or-phenomenon is particularly problematic for patients with ganizations to evaluate and enhance performance throughchronic illnesses such as asthma, for which measures of quality improvement initiatives, by purchasers and patientsintrinsic disease severity are poor. Therefore, it may be to inform health care decision making, or by accreditors anddifficult to find evidence to support valid process measures. regulators to monitor organizations.Secondly, when evidence linking process and outcomes is

absent yet providers believe the process is important, or Internal quality improvement initiativeseven if such evidence is available, providers may desire to

demonstrate the relationship between a process and outcome For internal organizational quality improvement initiatives, measure in their organization. This need may be especially more detailed, less aggregated measures of quality may be strong when a clinical unit or quality improvement effort is more helpful than summary measures. Providers want in- requesting additional resources from their administrators to formation regarding how to improve specific processes. As support an evaluation effort or a change in process. If the such, the unit of analysis for these initiatives tends to be professional and scientific community has not conducted the small, such as the individual unit, practice, or clinician. These needed studies, or these studies are thought to be inapplicable types of measure often require significant technical detail. An to a specific organization’s population, then demonstrating important consideration in these types of measure is that the link between process and outcome is prohibitively ex- clinicians believe that the process is related to the outcome pensive and often impossible to achieve for any one or- such that improvement in the process will result in im- ganization. provement in outcomes. Without this belief, it is often

Thirdly, while providers may care about process measures, necessary to establish the link between process and outcome. patients and non-clinicians generally place little value on Data collection for quality improvement can be made part them; they care about outcomes and believe it is the provider’s of routine care by existing staff and thus marginal costs responsibility to perform the appropriate processes and to are minimal. Electronic medical record systems play a very avoid harmful ones. Therefore, the measures mean little to important role here, and whenever possible, quality assess- consumers or purchasers for plan or provider selection, and ment efforts should attempt to obtain needed data by re- may be less useful to a provider organization in its marketing placing usual clinical data collection with collection of efforts. standardized data elements. Finally, when used internally for

Fourthly, most feasible process measures are usually in- quality improvement, the statistical significance of the results dicators for a very specific element of the care process rather is often less important than the practical interpretation or than comprehensive measures of how care is delivered. For the visual impact of differences in a graphic presentation. example, national asthma guidelines recommend a com- For example, data from these initiatives are often presented prehensive approach to care, including appropriate medication as run charts, indicating a rate or score trend over time,

with clinically important changes illustrated as crossing linesuse, periodic assessment of disease status, patient education

472

Process-based measures of health care quality

representing acceptable upper or lower bounds rather than to develop and maintain process measures have made them more feasible. To be valid, process measures should havestatistically significant differences. links to important outcomes, or should at least be determined by consensus methods to be judged by clinical experts to bePurchasers and patients important to patient outcomes. The past decade has broughtIf the audience for the measures is purchasers or patients and a greater emphasis on synthesizing the evidence basis forthe intent to provide information for health care purchasing how process of care affects outcomes and has made thisdecisions, summary or aggregated data are more helpful. information more readily available to the provider communityThese groups may prefer a comprehensive measure that is as well as the public. In the future this will provide thepresented in a way that is understandable to patients, benefit ultimate base for the development of process measures ofmanagers, and enrollees. For these groups, it is important quality.that the process measure is already demonstrated to have a Finally, practical strategies were reviewed for various usesstrong link to outcome, as they do not perceive it as their of process of care measures for internal quality improvement,role to conduct clinical studies establishing these links. It is for choice of providers by patients or purchasers, and formore important for these audiences that patients and enrollees accreditation and regulatory purposes.rather than providers believe the measure evaluates an im-

portant domain of quality. Statistical significance among groups becomes increasingly important; patients and enrollees

Referenceswant to know ‘which is better’. Despite this, further research is needed to design and present quality measures that enable

1. Marshall M, Shekelle P, Brook R, Leatherman S. Dying to Know:patients to make informed decisions regarding quality of care. Public Disclosure of Information about the Quality of Care. Santa Monica, CA and London, UK: The Nuffield Trust and RAND,

Accreditation and regulatory 2000. For an accreditation or regulatory audience, summary meas- 2. Miller T, Leatherman S. The National Quality Forum: a ‘me- ures for the relevant unit of analysis are helpful. For example, too’ or a breakthrough in quality measurement and reporting? in the US an organization such as the NCQA that accredits Health Aff (Millwood) 1999; 18: 233–237. health plans will use health plans as the unit of analysis.

3. Donabedian A. Evaluating quality of medical care. Millbank QThese groups generally present summary measures and the 1996; 44: 166–206.measures must be strongly linked to outcomes. For these

4. Lohr KN, Schroeder SA. A strategy for quality assurance inuses, acceptance by the clinical community is also important, Medicare. N Engl J Med 1990; 322: 1161–1171.and providers often assist in the development of the ac-

creditation policies. Finally, the burden on clinicians to collect 5. Palmer RH, Miller MR. Methodologic challenges in developing and analyze the data must be low, as results are not used for and implementing measures of quality of child health care. direct clinical feedback. Unfunded mandates can place an Ambul Pediatr 2001; 1: 39–52. increased burden on a taxed health care system, potentially

6. Palmer RH. Using health outcomes data to compare plans,mitigating the benefits of assessing quality. networks, and providers. Int J Qual Health Care 1998; 10: 477–483.

7. Palmer RH. Process based measures of quality: the need forConclusion detailed clinical data in large health care databases. Ann Intern Med 1997; 127: 733–738.In this essay, we have discussed the advantages and dis-

8. Berwick DM, Wald DL. Hospital leaders’ opinions of the HCFAadvantages of process measures of quality, and some strategies mortality data. J Am Med Assoc 1990; 263: 247–249.for implementing them for different purposes or audiences.

Process measures are highly acceptable to providers because 9. Jollis JG, Romano PS. Pennsylvania’s focus on heart attack— they demonstrate clearly how providers can improve their grading the scorecard. N Engl J Med 1998; 338: 983–987. outcomes. Clinicians are also more accountable for the pro-

10. Hannan EL, Kilburn H, Racz M, Shields E, Chassin MR.cess of care than its outcomes, which are affected by many Improving the outcomes of coronary artery bypass graft mor-other things. As electronic medical records become more tality in New York State. J Am Med Assoc 1995; 273: 209–213.common, process measures can be unobtrusively tracked

11. Green J, Wintfeld N. Report cards on cardiac surgeons; assessingas part of routine clinical care, which will aid in their New York State’s approach. N Engl J Med 1995; 332: 1229–1232.implementation. Process measures that are incorporated into

routine clinical data collection also provide a constant edu- 12. Rainwater JA, Romano PS, Antonius DM. The California Hos- cational reminder to clinicians about the correct process, and pital Outcomes Project: How useful is California’s report card eliminate duplicative data collection for quality assessment. for quality improvement? Jt Comm J Qual Improv 1998; 24: 31–39. On the other hand, implementing process measures can be

13. National Committee on Quality Assurance, USA: http://difficult because they require constant updating as the science www.ncqa.org. Note: website address at time of printing.of medicine advances. Joint efforts among providers, pro-

fessional societies, and external government or payer agencies 14. Jencks SF, Cuerdon T, Burwen DR et al. Quality of medical

473

H. R. Rubin et al.

care delivered to Medicare beneficiaries: A profile at state and 19. Angus DC, Clermont G, Kramer DJ et al. Short-term and long-term outcome prediction with the Acute Physiologic andnational levels. J Am Med Assoc 2000; 284: 1670–1676. Chronic Health Evaluation II system after orthotopic liver15. Rosenthal GE, Hammar PJ, Way LE et al. Using hospital transplantation. Crit Care Med 2000; 28: 150–156.performance data in quality improvement: The Cleveland Health

Quality Choice experience. Jt Comm J Qual Improv 1998; 24: 20. Clermont G, Angus DC, Dirusso SM et al. Predicting hospital 347–360. mortality for patients in the intensive care unit: A comparison

16. The Leapfrog Group, USA: http://www.leapfroggroup.com. of artificial neural networks with logistic regression models. Crit Note: website address at time of printing. Care Med 2001; 29: 291–296.

17. Pronovost PJ, Jenckes M, Dorman T et al. Organizational characteristics of intensive care units related to outcomes of abdominal aortic surgery. J Am Med Assoc 1999; 281: 1310–1317.

18. McGlynn EA. Choosing and evaluating clinical performance measures. Jt Comm J Qual Improv 1998; 24: 470–479. Accepted for publication 28 September 2001

474