Measuring healthcare quality
3.10 Attribution and risk-adjustment
Two further conceptual and methodological considerations are essential when embarking on quality measurement or making use of quality data, in particular with regard to outcome indicators. Both are related to the question of respon-sibility for differences in measured quality of care or, in other words, related to the question of attributing causality to responsible agents (Terris & Aron, 2009).
Ideally, quality measurement is based on indicators that have been purposefully developed to reflect the quality of care provided by individuals, teams, provider organizations (for example, hospitals) or other units of analysis (for example, networks, regions, countries) (see also above, Quality of quality indicators).
However, many existing quality indicators do not reflect only the quality of care provided by the target of measurement but also a host of factors that are outside the direct control of an individual provider or provider organization.
For example, surgeon-specific mortality data for patients undergoing coronary artery bypass graft (CABG) have been publicly reported in England and several states of the USA for many years (Radford et al., 2015; Romano et al., 2011).
Yet debate continues whether results actually reflect the individual surgeon’s
quality of care or rather the quality of the wider hospital team (for example, including anaesthesia, intensive care unit quality) or the organization and man-agement of the hospital (for example, the organization of resuscitation teams within hospitals) (Westaby et al., 2015). Nevertheless, with data released at the level of the surgeon, responsibility is publicly attributed to the individual and not to the organization.
Other examples where attributing causality and responsibility is difficult include outcome indicators defined using time periods (for example, 30-day mortality after hospitalization for ischemic stroke) because patients may be transferred between different providers and because measured quality will depend on care received after discharge. Similarly, attribution can be problematic for patients with chronic conditions, for example, attributing causality for hospitalizations of patients with heart failure – a quality indicator in the USA – is difficult because these patients may see numerous providers, such as one (or more) primary care physician(s) and specialists, for example, nephrologists and/or cardiologists.
What these examples illustrate is that attribution of quality differences to providers is difficult. However, it is important to accurately attribute causality because it is unfair to hold individuals or organizations accountable for factors outside their control. In addition, if responsibility is attributed incorrectly, quality improve-ment measures will be in vain, as they will miss the appropriate target. Therefore, when developing quality indicators, it is important that a causal pathway can be established between the agents under assessment and the outcome proposed as a quality measure. Furthermore, possible confounders, such as the influence of other providers or higher levels of the healthcare system on the outcome of interest, should be carefully explored in collaboration with relevant stakeholders (Terris & Aron, 2009).
Of course, many important confounders outside the control of providers have not yet been mentioned as the most important confounders are patient-level clinical factors and patient preferences. Prevalence of these factors may differ across patient populations and influence the outcomes of care. For example, severely ill patients or patients with multiple coexisting conditions are at risk of having worse outcomes than healthy individuals despite receiving high-quality care. Therefore, providers treating sicker patients are at risk of performing poorly on measured quality of care, in particular when measured through outcome indicators.
Risk-adjustment (sometimes called case-mix adjustment) aims to control for these differences (risk-factors) that would otherwise lead to biased results. Almost all outcome indicators require risk-adjustment to adjust for patient-level risk fac-tors that are outside the control of providers. In addition, healthcare processes may be influenced by patients’ attitudes and perceptions, which should be
taken into account for risk-adjustment of process indicators if relevant. Ideally, risk-adjustment assures that measured differences in the quality of care are not biased by differences in the underlying patient populations treated by different providers or in different regions.
An overview of potential patient (risk-) factors that may influence outcomes of care is presented in Table 3.7. Demographic characteristics (for example, age), clinical (for example, co-morbidities) and socioeconomic factors, health-related behaviours (for example, alcohol use, nutrition) and attitudes may potentially have an effect on outcomes of care. By controlling for these factors, risk-adjustment methods will produce estimates that are better comparable across individuals, provider organizations or other units of analysis.
The field of risk-adjustment is developing rapidly and increasingly sophisticated methods are available for ensuring fair comparisons across providers, especially for conditions involving surgery, risk of death and post-operative complications (Iezzoni, 2009). Presentation of specific risk-adjustment methods is beyond the scope of this chapter but some general methods include direct and indirect standardization, multiple regression analysis and other statistical techniques.
The selection of potential confounding factors needs to be done carefully, taking into account the ultimate purpose and use of the quality indicator that needs adjustment.
In fact, the choice of risk-adjustment factors is not a purely technical exercise but relies on assumptions that are often not clearly spelled out. For example, in several countries the hospital readmission rate is used as a quality indicator in pay for quality programmes (Kristensen, Bech & Quentin, 2015). If it is believed that age influences readmission rates in a way hospitals cannot affect, age should be included in the risk-adjustment formula. However, if it is thought that hospitals can influence elderly patients’ readmission rates by special discharge programmes for the elderly, age may not be considered a “risk” but rather an indicator for the hospitals to use for identifying patients with special needs. The same arguments apply also for socioeconomic status. On the one hand, there are good reasons to adjust for socioeconomic variables because patients living in poorer neighbourhoods tend to have higher readmission rates. On the other hand, including socioeconomic variables in a risk-adjustment formula would implicitly mean that it was acceptable for hospitals located in poorer areas to have more readmissions.
The assumptions and methodological choices made when selecting variables for risk-adjustment may have a powerful effect on risk-adjusted measured quality of care. Some critics (for example, Lilford et al., 2004) have argued that comparative outcome data should not be used externally to make judgements about quality of hospital care. More recent criticism of risk-adjustment methods has suggested
that risk-adjustment methods of current quality measurement systems could be evaluated by assigning ranks similar to those used to rate the quality of evidence (Braithwaite, 2018). Accordingly, A-level risk-adjustment would adjust for all known causes of negative consequences that are beyond the control of clinicians yet influence outcomes. C-level risk-adjustment would fail to control for several important factors that cause negative consequences, while B-level risk-adjustment would be somewhere in between.
3.11 Conclusion
This chapter has introduced some basic concepts and methods for the measure-ment of healthcare quality and presented a number of related challenges. Many different stakeholders have varying needs for information on healthcare quality and the development of quality measurement systems should always take into account the purpose of measurement and the needs of different stakeholders.
Quality measurement is important for quality assurance and accountability to make sure that providers are delivering good-quality care but they are also vital Table 3.7 Potential patient risk-factors
Demographic characteristics
• age
• sex/gender
• race and ethnicity
Socioeconomic factors
• educational attainment
• health literacy
• language(s)
• employment and occupation
• economic resources
• family characteristics and household composition
• housing and neighbourhood characteristics
• health insurance coverage
Attitudes and perceptions
• cultural beliefs and behaviours
• religious beliefs and behaviours, spirituality
• overall health status and quality of life
• preferences, values and expectations for healthcare services
Clinical factors
• acute physiological stability
• principal diagnosis
• severity of principal diagnosis
• extent and severity of co-morbidities
• physical functioning
• vision, hearing, speech functioning
• cognitive functioning
• mental illness, emotional health
Health-related behaviours and activities
• tobacco use
• alcohol, illicit drug use
• sexual practices (‘safe sex’)
• diet and nutrition
• physical activity, exercise
• obesity and overweight
Source: Iezzoni, 2009
for quality improvement programmes to ensure that these interventions lead to increases in care quality.
The development and use of quality measures should always be fit-for-purpose.
For example, outcome-based quality indicators, such as those used by the OECD, are useful for international comparisons or national agenda-setting but providers such as hospitals or health centres may need more specific indicators related to processes of care in order to enable quality improvement. The Donabedian frame-work of structure, process and outcome indicators provides a comprehensive, easily understandable model for classifying different types of indicator, and it has guided indicator development of most existing quality measurement systems.
Quality indicators should be of high quality and should be carefully chosen and implemented in cooperation with providers and clinicians. The increasing availability of clinical data in the form of electronic health records is multiplying possibilities for quality measurement on the basis of more detailed indicators.
In addition, risk-adjustment is important to avoid high-quality providers being incorrectly and unfairly identified as providing poor quality of care – and vice versa, to avoid that poor providers appear to be providing good quality of care.
Again, the increasing availability of data from electronic medical records may expand the options for better risk-adjustment.
However, most quality measurement initiatives will continue to focus – for reasons of practicality and data availability – only on a limited set of quality indicators. This means that one of the fundamental risks of quality measurement will continue to be important: quality measurement will always direct attention to those areas that are covered by quality indicators, potentially at the expense of other important aspects of quality that are more difficult to assess through quality measurement.
Nevertheless, without quality information policy-makers lack the knowledge base to steer health systems, patients can only rely on personal experiences or those of friends for choosing healthcare providers, and healthcare providers have no way of knowing whether their quality improvement programmes have worked as expected.
Quality information is a tool and it can do serious damage if used inappropri-ately. Seven basic principles of using quality indicators are summarized in Box 3.2. It is critical to be aware of the limitations of quality measurement and to be cautious of using quality information for quality strategies that provide powerful incentives to providers, such as public reporting (see Chapter 13) or P4Q schemes (see Chapter 14), as these may lead to potential unintended consequences such as gaming or patient selection.
Box 3.2 Seven principles to take into account when using quality indicators Principle 1: Indicators have to be fit-for-purpose
The choice of quality indicators should proceed from a clear definition of its intended purpose.
Indicators designed with an external focus (i.e. oversight, accountability, identifying outliers, patient choice) will require different characteristics from those designed with an internal focus (i.e.
quality improvement). For external use the quality measures should be sensitive to identify quality problems, and they should be capable of showing meaningful differences between providers. For internal use more specific quality measures are necessary to monitor progress over time and to provide signals that offer clear and actionable management responses.
Principle 2: Quality of measurement depends on quality of data and indicators The reliability of quality measures relates to the quality of the data on which they are based and the robustness of the method used to construct them. Reliability can be a concern where quality indicators are derived from databases that are only indirectly linked to the primary process of care delivery and data recording, for example, administrative billing data.
Principle 3: Quality measurement has limits
Quality of care has different dimensions (effectiveness, safety, patient-centredness) and one specific healthcare provider (for example, a hospital or GP practice) provides care via various processes involving many different professionals and technologies. Conclusions about all different quality aspects and all underlying services made on the basis of only a few indicators are likely to miss important non-measured aspects of care. Organizational context and local knowledge of confounding circumstances must be taken into account when interpreting even well-constructed indicators.
Principle 4: Outcome measures require risk-adjustment
Despite much progress, the validity of outcome measures is often debatable. Collecting information on outcomes like mortality and complications is useful but often it is hard to determine whether differences found are actually the result of differences in quality of care. For example, without risk-adjustment for complications and co-morbidities, differences in mortality found between hospitals may not be due to differences in the quality of care provided. One hospital may deal only with straightforward, uncomplicated patients whereas others (such as specialist centres) may treat the most complicated cases.
Principle 5: Composite indicators improve simplicity but may be misleading
Attempts have been made to construct composite indicators that summarize a broader suite of underlying measures. Although this approach has certain attractions – notably simplicity – the results can be misleading. Weaknesses of the underlying indicators are often disguised and the weighting between the various constituent indicators is often not based on empirical information or not reported at all. Thus, the summary “score” may suggest a clear result (for example, that
References
ACSQHC (2019). Indicators of Safety and Quality. Australian Commission on Safety and Quality in Health Care (ACSQHC): https://www.safetyandquality.gov.au/our-work/
indicators/#Patientreported, accessed 21 March 2019.
Baker DW, Chassin MR (2017). Holding Providers Accountable for Health Care Outcomes.
Annals of Internal Medicine, 167(6):418–23.
Braithwaite RS (2018). Risk Adjustment for Quality Measures Is Neither Binary nor Mandatory.
Journal of the American Medical Association, 319(20):2077–8.
Bray F, Parkin DM (2009). Evaluation of data quality in the cancer registry: Principles and methods.
Part I: Comparability, validity and timeliness. European Journal of Cancer, 45(5):747–55.
Busse R (2012). Being responsive to citizens’ expectations: the role of health services in responsiveness and satisfaction. In: McKee M, Figueras J (eds.) Health Systems: Health, wealth and societal well-being. Maidenhead: Open University Press/McGraw-Hill.
Calhoun C (2002). Oxford dictionary of social sciences. New York: Oxford University Press.
Campbell SM et al. (2008). Quality indicators for the prevention and management of cardiovascular disease in primary care in nine European countries. European Journal of Cardiovascular Prevention & Rehabilitation, 15(5):509–15.
Carinci F et al. (2015). Towards actionable international comparisons of health system performance:
expert revision of the OECD framework and quality indicators. International Journal for Quality in Health Care, 27(2):137–46.
one provider is better than another) but if underlying data and methods are weak, users may come to incorrect conclusions.
Principle 6: A league table raises interest but is not always fair
The same methodological limitations that apply to compound indicators also apply to league tables. Weaknesses in the underlying components may be masked, weighting is not necessarily user-based and ranking suggests real differences in the units being measured, i.e. hospitals, countries, etc. Additionally, without the presence of properly calculated confidence estimates, rank orders that imply absolute differences in quality may in fact be nothing more significant than chance. League tables, especially those published through official channels, should therefore be handled with care.
Principle 7: Be aware of gaming and unintended consequences
Overall reporting of information on quality of care can lead to performance improvement.
Nevertheless, reporting on certain aspects of care can lead to adverse effects such as gaming or outright cheating. For example, reporting on hospital mortality rates has in the past led hospital professionals to try to improve their rates by promoting that patients die elsewhere. Furthermore, if indicators focus on major diseases like diabetes and chronic heart failure, this may lessen interest in diseases that are less prominent in reporting and rewarding systems. Additionally, reporting on negative outcomes (safety, complications) should be balanced by reporting on positive outcomes (improved functioning, survival) – doing so will help to promote a balanced culture of risk control and risk taking in healthcare.
Source: based on OECD, 2010
Chan KS et al. (2010). Electronic health records and the reliability and validity of quality measures:
a review of the literature. Medical Care Research and Review, 67(5):503–27.
Cheng EM et al. (2014). Quality measurement: here to stay. Neurology Clinical Practice, 4(5):441–6.
Cylus J, Papanicolas I, Smith P (2016). Health system efficiency: how to make measurement matter for policy and management. Copenhagen: WHO, on behalf of the European Observatory on Health Systems and Policies.
Davies H (2005). Measuring and reporting the quality of health care: issues and evidence from the international research literature. NHS Quality Improvement Scotland.
Donabedian A (1980). The Definition of Quality and Approaches to Its Assessment. Vol 1.
Explorations in Quality Assessment and Monitoring. Ann Arbor, Michigan, USA: Health Administration Press.
EC (2016). So What? Strategies across Europe to assess quality of care. Report by the Expert Group on Health Systems Performance Assessment. European Commission (EC). Brussels:
European Commission.
Emilsson L et al. (2015). Review of 103 Swedish Healthcare Quality Registries. Journal of Internal Medicine, 277(1):94–136
Evans SM et al. (2009). Prioritizing quality indicator development across the healthcare system:
identifying what to measure. Internal Medicine Journal, 39(10):648–54.
Fitzpatrick R (2009). Patient-reported outcome measures and performance measurement. In:
Smith P et al. (eds.) Performance Measurement for Health System Improvement: Experiences, Challenges and Prospects. Cambridge: Cambridge University Press.
Forde I, Morgan D, Klazinga N (2013). Resolving the challenges in the international comparison of health systems: the must do’s and the trade-offs. Health Policy, 112(1–2):4–8.
Freeman T (2002). Using performance indicators to improve health care quality in the public sector: a review of the literature. Health Services Management Research, 15:126–37.
Fujisawa R, Klazinga N (2017). Measuring patient experiences (PREMS): Progress made by the OECD and its member countries between 2006 and 2016. Paris: Organisation for Economic Co-operation and Development (OECD).
Fujita K, Moles RJ, Chen TF (2018). Quality indicators for responsible use of medicines: a systematic review. BMJ Open, 8:e020437.
Gardner K, Olney S, Dickinson H (2018). Getting smarter with data: understanding tensions in the use of data in assurance and improvement-oriented performance management systems to improve their implementation. Health Research Policy and Systems, 16(125).
Goddard M, Jacobs R (2009). Using composite indicators to measure performance in health care. In: Smith P et al. (eds.) Performance Measurement for Health System Improvement:
Experiences, Challenges and Prospects. Cambridge: Cambridge University Press.
Hurtado MP, Swift EK, Corrigan JM (2001). Envisioning the National Health Care Quality Report. Washington, DC: National Academy Press.
ICHOM (2019). Standard Sets. International Consortium for Health Outcomes Measurement (ICHOM): https://www.ichom.org/standard-sets/, accessed 8 February 2019.
Iezzoni L (2009). Risk adjustment for performance measurement. In: Smith P et al. (eds.) Performance Measurement for Health System Improvement: Experiences, Challenges and Prospects. Cambridge: Cambridge University Press.
IQTIG (2018). Methodische Grundlagen V1.1.s. Entwurf für das Stellungnahmeverfahren.
Institut für Qualitätssicherung und Transparenz im Gesundheitswesen (IQTIG). Available at: https://iqtig.org/das-iqtig/grundlagen/methodische-grundlagen/, accessed 18 March 2019.
Kannan V et al. (2017). Rapid Development of Specialty Population Registries and Quality Measures from Electronic Health Record Data. Methods of information in medicine, 56(99):e74–e83.
Kelley E, Hurst J (2006). Health Care Quality Indicators Project: Conceptual framework paper.
Paris: Organization for Economic Co-operation and Development (OECD). Available at:
https://www.oecd.org/els/health-systems/36262363.pdf, accessed on 22/03/2019.
De Koning J, Burgers J, Klazinga N (2007). Appraisal of indicators through research and evaluation (AIRE). Available at: https://www.zorginzicht.nl/kennisbank/PublishingImages/Paginas/AIRE-instrument/AIRE%20Instrument%202.0.pdf, accessed 21 March 2019.
Kristensen SR, Bech M, Quentin W (2015). A roadmap for comparing readmission policies with application to Denmark, England and the United States. Health Policy, 119(3):264–73.
Kronenberg C et al. (2017). Identifying primary care quality indicators for people with serious
Kronenberg C et al. (2017). Identifying primary care quality indicators for people with serious