Aggregating information in composite indicators

Given the complexity of healthcare provision and the wide range of relevant quality aspects, many quality measurement systems produce a large number of quality indicators. However, the availability of numerous different indicators may make it difficult for patients to select the best providers for their needs and for policy-makers to know whether overall quality of healthcare provision is improving. In addition, purchasers may struggle with identifying good-quality providers if they do not have a metric for aggregating conflicting results from

Table 3.3 Strengths and weaknesses of different types of indicators Structure indicators Process indicators Outcome indicators STRENGTHS Easily available. Many

structural factors are evident and easily reportable Stable. Structural factors

are relatively stable and often easy to observe

Easily available. Utilization of health technologies is often easily measured Easily interpreted. Compliance

with process indicators can often be interpreted as good quality without the need for case-mix adjustment or inter-unit comparisons Attribution. Processes are

directly dependent on actions of providers Smaller sample size needed.

Significant quality deficiencies can be detected more easily

Unobtrusive. Care processes can frequently be assessed unobtrusively from stored data

Indicators for action. Failures identified provide clear guidance on what must be remedied

Focus. Directs attention towards the patient and helps nurture a “whole system”

perspective

Goals. Represent the goals of care more clearly Meaningful. More meaningful to

patients and policy-makers Innovation. Encourages

providers to experiment with new modes of delivery Far-sighted. Encourages

providers to adopt long-term strategies (for example, health promotion) that may realize long-term benefits Resistant to manipulation.

Less open to manipulation but providers may engage in risk-selection or upcoding to influence risk-adjustment

WEAKNESSES Link to quality is very weak. Can only

Salience. Processes of care may have little meaning to patients unless the link to outcomes can be explained Specificity. Processes

indicators are highly specific to single diseases or procedures and numerous indicators may be required to represent quality of care provided

Ossification. May stifle innovation and the development of new modes of care

Obsolescence. Usefulness may dissipate as technology and modes of care change Adverse behaviour. Can be

manipulated relatively easily and may give rise to gaming and other adverse behaviours

Measurement definition.

Relatively easy to measure some outcome aspects validly and reliably (for example, death) but others are notoriously difficult (for example, wound infection) Attribution. May be influenced

by many factors outside the control of a healthcare organization

Sample size. Requires large sample size to detect a statistically significant effect Timing. May take a long time to

observe

Interpretation. Difficult to interpret if the processes that produced them are complex or occurred distant from the observed outcome Ambiguity. Good outcomes can

often be achieved despite poor processes of care (and vice versa)

Source: authors’ compilation based on Freeman, 2002 and Davies, 2005

different indicators. As a result, some users of quality information might base their decisions on only a few selected indicators that they understand, although these may not be the most important ones, and the information provided by many other relevant indicators will be lost (Goddard & Jacobs, 2009).

In response to these problems, many quality measurement initiatives have devel-oped methods for combining different indicators into composite indicators or composite scores (Shwartz, Restuccia & Rosen, 2015). The use of composite indicators allows the aggregation of different aspects of quality into one measure to give a clearer picture of the overall quality of healthcare providers. The advan-tage is that the indicator summarizes information from a potentially wide range of individual indicators, thus providing a comprehensive assessment of quality.

Composite indicators can serve many purposes: patients can select providers based on composite scores; hospital managers can use composite indicators to benchmark their hospitals against others, policy-makers can use composite indicators to assess progress over time, and researchers can use composite indi-cators for further analyses, for example, to identify factors associated with good quality of care. Table 3.4 summarizes some of the advantages and disadvantages of composite indicators.

The main disadvantages of composite indicators include that there are different (valid) options for aggregating individual indicators into composite indicators and that the methodological choices made during indicator construction will influence the measured performance. In addition, composite indicators may lead to simplistic conclusions and disguise serious failings in some dimensions.

Furthermore, because of the influence of methodological choices on results, the selection of constituting indicators and weights could become the subject of political dispute. Finally, composite indicators do not allow the identification of specific problem areas and thus they need to be used in conjunction with individual quality indicators in order to enable quality improvement.

There are at least three important methodological choices that have to be made to construct a composite indicator. First, individual indicators have to be chosen to be combined in the composite indicator. Of course, the selection of indicators and the quality of chosen indicators will be decisive for the reliability of the over-all composite indicator. Secondly, individual indicators have to be transformed into a common scale to enable aggregation. There are many methods available for this rescaling of the results, including ranking, normalizing (for example, using z-scores), calculating the proportion of the range of scores, and grouping scores into categories (for example, 5 stars) (Shwartz, Restuccia & Rosen, 2015).

All of these methods have their comparative advantages and disadvantages and there is no consensus about which one should be used for the construction of composite indicators.

Thirdly, weights have to be attached to the individual indicators, which signal the relative importance of the different components of the composite indicator.

Potentially, the ranking of providers can change dramatically depending on the weights given to individual indicators (Goddard & Jacobs, 2009). Again, several options exist. The most straightforward way is to use equal weights for every indi-cator but this is unlikely to reflect the relative importance of individual measures.

Another option is to base the weights on expert judgement or preferences of the target audience. Further options include opportunity-based weighting, also called denominator-based weights because more weight is given to indicators for more prevalent conditions (for example, higher weights for diabetes-related indicators than for acromegaly-related indicators), and numerator-based weights which give more weight to indicators covering a larger number of events (for example, higher weight on medication interaction than on wrong-side surgery). Finally, yet another option is to use an all-or-none approach at the patient level, where a score of one is given only if all requirements for an individual patient have been met (for example, all five recommended pre-operative processes were performed).

Again, there is no clear guidance on how best to construct a composite indica-tor. However, what is important is that indicator construction is transparent and that methodological choices and rationales are clearly explained to facilitate understanding. Furthermore, different choices will provide different incentives for improvement and these need to be considered during composite construction.

Table 3.4 Advantages and disadvantages of composite indicators

Advantages Disadvantages

• Condense complex, multidimensional aspects of quality into a single indicator.

• Easier to interpret than a battery of many separate indicators.

• Enable assessments of progress of providers or countries over time.

• Reduce the number of indicators without dropping the underlying information base.

• Place issues of provider or country performance and progress at the centre of the policy arena.

• Facilitate communication with general public and promote accountability.

• Help to construct/underpin narratives for lay and literate audiences.

• Enable users to compare complex dimensions effectively.

• Performance on indicator depends on methodological choices made to construct the composite.

• May send misleading messages if poorly constructed or misinterpreted.

• May invite simplistic conclusions.

• May be misused, if the composite construction process is not transparent and/or lacks sound statistical or conceptual principles.

• The selection of indicators and weights could be the subject of political dispute.

• May disguise serious failings in some dimensions and increase the difficulty of identifying remedial action, if the construction process is not transparent.

• May lead to inappropriate decisions if dimensions of performance that are difficult to measure are ignored.

Source: based on OECD, 2008