Explainability in Medical AI Legal Regulation of Explainability in Clinical Decision Support Systems

(1)

Explainability in Medical AI

Legal Regulation of Explainability in Clinical Decision Support Systems

Candidate number: 9016

Submission deadline: 1 December 2021 Number of words: 17.858

(2)

i

Abbreviations

ACM Association for Computing Machinery ADHD Attention Deficit- / Hyperactivity Disorder

AI Artificial Intelligence

AI Act European Commission Proposal for an Artificial Intelligence Act (COM(2021) 206 final)

Art29WP Article 29 Data Protection Working Party (from 2018 onwards replaced by EDPB)

CEN European Committee for Standardisation

CENELEC European Committee for Electrotechnical Standardisation CDSS Clinical Decision Support System(s) (as defined in Chapter 2.3) CFR Charta of Fundamental Rights of the European Union

CJEU Court of Justice of the European Union

DARPA Defense Advanced Research Projects Agency

DCMS Department for Digital, Culture, Media and Sport (UK)

DL Deep Learning

DPIA Data Protection Impact Assessment

EC European Commission

E.g. Example given/for example

EP European Parliament

EDPB European Data Protection Board

f. following page

ff. following pages

GDPR General Data Protection Regulation (Regulation (EU) 2016/679) GPSD General Product Safety Directive (Directive 2001/95/EC)

HLEG-AI High-Level Expert Group on Artificial Intelligence set up by the European Commission in June 2018

ICO Information Commissioner's Office, UK IEC International Electrotechnical Commission

IP Intellectual Property

ISO International Organisation for Standardisation

IVDR In Vitro Diagnostic Device Regulation (Regulation (EU) 2017/746) MDCG Medical Device Coordination Group

MDR Medical Devices Regulation (Regulation (EU) 2017/745)

ML Machine Learning

NCSC National Cyber Security Center, United Kingdom

OECD Organisation for Economic Co-operation and Development XAI Explainable Artificial Intelligence

(5)

1

1 Introduction

1.1 General remarks

Medical AI¹ can improve our healthcare systems by lowering the costs and at the same time ensuring higher quality of medical care.² It can e.g. detect early signs of cardiac arrest,³ assist in the treatment of cancer patients,⁴ and be of valuable support to clinicians in "over-burdened and under-staffed healthcare settings".⁵ An increasing number of Medical AI systems and applications are being invented and developed. Yet, Medical AI, especially if based on Machine Learning (ML), is not widely deployed in clinical settings.⁶ ML-based systems that take into account over 100 factors are only used in 2% of decisions made in healthcare, whereas 80% of healthcare decisions are still made based on approximately seven factors, inter alia because the latter decisions are more transparent.⁷ Even though ML-based systems might perform better, they are nevertheless not yet used on a large scale due to a lack of understanding of their outputs and their non-explainability.⁸

"Explainability can be understood as a characteristic of an AI-driven system allowing a person to reconstruct why a certain AI came up with the presented predictions."⁹ Research on explainability methods started as far back as the 1970s.¹⁰ As systems became more complex and opaque due to new technological developments,¹¹research increased again in the last few years.¹²Ex- plainability was also identified as one of the main topics of AI implementation,¹³ and there is no lack of ideas or solutions for making AI systems more explainable.¹⁴ However, incentives for producers to provide explanations for AI are still lacking for several reasons.¹⁵ Explainabil- ity is often seen as a non-functional requirement.¹⁶ It is not needed for the system to function

1 Medical AI means AI employed in a medical context. This includes physical AI like robots that assist in surgery and intelligent protheses, or virtual AI such as systems used for decision support in treatment and diagnosis (Amisha et.al., "Artificial intelligence in medicine", 2328). This paper only refers to virtual Medical AI.

2 Higgins, Madai, "From Bit to Bedside", 1.

3 Sharma et.al., "Reasonable Explainability", 3.

4 Payrovnaziri et.al., "Explainable artificial intelligence models", 1177f. and references cited therein.

5 Pierce et.al., "A riddle, wrapped in a mystery", 2.

6 Higgins, Madai, "From Bit to Bedside", 1.

7 ACM, "Healthcare AI", minute 21.00 (percentages relate to the year 2018); Tonekaboni et.al., "What Clinicians Want", 2.

8 Shortliffe, Sepúlveda, "Clinical Decision Support", 2199.

9 Amann et.al., "Explainability for AI", 2. A more detailed definition of explainability follows (Chapter 2.2).

10 See e.g. Shortliffe, Buchanan, "Inexact Reasoning in Medicine".

11 Köhl et.al., "Non-Functional Requirement", 363.

12 Arrieta et.al., "Explainable Artificial Intelligence", 83.

13 Hamon et.al., "Robustness and Explainability", 2 and 4; HLEG-AI, "Ethics Guidelines", 13 and 18.

14 Arrieta et.al., "Explainable Artificial Intelligence", 87-99.

15 ICO, "Project Explain", 22.

16 Köhl et.al., "Non-Functional Requirement".

(6)

2 but only implemented due to legal or moral considerations. Engineers and developers, however, usually focus on functional requirements.¹⁷ Furthermore, even with eventual long-term benefits, the implementation of explainability is associated with high costs.¹⁸ In addition, and this will be the focus of this thesis, policy documents clarifying explainability requirements are still lacking.¹⁹

1.2 Methodology, Scope and Research Question

This research is limited due to time and space constraints and is almost exclusively based on English language sources such as academic journals, books, and reports or information provided by ISO, IEC, and European institutions that were available online or through the University library. Especially the section on technical standards (Chapter 3.5.4) is only based on information available online and might therefore not be complete since I did not have access to the relevant standards from the ISO or IEC standardisation bodies. This work does not claim to cover the whole field of explainability in the medical context and all the conclusions drawn in the course of this work are tentative and not final. For simplification, only the male pronoun forms are used, but they are meant to include all genders.

Motivations to provide explainability have been more focused on the decision-subject instead of the individual using the decision-support tool.²⁰ They have so far mainly come from a legal or computer science perspective to either prevent and contest²¹ discriminatory and non-compli- ant decisions or to better control, update and adjust the AI system and make it more accurate.²² This work considers another perspective, namely that of the individuals who are actively using the AI system. Individuals working in medical education and research, as well as clinicians, require explainability of Medical AI.²³ This work only focuses on clinicians (e.g. doctors, nurses, administrative staff) and their needs for explainability in a clinical setting in their eve- ryday tasks. It does not include any medical systems or applications used by consumers or patients in a more private setting,²⁴ and puts the focus on Clinical Decision Support Systems (CDSS), as one category of Medical AI. This work only analyses the regulatory field in an EU context. Explainability can also have an impact on data privacy or cybersecurity, as shown in

17 Leenes, Lucivero, "Laws on Robots", 216f.

19 Beaudouin et.al., "AI Explainability", 2; Köhl et.al., "Non-Functional Requirement", 363.

20 Edwards, Veale, "Slave to the algorithm", 54.

21 Ploug, Holm, "Contestable AI diagnostics". They focus on the patient (decision-subject) and analyse the contestable aspects of AI diagnostic decisions and the corresponding explainability requirements.

22 Brkan, Bonnet, "Legal and Technical Feasibility", 19.

23 Holzinger et.al., "Causability and explainability", 3.

24 Gerke et.al., "Ethical and legal challenges", 301.

(7)

3 Chapter 4.3, but this work only focuses on explainability. It leaves data privacy, cybersecurity, or any other issues with regard to AI and CDSS aside.

This thesis asks how to best achieve explainability of increasingly automated and autonomous CDSS through legal regulations in a way that ensures clinicians' demands for the explanations are met.

Therefore, it analyses why an explanation of CDSS is needed and based on that what explainability requirements clinicians demand of such systems. Then it shows that current regulations or regulatory proposals only partly require that those demands are met. Next, because current regulations are not sufficient, arguments for and against including a statutory provision on Ex- plainability-by-Design are discussed. This work analyses whether such an approach can ensure that clinicians' demands on explanations of CDSS are met. Finally, it briefly reflects on how a regulation could look like and how the requirements must be set out. A future regulation would have to provide developers with the tools and information to design their CDSS so that its recommendation is explainable to clinicians.

It concludes that the legal regulation of explainability, even in the small field of CDSS with clinicians as the addressees of the explanation, is a very complex task that has not yet been undertaken sufficiently. The new AI regulatory trend of adding 'by-Design' to a value can be an important first step. However, it is not the cure-all either, as it also cannot ensure that clinicians' very specific needs for explainability of CDSS are met. A legal provision on Explainability-by- Design would in addition have to refer to technical standards or guidelines that can set out in more detail what explainability requirements are needed in CDSS.

2 Terms and Definitions

There is no uniform definition for terms such as regulation and explainability, as shown below.

This section does not claim to find the one, universal definition but defines these terms, as well as CDSS, for this work.

2.1 Regulation and technical standards

The term regulation is mainly used in a narrow sense, as it refers only to the legal rules, or rules referenced in statutes and legal instruments, that can be enforced through the legal system.²⁵ The definition is kept narrow because, due to limited space, this paper only analyses whether legal instruments require that CDSS must be explainable in a way that is useful to clinicians. It

25 Yeung, "Design for the Value", 448.

(8)

4 does not discuss a regulation through other tools, such as the market or social norms,²⁶ that would require a broader definition of the term 'regulation'.

The only exception is the analysis of the by-Design approach. While this work assumes that an Explainability-by-Design requirement would be embedded in a legal regulation²⁷ which would be part of the narrow definition of regulation, it inherently involves the regulatory aspect of code or architecture²⁸. Therefore, the analysis of the by-Design regulation in Chapter 4 also refers to code, or design, as a regulatory tool. For this part, one could define regulation more broadly as a "process to produce a broadly defined outcome [...] directed at a sphere of social activity according to defined standards or purposes that affect others".²⁹ The outcome to be produced would be an explainable CDSS that can be used for the activity of practicing medicine and is developed according to legal provisions (defined standards) that set out the explainability requirements. This would affect others, as an explainable CDSS allows clinicians to validate the recommendations of the system and to ensure that patients give their informed consent (see Chapter 3.3). I deliberately did not choose Julia Black's more common definition of regulation because she defined it as an "attempt to alter the behaviour of others".³⁰ An obligation for explainability of CDSS might aim to alter the behaviour of developers by forcing them to implement explainability in their CDSS but the focus is on how the explainability requirement affects others, such as clinicians and patients.

Standards in this paper refer to internationally agreed upon (technical) standards drawn up by experts in recognized standardisation bodies like ISO or IEC. Standards aim to set out (technical) specifications to harmonise the design, manufacturing, installation, or testing of certain products or processes. Compliance with such standards is typically voluntary³¹ but often laws and regulations refer to these technical standards. Standards can be updated faster and include more details than the law, which should remain technology-neutral.³²

2.2 Explainability of AI

AI is a field of computer science that uses data processing to "perform functions normally associated with human intelligence, such as reasoning, learning, and self-improvement".³³

26 Lessig, "The Law of the Horse", 507ff.

27 Similar to Data Protection-by-Design (Art. 25 GDPR).

28 Lessig, "The Law of the Horse", 507ff.

29 Yeung, "Design for the Value", 455.

30 Black, "Critical Reflections on Regulation", 26.

31 Art. 2(1) Regulation (EU) No 1025/2021.

32 IEC, "Understanding standards"; ISO, "Standards"; Kamara, "Co-regulation in EU", 10.

33 ISO/IEC 2382:2015, 1 Terms and Definitions nr. 2121393.

(9)

5 Simpler AI models, that operate on static, rule-based systems, are easier to understand.³⁴ AI models based on ML and similar concepts, on the other hand, do not have explicit rules that could be easily reconstructed and are therefore often opaque.³⁵ AI models can be opaque for several reasons. The developers of the algorithm might want to keep them a trade secret, only a few experts can understand coding language and, most importantly and most problematic, the models are complex and the difference between the "mathematical optimization in high-dimen- sionality characteristics" of ML and human reasoning and interpretation makes them difficult to understand.³⁶ Deep Learning (DL) models, as part of ML, are built in a way that allows them to learn from data. They construct the rules and logic by themselves to reach a predefined outcome with the highest possible accuracy and performance. Their focus is therefore performance accuracy, not interpretability.³⁷ ML models are based on millions of data and calculate their output based on the correlations between those data. Human reasoning, on the other hand, looks ultimately for causality and contrastive explanations (why X instead of Y), not solely for correlations.³⁸ For these reasons, ML-based systems are less understandable, even to the developers who built them.³⁹

Explanations could help to better understand these opaque AI systems, as they help to "gain further insight or evidence" of how a decision was reached.⁴⁰ Explanations can improve trust in the system, uncover possible errors or help to justify decisions. An explanation is essential for individuals to make informed decisions based on the recommendation of an AI system.⁴¹ There is no single, universal definition for explainability and the term is often used interchange- ably with legibility, interpretability, or transparency.⁴² For a system to be explainable, it must be both transparent and faithful,⁴³ (accurate, complete, and detailed information) as well as comprehensible (clear, readable, and understandable to humans). An individual must be able to understand "the functionality, the impact, the consequences and the rationales of decision-making processes".⁴⁴ Social sciences or psychology also emphasise the need for explanations to be

34 Brkan, Bonnet, "Legal and Technical Feasibility", 23f.

35 Hamon et.al., "Robustness and Explainability", 4, see also 10f. for a more detailed description of ML.

36 Burrell, "How the machine 'thinks'", 2-5.

37 Edwards, Veale, "Slave to the algorithm", 26.

38 Holzinger et.al., "Causability and explainability", 3; Brkan, Bonnet, "Legal and Technical Feasibility", 28.

39 London, "Black-Box Medical Decisions", 16f. For more details on opacity in ML see e.g. Bygrave, "Machine Learning, Cognitive Sovereignty", 3-9; Burrell, "How the machine 'thinks'".

40 Liao et.al., "Questioning the AI", 5.

41 Köhl et.al., "Non-Functional Requirement", 363.

42 Murdoch et.al., "Interpretable machine learning", 22071f.; Markus et.al., "The role of explainability", 3.

43 Markus et.al., "The role of explainability", 3.

44 Malgieri, Comandé, "Right to Legibility", 245 and 264f.

(10)

6 adapted to the user and the context.⁴⁵ The explanation must consequently be relevant in the specific context and their form of delivery (visualization, natural language, or mathematical equations) tailored to the audience to guide their next actions.⁴⁶

An explainable system can therefore be defined as "[a] system S [that] is explainable by means M with respect to aspect Y of an explanandum X, for target group G in context C, if [...] M is able to produce an E [explanation] in context C [...], [that is understandable] for [any repre- sentative of] G in C."⁴⁷ This definition takes account of the fact that, depending on the context and addressee, different explanations are needed and bears in mind that 'understanding' is an important part of the explanation. Moreover, the means for the explanation can, according to this definition, be built into the system (interpretable model) as well as be independent of it (post-hoc explanation).⁴⁸

The different technical explainability methods and techniques⁴⁹ will not be analysed further here. Nevertheless, it is important to understand the difference between interpretable models and post-hoc explanations of models. A post-hoc explanation is usually provided for models that are black boxes and not themselves interpretable (typically ML and DL models). These explanations can be seen as a second, accompanying model that is built to explain the first model without having insights into how it works. An interpretable system, on the other hand, is built in a way or uses such AI models that make it possible to explain its recommendations without the need for additional systems. However, interpretable models may not be as accurate in their predictions and might not always be possible to build.⁵⁰

2.3 Clinical Decision Support Systems

CDSS are employed in the medical context to support a clinician, like a doctor, nurse, or ad- ministrator, in their medical decision making e.g. regarding diagnostics, disease management, or drug control. They are used as an additional tool, providing recommendations or suggestions that the clinician can combine with his own experience and judgement to make decisions. CDSS often use existing digital health data, such as electronic health records, and compare this data with the data entered about an individual patient.⁵¹

45 Hamon et.al., "Robustness and Explainability", 24; Miller, "Explanation in artifical intelligence", 3.

46 Murdoch et.al., "Interpretable machine learning", 22071f.

47 Köhl et.al., "Non-Functional Requirement", 365f. This definition is a combination of their definitions of 'explanation' and 'explainable system'.

48 Köhl et.al., "Non-Functional Requirement", 365f.

49 See e.g. Arrieta et.al., "Explainable Artificial Intelligence", 87-99; Brkan, Bonnet, "Legal and Technical Feasibility", 33-38.

50 Markus et.al., "The role of explainability", 3; Hamon et.al., "Robustness and Explainability", 13.

51 Sutton et.al., "Clinical Decision Support Systems", 1f.

(11)

7 CDSS that make use of AI can be automated or autonomous.⁵² CDSS that use rule-based models with if-then statements and follow expert medical knowledge are automated and can be understood and explained more easily. Autonomous CDSS, on the other hand, are based on more complex and opaque AI. They use ML and DL to train the algorithm on data sets from patients.

That way, predictions or observations can be made that would have been impossible without the use of AI and the massive amounts of data it can analyse. While a lot of CDSS today still operate with a rule-based system, CDSS based on more complex AI are developed and likely to be used in the near future.⁵³ This paper will be mainly concerned with the more opaque autonomous CDSS, as automated CDSS usually do not face the same issues in terms of explainability.

3 Explainability in Clinical Decision Support Systems

The form of an explanation and its degree of detail depend on several factors. Those consist of the context, including impact factors and the degree of risk, the addressees of the explanation and the information they need to know, including operational factors like user trust and system validation, the regulations that the AI must comply with, the technical possibilities and the economic considerations.⁵⁴ As stated in the introduction, this paper focuses on CDSS in a medical context with clinicians as the main addressees of the explanation in an EU regulatory field.

While it will not particularly focus on current technical solutions and possibilities,⁵⁵ it tries to only consider or propose options that are or can be technically feasible, at least for most CDSS.

3.1 Current uses and future developments

CDSSs are employed for more accurate and faster diagnosis, better and more individualized treatment, or more efficient management of organisational tasks and can help to improve patients' treatments or even safe lives, as shown with a few examples below.

CDSS are often used in fields where medical research has not yet found any solutions, like in Alzheimer's Disease research,⁵⁶ or in the field of oncology to identify cancer cells and develop individualized cancer treatments.⁵⁷ They are used to predict or detect diseases earlier, especially

52 Brkan, Bonnet, "Legal and Technical Feasibility", 23f.

53 Sutton et.al., "Clinical Decision Support Systems", 1f. and references cited therein; Amann et.al., "Explainability for AI", 2.

54Beaudouin et.al., "AI Explainability", 3f.; Sharma et.al., "Reasonable Explainability", 5.

55 For technical details on different explanation forms for AI systems see e.g.: Brkan, Bonnet, "Legal and Technical Feasibility", 33-38 and 46-50; Arrieta et.al., "Explainable Artificial Intelligence", 87-99; Hall et.al., "Inter- preting Machine Learning"; Payrovnaziri et.al., "Explainable artificial intelligence models".

57 Payrovnaziri et.al., "Explainable artificial intelligence models", 1177f. and references cited therein.

(12)

8 in time-critical situations. CDSS can detect sepsis which is one of the leading causes of death,⁵⁸ or be employed in cardiology to detect early signs of cardiac arrest.⁵⁹ But they are also used in non-time-critical areas e.g. to diagnose balance disorders like Meniere's disease.⁶⁰ From an organisational standpoint, they can help to allocate medical resources by predicting the likelihood of a patient to be readmitted or the need for end of life care,⁶¹ ranking the risk of incoming patients in an Emergency Department,⁶² or detecting cardiac arrest signs in the voice of a caller of the emergency number.⁶³ CDSS functions include prescription and drug control, disease management, alarm systems, clinical management, diagnostic support including imaging, la- boratory and pathology, patient decision support, workflow improvement, and many more.⁶⁴ Describing all use cases of Medical AI and CDSS would go beyond the scope of this paper.⁶⁵ As outlined in the introduction, Medical AI is still scarcely used at present time, inter alia due to its opacity. To benefit from its advantages, the matter of explainability should be regulated to contribute to a greater use of these systems.

3.2 Critical Reflection on the Need for Explanations

Some argue that there is no need for explainability in Medical AI, as shown below. If this were true, it would make a regulation of explainability for CDSS superfluous.

In medicine, actions are sometimes taken not based on an explicit understanding of how some- thing works and what factors influence it but based on empirical evidence from randomized clinical trials and medical experience, e.g. in the past the use of aspirin as a pain killer,⁶⁶ or nowadays the prescription of lithium as a mood stabilizer. Moreover, an overreliance on theo- retical, causal explanations instead of practical evidence can, in rare cases, even lead to mistakes in medicine that harm patients. One example is the practice of removing as much tissue from the breast as possible to lower the chances of cancer reoccurring. A theory proved wrong after intensive clinical testing. Furthermore, clinicians are already now not able to evaluate the

58 Henry et.al., "TREWScore for septic shock".

60 Bussone et.al., "The Role of Explanations", 161.

61 Ahmad, "Interpretable Machine Learning in Healthcare", 1.

62 Tonekaboni et.al., "What Clinicians Want", 4.

63 Vincent, "AI that detects cardiac arrest".

64 Sutton et.al., "Clinical Decision Support Systems".

65 For a list of AI applications in medicine and further references please refer to these works and the references cited therein: Obermeyer, Emanuel, "Predicting the Future"; Kieslich et.al., "Threats of Artifiical Intelligence Scale", 6; Payrovnaziri et.al., "Explainable artificial intelligence models", 1177f.; Ordish et.al., "Algorithms as medical devices", 21.

66 Goldberg, "Aspirin".

(13)

9 immense amounts of medical factors and pathophysiological evidence when assessing the individual patient.⁶⁷ Hence, some opacity remains even in traditional medicine.

However, there is often a difference between this opacity in medical decision-making and the opacity generated by AI. In traditional medicine, some experts can still understand and explain why e.g., a certain blood test is better than another, even if the clinician himself cannot explain it. Moreover, there might still be some useful information, such as facts about general characteristics that led to one or the other diagnosis. Those would enable a clinician to explain his reasoning to the patient. With opaque CDSS, however, this might not be possible. Even data scientists as experts cannot always explain the AIs' reasoning, and statistics and correlations might be based on random data like holiday trips or hair colour. Those do not enable a clinician to explain the system's reasoning to the patient.⁶⁸ Moreover, empirical evidence in medicine is often derived from years of effective use, but such historical data has not yet been established for CDSS.⁶⁹

Another reason not to require explanations would be that CDSS must pass strict validation or certification processes⁷⁰ that prove their performance accuracy. If the CDSS predictions are accurate, explainability might not be considered relevant.⁷¹ Yet, even with certified CDSS, clinicians might still need to double-check the recommendations and evaluate their plausibility (Chapter 3.3.2).⁷²

It is true that, in low-risk applications, explanations might be disregarded in favour of higher accuracy or lower costs. For cost predictions or predictions of the number of patients (not their condition) arriving at the Emergency Department, explanations might not be necessary. How- ever, with higher risks⁷³ and closeness to the patient, e.g. predictions on disease risk or progres- sion, explanations and information gain relevance.⁷⁴ Explanations are also necessary when consequences are far-reaching (e.g. recommendation of surgery) or the cost of a mistake is high (e.g. wrong classification of a malignant tumour).⁷⁵ Where necessary and possible, an

67 London, "Black-Box Medical Decisions", 17f.

68 Bjerring, Busch, "Patient-Centered Decision-Making", 363-367.

69 Cutillo et.al., "Machine intelligence in healthcare", 2.

70 As e.g. outlined in the MDR, see Chapter 3.5.3.

71 Amann et.al., "Explainability for AI", 5.

72 Reimer et.al., "Going beyond Explainability", 189.

73 For one way how to classify risks (likelihood and magnitude) see Terry, "Informed Consent in Clinical Medicine", 564.

74 Ahmad et.al., "Interpretable Machine Learning in Healthcare", 3.

75 ACM, "Healthcare AI", minute 27.

(14)

10 explanation should not be dismissed just because this has not always been the norm in traditional medicine.⁷⁶

3.3 Reasons for Explainability

Since clinicians will be the ones actively working with CDSS, their demands are relevant and should be taken into account. This section reflects on clinicians' needs for explainability in their daily work.

3.3.1 Informed Consent and Patient Autonomy

The principle of informed consent is set out in Art. 3(2)(a) CFR, which states that "[i]n the fields of medicine and biology [...] the free and informed consent of the person concerned" must be respected. Informed consent is also part of the principle of autonomy in biomedical ethics.⁷⁷ It includes that the patients' values and concerns are taken into account⁷⁸ and that doctor and patient are seen as partners in the decision-making. Sufficient information and explanations are required for the patient to make a free informed decision about the medical acts to be per- formed.⁷⁹ This means the clinician must inform the patient comprehensively about the processes, including the medical information such as his symptoms and congruent diagnosis, and the decisions to be made, and the patient must understand this information.⁸⁰

The meaning of 'informed consent' is not completely clear in practice. The ever-growing complexity of the medical field means that an explanation down to the last detail often cannot be given to the patient. On the one hand, the introduction of AI might therefore not change much in practice. One could argue that a patient must only be broadly informed about how the AI works, its core principles, and possible risks that could occur.⁸¹ On the other hand, more information on ML techniques, data inputs, possible biases, or the use of and reliance on AI might have to be given to the patient for him to give valid informed consent.⁸² This information could help to give scientific evidence and explain the connection between this evidence and the patient's clinical condition which is an integral part of shared decision making⁸³ and bears relevance in an otherwise often subjective decision-making process of selecting a health care

76 Ahmad et.al., "Interpretable Machine Learning in Healthcare", 3.

77 Amann et.al., "Explainability for AI", 7; Beauchamp, Childress, "Principles of Biomedical Ethics".

78 Epstein et.al., "Patient-Centered Health Care", 1491.

79 Schneeberger et.al., "Legal Framework Medical AI", 219f.

80 Sugarman, "Missing the Informed in Consent", 319; Amann et.al., "Explainability for AI", 3; Terry, "Informed Consent in Clinical Medicine", 564.

81 Schneeberger et.al., "Legal Framework Medical AI", 219f.; Amann et.al., "Explainability for AI", 3f. and 7.

82 Gerke et.al., "Ethical and legal challenges", 301.

(15)

11 provider or clinician.⁸⁴ Such explanations usually rely on connections between certain symptoms, diagnoses, and treatment options. Non-explainable AI cannot provide these connections.

Hence, the clinician cannot explain to the patient why he reached this diagnosis and why one treatment is recommended over another. Yet, those facts are important for the patient to make his informed decision.⁸⁵

Consequently, the CDSS recommendation should, wherever possible, be explainable and understandable and enable a clinician to ensure that the informed consent of the patient is given.

Such an explanation could include information on training methods, on how a specific decision was derived from input data, or on the weight placed on different values.⁸⁶ And while the con- cept of 'informed consent' and shared decision-making can rarely be fully achieved in practice today,⁸⁷ this does not mean that future CDSS should not strive to come as close to it as possible.

Otherwise, opaque AI models would be a step back to where the doctor, or in this case the AI, 'knows best' and makes decisions individually.⁸⁸

3.3.2 Clinician Trust and Outcome Validation

When asking a medical expert for his opinion, clinicians expect that he can justify his decision based on medical knowledge, including the factors that influenced the decision. The same can be expected from CDSS that act as 'experts' and give recommendations.⁸⁹ Medical decisions, including diagnoses and treatments, should never be based on a simple Yes/No classification by an algorithm but should be informed decisions taken by the clinician, hence an explanation of the decision is necessary.⁹⁰

An explanation of how a decision was reached by the CDSS and what factors influenced the (specific) outcome allows the clinician to evaluate the CDSS recommendation by verifying it against his medical judgement and experience. This, in turn, leads to the clinician understanding the CDSS recommendation, being able to explain the decision to the patient, trusting the system, and taking its recommendations into account.⁹¹ Especially in cases of disagreement between the clinician's judgement and the CDSS recommendation, an explanation is relevant. A clinician often does not have the time for a detailed analysis to find the reasons for this discrepancy. He

84 Schneeberger et.al., "Legal Framework Medical AI", 220.

85 Bjerring, Busch, "Patient-Centered Decision-Making", 357f. and 360-363.

86 Schneeberger et.al., "Legal Framework Medical AI", 219f.; Amann et.al., "Explainability for AI", 3 and 6f.

88 McDougall, "Computer knows best?", 158.

89 Swartout, "XPLAIN", 286.

90 Böhle et.al., "Layer-Wise Relevance Propagation", 2.

91 Amann et.al., "Explainability for AI", 5; Avati et.al., "Improving palliative care", 61f.; Diprose, "Physician understanding", 595f.

(16)

12 might therefore either follow or not follow the CDSS recommendation against his better judgement so as not to be held accountable.⁹² This over- or under-trust⁹³ can lead to wrong decisions.

When the clinician does not understand the CDSS recommendation, it will be just one more piece of information among the many others that a stressed clinician must process.⁹⁴ He might either place less weight on it (under-trust), even though following the recommendation might have benefited the patient. Or he might place more weight on a possibly incorrect recommendation (over-trust), as part of a 'machine/automation bias' symptom,⁹⁵ without first validating the recommendation.⁹⁶ This trust also reaches over into the clinician-patient relationship. The patient will trust the system more if he (or the clinician) can understand and explain it and is, therefore, more likely to consent to its use.⁹⁷

A validation of the result through detailed information on the input factors that led to a recommendation is also necessary to ensure that the correct actions are taken. E.g. a CDSS alerting the clinician in case of acute kidney injury might be triggered because of decreased urine production (one symptom for a kidney injury). This would be treated with a high fluid intake.

However, if the decreased urine production had another cause, a fluid treatment can have fatal consequences. Therefore, a detailed explanation of how an input factor led to the recommendation is essential to ensure that the correct steps are taken, and the right treatment is adminis- tered.⁹⁸

This goes hand in hand with upholding the biomedical ethical principles of beneficence and non-maleficence.⁹⁹ These principles state that the optimal outcome for the patient must be reached and no (un)intentional harm should be inflicted upon the patient due to the wrong use of medical means. The explanation allows clinicians to assess whether the system's recommendations might have to be adapted to the individual patient to benefit him better and makes him confident in following the CDSSs recommendations.¹⁰⁰

While bias in AI can be greatly reduced by ensuring diverse training data, building systems without any bias is unlikely to be accomplished. Therefore, another part of a clinician evaluating the outcome is to make sure it is not biased and therefore leads to a wrong diagnosis or

93 Bussone et.al., "The Role of Explanations", 161 and references cited therein.

95 Challen et.al., "Clinical Safety", 234.

96 Tonekaboni et.al., "What Clinicians Want", 6.

97 Bjerring, Busch, "Patient-Centered Decision-Making", 365-367.

99 Amann et.al., "Explainability for AI", 7; Beauchamp, Childress, "Principles of Biomedical Ethics".

(17)

13 treatment.¹⁰¹ Bias in itself is just a "mathematical and statistical consequence of an algorithm and its data".¹⁰² It becomes problematic when it leads to unfair discrimination because it con- flicts with legal or ethical values. For this paper, bias means that the CDSS will perform worse or wrong for one group of patients or individual patients compared to another group. This can be because some groups are not reflected in the CDSS training data (explicit bias from sam- pling), because the data that is used e.g. from electronic health records is incomplete, outdated, or already biased as it was collected based on human decisions (social bias),¹⁰³ or because correlations are drawn without taking the context into account (implicit bias).¹⁰⁴

An explanation revealing the influence of input factors on the output can help the clinician to validate the CDSS recommendation against his judgement and detect situations, contexts, or patient groups where the system should not be employed because the training data does not reflect the situation or patient.¹⁰⁵ One example of a recommendation based on implicit bias is a CDSS that predicts the risk of death for pneumonia patients. It classified patients suffering from asthma with a lower risk than healthy patients. This ranking is correct if one takes standard medical practice into account where asthma patients will be immediately admitted to the Inten- sive Care Unit and closely monitored and therefore have higher chances of survival. However, it would be incorrect if it is used to decide whether to admit the patient to the hospital. Asthma patients must be admitted in any case, but the CDSS might suggest the opposite by classifying them at a lower risk. Here, having an explanation and the context of why asthma patients are classified at lower risk is important for a clinician to take the right action.¹⁰⁶This example shows the relevance of an explanation for discovering and distinguishing between a mere (wrong) correlation (lower risk due to asthma) and the causation (lower risk due to the preferred medical treatment of asthma patients in the training data).¹⁰⁷

3.3.3 Liability

While liability is an important topic, it will only be covered broadly here, as it is not part of a clinicians' daily work with CDSS.¹⁰⁸

101 Amann et.al., "Explainability for AI", 5 and 7f.

102 Fletcher et.al., "Addressing Fairness, Bias", 6.

103 Parikh et.al., "Addressing Bias", 2377.

104 Fletcher et.al., "Addressing Fairness, Bias", 6.

105 Amann et.al., "Explainability for AI", 5 and 7f.

106 London, "Black-Box Medical Decisions", 19; Caruna et.al., "Predicting Pneumonia Risk", 1721f.

107 Ordish et.al., "Algorithms as medical devices", 26.

108 For more information on current developments regarding liability of AI systems: EC, "COM(2020) 64 final".

(18)

14 Establishing the liable party (manufacturer, software developer, clinician, hospital, IT provider, etc.) can be difficult when AI systems are involved.¹⁰⁹ Clinicians can in general be held liable for treatment errors if the patient did not give his informed consent (see Chapter 3.3.1). Fur- thermore, they must provide treatment following the due standard of care and current medical knowledge. When using CDSS, a clinician might therefore be required to assess whether using the system is part of the standard of care and validate its recommendation (see Chapter 3.3.2).

With opaque AI models, it will be difficult to determine whether he was at fault for using the CDSS in the first place and whether he was at fault for deviating from or following the system's recommendations despite thinking the recommendation is (in)correct.¹¹⁰

EU regulation has not yet agreed on a liability scheme for AI and whether and where to appoint strict liability or an adaption to the burden of proof. In a strict liability scheme, explainability holds less relevance since fault is irrelevant. For fault-based liability, however, the injured party, or when the burden of proof is reversed, the liable party, has to prove the fault (or non- fault) of the liable party and the causal link between this fault and the damage.¹¹¹ In civil liability law, the burden of proof might be shifted from patients to doctors in the future, due to the complexities of AI.¹¹² Proving a fault like following or deviating from the system's recommendation to establish liability can be impossible or at least very difficult if the AI is not transparent.

Explainability could help to open this black-box and to determine fault and liability.¹¹³ 3.3.4 Summary

In summary, explainability ensures clinicians can follow their medical legal and ethical obliga- tions, as it enables them to make sound, validated decisions and judgements about diagnoses, treatments, or resource allocations, together and with the informed consent of the patient. This, in combination with improving the legal certainty for liability, will help clinicians to accept and trust these systems so that CDSS can be used to the patients' benefit.

3.4 Explainability Requirements of CDSS

This section analyses which explainability requirements would have to be set out in a regulation for the purposes and use cases shown above. Depending on the immediate goal of the explanation, different explanation forms or explainability methods¹¹⁴ can become relevant for

109 EC, "COM(2020) 64 final", 14f.; Schneeberger et.al., "Legal Framework Medical AI", 218.

110 Schönberger, "Artificial intelligence in health care", 197; Schneeberger et.al., "Legal Framework Medical AI", 219.

111 EC, "COM(2020) 64 final", 12.

113 EC, "COM(2020) 64 final", 15; Köhl et.al., "Non-Functional Requirement", 363.

114 See examples of explainability models, classified by three explanation forms (example-, model-, and attribution based), and their use cases in medicine here: Markus et.al., "The role of explainability", 4-6.

(19)

15 clinicians. The aim here is to show the complexity of clinicians' explainability demands and that several explainability techniques must be considered. This section does not propose a so- lution that links all technically available explainability techniques¹¹⁵ to clinicians' explainability demands of CDSS. This would go beyond the scope of this paper.

Example-based explanations, like counterfactuals¹¹⁶ or prototypes, are often structured in a way that resembles how clinicians communicate with patients. They are therefore well suited when clinicians want to communicate the CDSS recommendation to the patient.¹¹⁷ In addition, to support shared decision-making, the explanation should facilitate a value-flexible approach by enabling doctors to discuss treatment options with their patients and take the individual patients' values into account. This is important in a medical context because the relevance of specific values can vary from patient to patient. Some might value the treatment with the highest life expectancy, while others prefer the least painful treatment.¹¹⁸

Attribution- or model-based explanations, as post-hoc explanation methods, give information about the relevance of different variables or factors, the interactions between them, or changes in the state of an individual patient,¹¹⁹ that influenced the outcome. This influence of variables on the recommendation is important for clinicians when they want to validate the CDSS recommendation and justify their decisions. The explanation should, if possible, be aligned with the current analytical process of evidence-based medical decision-making to support a better understanding [coherence and personalisation]¹²⁰. It should include the influence of variables for the specific patient as well as for the whole population or should alternatively be generalisable [completeness].¹²¹

When showing the influence of factors on the recommendation, it is important not to create over-reliance on a possibly wrong recommendation. A simple listing of relevant factors can lead to the wrong impression that the CDSS is up-to-date on medical knowledge simply because it lists a multitude of factors. This can be avoided by also including the factors that point against the recommendation, including the weight each factor had on the recommendation, and providing the pathological or biomedical link between the factors and the diagnosis instead of just

115 See e.g. a detailed description of different post-hoc explainability techniques for ML models and deep neural networks here: Arrieta et.al., "Explainable Artificial Intelligence", 92-99.

116 See e.g.: Wachter et.al., "Counterfactual Explanations".

117 Markus et.al., "The role of explainability", 5f.; Tonekaboni et.al., "What Clinicians Want", 5ff.

118 McDougall, "Computer knows best?", 157f.

119 But this has hardly been investigated for clinical AI (Tonekaboni et.al., "What Clinicians Want", 8).

120 The terms provided in square brackets in this chapter are taken from: Sokol, Flach, "Explainability fact sheets", 59-61. Further definitions and general details on these usability requirements can be found there.

121 Markus et.al., "The role of explainability", 5f.; Tonekaboni et.al., "What Clinicians Want", 5ff.

(20)

16 showing a collection of data the clinician is already aware of. Influence factors should also take the medical history and symptoms into account. Clinicians might otherwise not trust a correct result as the system's reasoning seems different from their medical reasoning.¹²²

Information on training data, confidence scores, methods to show how the input affected the output, and the robustness of the systems (showing how applicable and reliable the model is for the specific patient and where it may fall short or be biased)¹²³ should be included. Those help clinicians to understand for which individuals the system performs well [contextfullness (sic!)]

and are important to generate their trust in the CDSS.¹²⁴ The exact meaning of confidence scores must be elaborated and put into context to the confidence scores of other recommendations.

The explanation should include a differential diagnosis, meaning an alternative diagnosis that can be ruled out through reasoning and comparison of the clinical data, symptoms, etc. Further- more, having a description of the typical symptoms and risks of the predicted disease can in- crease clinicians' trust and support them in making a diagnosis.¹²⁵ Another factor for clinicians to trust the CDSS is consistency. When a prediction changes or an alert is issued, they expect a clinically relevant change in the patient and consequently also a change in the explanation.¹²⁶ In general, the explanation must be tailored to the purpose and given in a way that makes it possible to integrate it into the workflow [actionability]. Especially in time-sensitive and cha- otic environments like the Emergency Department or the Intensive Care Unit, explanations must be brief and to the point. They must allow the clinician to see and evaluate the relevant information for this prediction with one glance, instead of distracting him with too much information [parsimony and level of complexity]. The explanation should support not only the in- herent purposes of CDSS, e.g. qualifying cells as (non)metastatic but also the external purposes, e.g. supporting the clinician in deciding on the next steps in diagnosis and treatment,¹²⁷ whether those are immediate interventions, further tests, or the correct allocation of resources. There- fore, if possible, the explanation should also include causal mechanistic reasoning. Instead of just getting the qualification of cells as metastatic or non-metastatic or of a tumour as malignant or benign, the clinician then also gets an explanation for why they were classified this way.¹²⁸ A user-friendly visualization will help the clinician in time-critical situations and when faced with various demands, but more detailed explanations should also be available upon request

122 Bussone et.al., "The Role of Explanations", 164-168.

123 Chakraborty et.al., "Explainability for Healthcare", 13.

124 Cutillo et.al., "Machine intelligence in healthcare", 3f.

125 Bussone et.al., "The Role of Explanations", 167f.

126 Tonekaboni et.al., "What Clinicians Want", 9f.

127 Tonekaboni et.al., "What Clinicians Want", 6 and 9.

128 Ratti, "Explainable AI and medicine".

(21)

17 [interactiveness (sic!)].¹²⁹ Integration into the workflow also means that the explanation should not generate more work for the clinician. It should make the verification of the recommendation simple.¹³⁰

To give an example, an explanation for a CDSS that predicts patient readmittance could consist of the classification result (re-admittance or not), the accuracy of the model for making this prediction, the features that influenced the decision with the top five features highlighted (e.g.

how many times the patient was admitted to the hospital, whether he was admitted as an emergency, changes in his medication), and what impact changing one factor has on the outcome.

The explanation would be presented in different charts for an easy and quick understandable overview but include the option to retrieve further details.¹³¹ Another, more detailed example includes several suggestions for how a user interface (visualisation of the explanations) might look like for an ADHD¹³² diagnosis in a child. It includes data on the 20 most important information elements in clinical decision-making. These encompass the input data, contradicting information that would point against this diagnosis, the level of certainty for that diagnosis, which data would improve that certainty, differential diagnoses, and how the system performs in similar cases.¹³³

Moreover, clinicians often prefer local explanations that focus on why a specific recommendation was given for an individual patient. Yet, global explanations that focus on the whole model, on an entire population, or on sub-groups of the population can also become relevant, depending on whether the clinician needs a more generalisable explanation or not.¹³⁴

Several technical explanation methods can satisfy some of these explainability requirements and are simple to generate, like local explanations or counter-factual faithfulness that provide information on the main factors and their influence on the recommendation. Others pose greater difficulties but are still possible to be produced, such as providing pro and contra arguments to support a decision.¹³⁵ These explanation methods would e.g. allow a differential diagnosis or counter over-trust issues. Causal reasoning and providing the biomedical links between input factors and the recommendation might be more difficult and not possible for every CDSS since ML and especially DL models today rely primarily on correlations. Therefore, most of the existing explanation forms aim to show these correlations and not the causations. However, there

129 Tonekaboni et.al., "What Clinicians Want", 5f.

131 Meacham et.al., "Towards Explainable AI", 945-950.

132 Attention Deficit- / Hyperactivity Disorder.

133 Schoonderwoerd et.al., "Human-centered XAI", 8f. and Annex.

135 Brkan, Bonnet, "Legal and Technical Feasibility", 46-50.

(22)

18 are some approaches to explain AI systems using causality as well.¹³⁶ The same is true for systems that are themselves explainable. Their explanations of the recommendations are mainly based on correlations and, while not impossible, showing causation remains difficult.¹³⁷ 3.5 Explainability required by Regulations

This section analyses whether current European regulations like the CFR, the GDPR or the MDR, international technical standards, or regulatory proposals like the AI Act impose relevant explainability requirements for CDSS according to the requirements just established in Chapter 3.4, to determine whether current regulations obligate developers to include these specific explainability requirements in their CDSS.

3.5.1 Fundamental Rights Law

Art. 3(2)(a) CFR establishes that in medicine, the free and informed consent of the person concerned must be given. The same is stated in Art. 5 of the Convention on Human Rights and Biomedicine adopted by the Council of Europe. It will be difficult for a patient to give informed consent if the CDSSs recommendation cannot be explained (see Chapter 3.3.1). Art. 3(2)(a) CFR therefore indirectly requires an explanation of CDSS. Also, the provisions against non- discrimination (Art. 20-26 CFR) could indirectly include the obligation to make CDSS explainable,¹³⁸ as explainability can help to uncover bias and hence avoid discrimination (see Chapter 3.3.2). However, it must be kept in mind that the Charter does not apply directly to e.g. developers of CDSS. Rather, it is addressed to the EU institutions and Member States when they implement EU law and is additionally used as a tool for interpretation (Art. 51, 52(5) CFR).

Still, it is also relevant for developers. The state has a duty to protect individuals from third parties interfering with these fundamental rights. It can, therefore, be obligated to regulate the provision of health services, including CDSS and their explainability, in a way that ensures patients do not suffer any damages or disadvantages.¹³⁹ While an indirect requirement for explainability can hence be derived from Fundamental Rights Law, it states no explicit obligation for explainability of CDSS and even less specifies the requirements for explainability.

3.5.2 General Data Protection Regulation

Even though it remains contested whether the right to an explanation exists in the GDPR,¹⁴⁰ the GDPR must be considered as one of the main European regulations on explainability of

136 Marcus, "Deep Learning", 12f.; Brkan, Bonnet, "Legal and Technical Feasibility", 28, 49 and references in notes 105-108 and 116; Halpern, "Axiomatizing causal reasoning".

137 Caruna et.al., "Predicting Pneumonia Risk", 1730.

140 See e.g. references in: Wachter et.al., "Counterfactual Explanations", 842 (footnote 1); Schneeberger et.al.,

"Legal Framework Medical AI", 212.

(23)

19 automated decisions. The first question is whether the GDPR, and Art. 22 on automated decisions, in particular, apply to CDSS since the right to an explanation or general information is linked to decisions based solely on automated processing (Art. 22 and 13-15 GDPR).

CDSS process personal data and therefore fall under the scope of the GDPR (Art. 2(1)¹⁴¹).

CDSS work mainly with patient's data concerning health¹⁴² (Art. 4(15)) and this personal data can generally be related to the patient as an identified or identifiable natural person (Art. 4(1)), except if the CDSS uses only pseudonymised data (Art. 4(5)). Processing takes place according to Art. 4(2), as a CDSS uses automated means to structure a patient's health data to make a recommendation or prediction about the diagnosis or treatment¹⁴³.

Art. 22 applies when the data subject is "subject to a decision based solely on automated processing" and if that decision results in legal or similarly significant effects. In most cases, CDSS will have a similarly significant effect, especially when they provide recommendations in the context of treatment and diagnosis.¹⁴⁴ A decision is based solely on automated means if it is made by technological means and excludes any human involvement in the decision-making process. If a human, who has the competence and authority to evaluate and change the recommendation, acts in a way that influences the final decision, the decision is not based solely on automated means.¹⁴⁵ Clinicians will, in most use cases of CDSS today, still actively evaluate the recommendation and then make their decision, meaning Art. 22 would not be applicable.

However, sometimes clinicians with less medical knowledge, such as administrative staff, might also be instructed to strictly follow the CDSS recommendation without questioning it, making Art. 22 applicable.¹⁴⁶

Special categories of personal data, like health data, usually may not be processed (Art. 9(1)).

Art. 22(4) gives an exception for processing such data regarding automated decisions. The use of the CDSS that processes health data would not be prohibited¹⁴⁷ if Art. 9(2)(a) (explicit consent) or (g) (reasons of substantial public interest) apply. Assuming that either the patient con- sented to the use of his data for automated decision-making by a CDSS or that the processing

141 If not stated otherwise, Art. listed in this chapter refer to the GDPR.

145 Art29WP, "WP251rev.01", 8f. and 20f.

146 Schneeberger et.al., "Legal Framework Medical AI", 213-215.

147 Whether Art. 22(1) gives a right to the data subject or is an outright prohibition is also contested (see e.g.

Art29WP, "WP251rev.01", 19; Bygrave, "Minding the Machine", 252f.), but the distinction is not relevant here.

Explainability in Medical AI Legal Regulation of Explainability in Clinical Decision Support Systems