Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps

ContentslistsavailableatScienceDirect

Medical Image Analysis

journalhomepage:www.elsevier.com/locate/media

Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps

Kristoffer Wickstrøm

^1,∗

, Michael Kampffmeyer

¹

, Robert Jenssen

¹

Department of Physics and Technology, UiT The Arctic University of Norway, Tromsø NO-9037, Norway

a rt i c l e i n f o

Article history:

Received 10 May 2019 Revised 14 November 2019 Accepted 14 November 2019 Available online 20 November 2019 Keywords:

Polyp segmentation Decision support systems Fully convolutional networks Monte carlo dropout Guided backpropagation

Monte carlo guided backpropagation

a b s t r a c t

Colorectalpolypsareknown tobepotentialprecursorstocolorectal cancer,whichisone ofthe lead- ingcausesofcancer-relateddeathsonaglobalscale.Earlydetectionandpreventionofcolorectalcancer isprimarilyenabledthroughmanualscreenings,wheretheintestinesofapatientisvisuallyexamined.

Suchaprocedurecanbechallengingandexhaustingforthepersonperformingthescreening.Thishas resultedinnumerousstudiesondesigningautomaticsystemsaimedatsupportingphysiciansduringthe examination.Recently, suchautomaticsystemshave seenasignificantimprovement as aresult ofan increasingamountofpubliclyavailablecolorectalimageryand advancesindeep learningresearchfor objectimagerecognition.Specifically,decisionsupportsystemsbasedonConvolutionalNeuralNetworks (CNNs)havedemonstratedstate-of-the-artperformanceonbothdetectionandsegmentationofcolorec- talpolyps.However,CNN-basedmodelsneedtonotonlybepreciseinordertobehelpfulinamedical context.Inaddition,interpretabilityanduncertaintyinpredictionsmustbewellunderstood.Inthispa- per,wedevelopandevaluaterecentadvancesinuncertaintyestimationandmodelinterpretabilityinthe contextofsemanticsegmentationofpolypsfromcolonoscopyimages.Furthermore,weproposeanovel methodforestimatingtheuncertaintyassociatedwithimportantfeaturesintheinputanddemonstrate howinterpretability anduncertainty can bemodeledinDSSs forsemanticsegmentation ofcolorectal polyps.Resultsindicatethatdeepmodelsareutilizingtheshapeandedgeinformationofpolypstomake theirprediction.Moreover,inaccuratepredictionsshowahigherdegreeofuncertaintycomparedtopre- cisepredictions.

© 2019TheAuthors.PublishedbyElsevierB.V.

ThisisanopenaccessarticleundertheCCBY-NC-NDlicense.

(http://creativecommons.org/licenses/by-nc-nd/4.0/)

1. Introduction

ColorectalCancer(CRC)isone oftheleadingcausesofcancer- related deaths worldwide (Siegel et al., 2017; Chen et al., 2016;

Larsen,2016), withan estimatedfive-yearsurvivalrateforanad- vancedstageCRCdiagnosisof14%.Theestimatedsurvivalratefor early diagnosisis 90%(Larsen,2016).Currently, thegoldstandard forCRCpreventionisthroughregularcolonoscopyscreenings.One of themain tasksduringa screening isto locatesmallabnormal growths called polyps, which are known to be possible precur- sors to CRC. Hence, increasing the detection rateof polyps is an importantcomponentforreducing mortalityrates.However, such screeningsaremanualproceduresperformedbyphysiciansandare thereforeaffectedbyhumanfactorssuchasfatigueandexperience.

Onestudyhasestimatedthepolypmissrateduringascreeningto

∗ Corresponding author.

E-mail address: kristoffer.k.wickstrom@uit.no (K. Wickstrøm).

1UiT Machine Learning Group ( http://machine-learning.uit.no ).

bebetween8–37%,dependingonthe sizeandtypeofthepolyps (VanRijnetal.,2006).Apossiblemethodforincreasingpolypde- tection rateis to design Decision Support Systems (DSSs), which could aidphysicians duringorafterthe procedure.Adependable androbustDSSwouldhavetheadvantageofnotbeinginfluenced byhumanfactorsandcould alsoprovideasecondopinionforin- experiencedpractitioners.

One popular approach fordeveloping DSSs has been through machine learning, with promising resultson a rangeof different tasks like braintumor segmentation (Havaei etal., 2017), retinal vesselsegmentation (Guo etal., 2019), melanoma lesionsegmen- tation (Nida et al., 2019), and colorectal polyp detection (Bernal etal., 2015;2014; Liu,2017; Ribeiro etal., 2016). In the context ofCRCprevention,therehavebeenanumberofstudiesonthede- tectionofpolypswithencouragingresults(Tajbakhshetal.,2016;

Hwangetal., 2007; Alexandreetal., 2007;Wimmer etal., 2016;

Häfneret al., 2015), but polyp segmentation has proven to be a challengingtaskandthe necessaryprecision hasbeendifficult to obtain(Bernaletal.,2015;2014;CondessaandBioucas-Dias,2012).

https://doi.org/10.1016/j.media.2019.101619

1361-8415/© 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ )

However,asaconsequenceofincreasingamountsofpubliclyavail- ablecolonimagery combinedwithadvances indeep learningre- searchforimageanalysis,recentstudiesbasedondeeplearningfor colorectalpolyp segmentation have shownpromising results and a significant increase in precision (Vázquez et al., 2016; Brandao etal.,2017;Urbanetal.,2018).

High precision isa crucialcomponentofanyreliable DSS,but other constituentsare also vital in orderto engineer dependable DSSs. Physiciansare tasked with making decisions that can have fatalconsequencesandtheygotogreatlengthsinordertoensure thatthedecisiontheymakeislikelytohaveafavorableoutcome.

Therefore,atrustworthy DSSshould providea measure ofuncer- taintytoaccompany itspredictionsuch thatphysicians canmake well-informeddecisions.AnotherintegralpartofadependableDSS istocommunicatetotheuserwhatfactorsinfluencesaprediction.

Withoutsuchinformation,theusercannotdetermineifthemodel isdetecting featuresthat are actually associatedwiththedisease inquestionorifitisexploitingartifactsinthedata.Forinstance,a studybyZechetal.(2018)uncoveredthatadeeplearningmodel taskedwithdiagnosingdiseasefromx-rayimageshadlearnedto exploitinformation inmetal tokens includedin thex-ray images forinference insteadofdetectingdisease-specificsfeatures.When themodelisthenpresentedwithanimagewithouttheseartifacts theprecisiondropsconsiderably.

Despitethe obviousbenefitofincreasedperformance,systems basedondeeplearninghavenoinherentwayofrepresentingthe uncertaintyassociatedwitha model’spredictionnordotheypro- vide anyindicationasto what features in theinput influences a particularprediction.Thislackoftheoreticalunderstandingforthe underlyingmechanicsofdeepmodelshaveresultedindeeplearn- ing basedmodels often beingreferred to as ”blackboxes” (Alain andBengio, 2017; Shwartz-ZivandTishby, 2017;Yu andPríncipe, 2018). Multiple recent studies have proposed methods that, to some extent, address the lack of transparency (Gal and Ghahra- mani,2016;KendallandGal,2017;Springenbergetal.,2015;Zeiler andFergus, 2014;Bach et al., 2015; Simonyan et al., 2013), and they have seen some use in analysis of medical images (Dubost etal., 2019;Zech et al., 2018) However, these methods have yet tobeutilizedinDSSs forcolorectalpolypsegmentationbasedon deeplearning.

Ourcontributionsarethefollowing:²

• We incorporate and develop recent advances in the field of deep learning for semantic segmentation of colorectal polyps in ordertocreatedeep modelsthat provide uncertaintymea- sures along with their prediction. Results indicate that erro- neouspredictionsshow asignificantly higherdegreeofuncer- taintycomparedtocorrectpredictions.Furthermore,wemodel input feature importance to createinterpretable deepmodels.

Results showthat ourmodels areconsidering shapeandedge informationinordertosegmentpolyps.

• We propose anovel methodfor estimatinguncertaintyinthe importanceofinputfeatures,whichwerefertoasMonteCarlo Guided Backpropagation, and demonstrate how this method canbeusedinthecontextofcolorectalpolypsegmentation.

Totheauthors’knowledge,noneoftheabovepointshavebeen previously explored in the context of semantic segmentation of colorectalpolyps.

2This work significantly extends our preliminary study ( Wickstrøm et al., 2018 ) by: (1) Including U-Net in our analysis; (2) significantly extending our experimental section by including new experiments on the 2015 MICCAI polyp detection chal- lenge ( Bernal et al., 2017 ) and the Endoscene dataset ( Vázquez et al., 2016 ) (3) proposing a novel method for estimating uncertainty in the importance of input features and evaluating our proposed method on two polyp segmentation datasets;

(4) providing a more thorough literature background discussion and placing our work into a broader context.

2. Modelsandmethods

In this section we introduce Fully Convolutional Networks (FCNs) anddescribethe threearchitecturesutilized inthisstudy.

Next, we explain how we incorporate uncertainty and inter- pretability indeeplearningbased DSSs(Sections2.2and2.3). Fi- nally,wepresentourmethodforestimatingtheuncertaintyasso- ciatedwiththeimportanceofinputfeatures(Section2.4).

2.1. Fullyconvolutionalnetworks

FCNsareCNNsparticularlysuitedtotackleperpixelprediction problems like semantic segmentation, i.e.providing a probability scoreforwhatclasseachpixelbelongsto.Forinstance,inthecase ofsemanticsegmentationofcolorectalpolyps,eachpixelislabeled asapolyporaspartofthecolon(backgroundclass).Segmentation isconsidereda morechallengingtaskthandetectingorlocalizing an objectinan image, butprovides moreinformation.The shape informationprovided by a meaningfulsegmentation map can for example be usedto study anatomicalstructures or inspectother regionsofinterest(Sharmaetal.,2010).

We investigate three architectures for the task of polyp seg- mentation, namely the Fully Convolutional Network 8 (FCN- 8) (Shelhameret al., 2017), U-Net (Ronneberger et al., 2015) and SegNet (Badrinarayanan et al., 2017) for the following reasons.

These networks have been applied in a number of different do- mains andare chosen to form a well-understood foundationfor our studies. This enables uncertainty andinterpretability experi- ments tobe themain focus.Previous useofthe FCN-8forpolyp segmentation has shownpromising results(Vázquez etal., 2016;

Brandaoetal.,2017).SegNethasbeenshowntoachievecompara- bleresultstotheFCN-8insomeapplicationsbutisalessmemory intensiveapproach withfewerparameters tooptimize.U-Nethas previouslydemonstratedencouragingresultsonmedicaltasksand doesalsocontainfewerparametersthantheFCN-8,thusproviding a lightweight alternative. We include thesedifferent networksin thisstudyin orderto compare what features areconsidered im- portant bydifferentmodels andhow uncertaintyestimatesdiffer amongnetworks.Theinterestedreadercanfindadetaileddescrip- tionalongwithfiguresofthethreemodelsinAppendixA. 2.2. Uncertaintyinfullyconvolutionalnetworks

Despitetheir successonanumberofdifferenttasks,CNNsare not without flaws. One of theseflaws, whichbecomes especially apparentformedicalapplications,istheir inabilitytoprovideany notionofuncertaintyintheirprediction.Whenaphysicianiscon- sideringthesymptoms ofapatientandcontemplateswhatmedi- cationtoprescribe theremightbeseveralviableoptions,andthe final decisionmightspellthedifference betweenafatal orfavor- able outcome. Since the stakes are so high, physicians will have toweightthedifferentoptionsandreflectonwhichchoiceismost likelytohaveafavorableoutcome.Ifaphysiciandecidestoconsult aDSSbasedonaCNN,sheorhewouldbepresentedwitharec- ommendationthat has noindicationasto how likelya desirable outcome is,thusmaking it difficultforthephysician totrust the system.Althoughthesoftmaxoutputregularlyfoundattheendof aCNNissometimesinterpretedasmodelconfidence,thisisgener- allyill-advised(Gal andGhahramani,2016) andotherapproaches mustbeconsidered.

In contrast, Bayesian models provide a framework which naturally includes uncertainty by modeling posterior distri- bution for the quantities in question. Given a dataset D≡

xn∈R^D,yn∈R^C

N

n=1, where xn denotes an input vector andyn

denotes its corresponding one-hot encodedlabel vector, the pre- dictivedistributionofaBayesianneuralnetworkforanewpairof

Fig. 1. Illustration of the Monte Carlo Dropout procedure. The same input image is passed through a trained FCN with Dropout applied T times, resulting in T different predictions. The standard deviation of each pixel is then estimated based on these T predictions.

samples{x∗,y∗}canbemodeledas:

p

(

^y∗

|

^x∗,D

)

=

p

(

^y∗

|

^x∗,W

)

^p

(

^W

|

^x∗,D

)

^{dW (1)}

In Eq. (1), W refers to the weights of the model, p(y∗|x∗, W) is thesoftmaxfunctionappliedtotheoutputofthemodel,denoted

byf_W(x∗),andp(^W

|

^x∗,D)^{istheposteriorovertheweightswhich} capturethesetof plausiblemodelparameters forthegivendata.

Obtaining p(y∗|x∗, W) only requires a forward pass of the net- work,buttheinabilitytoevaluatetheposterioroftheweightsan- alyticallymakesBayesianneuralnetworkscomputationallyinfeasi- ble.Tosidesteptheproblematicposterioroftheweights,(Galand Ghahramani,2016) proposed toincorporateDropoutasa method forsamplingsetsofweightsfromthetrainednetworktoapproxi- matetheposterioroftheweights.Thepredictivedistributionfrom Eq.(1)canthenbeapproximatedusingMonteCarlointegrationas follows:

p

(

^y∗

|

^x∗,D

)

^{≈ 1}_T T

t=1

Softmax

(

^fW∗t

(

^x∗

))

⁽²⁾

where T is the number of sampled sets of weights and W^∗tis a setofsampledweights.Inpractice,thepredictivedistributionfrom Eq.(2)canbe estimatedby runningT forwardpassesofa model withDropoutappliedtoproduceTpredictionsandthencomputing thestandarddeviationoverthesoftmaxoutputsoftheTsamples.

Wewillrefer totheseuncertaintyestimatesasuncertaintymaps.

ThismethodofutilizingDropoutforsamplingfromthe posterior ofthepredictivedistributionisreferredtoasMonteCarloDropout, andthemethodisillustratedinFig.1.

2.3.Interpretabilityinfullyconvolutionalnetworks

Anotherdesirableproperty whichCNNs lackisinterpretability, i.e.beingableto determinewhat features induce thenetwork to produceaparticularprediction.Forinstance,aphysicianmightbe interestedindiscerningwhatinformationthepredictionofagiven DSSisbasedon,andifitconcurswithmedicalknowledge.ACNN- basedDSShasnoinherentwayofprovidingsuchan explanation.

However, several recent works have proposed different methods toincreasenetworkinterpretability(ZeilerandFergus,2014;Bach et al., 2015). In this paper, we evaluate anddevelop the Guided Backpropagation(Springenbergetal.,2015)techniqueforFCNs on thetaskofsemanticsegmentationofcolorectalpolypsinorderto

Fig. 2. Figure displays the prediction, uncertainty map, and interpretability map for the FCN-8, SegNet and U-Net, for the input image shown in the leftmost column. Best viewed in color.

Fig. 3. Precision and recall vs uncertainty plot for background and polyp class on the Endoscene test set.

assesswhichpixelsintheinputimagethenetworkdeemsimpor- tantforidentifyingpolyps. WechooseGuidedBackpropagationas itisknown toproduceclearervisualizationsofsalient inputpix- elscomparedtoothermethods(ZeilerandFergus,2014;Simonyan etal., 2013). We refer to these visualizationsof salient pixels as interpretabilitymaps.

ThecentralideaofGuidedBackpropagationistheinterpretation ofthe gradients of the network withrespect to an input image.

Simonyanetal.(2013) exploitedthat,foragivenimage,themag- nitudeof the gradients indicate which pixels in the input image needtobechangedtheleasttoaffectthepredictionthemost.By utilizingbackpropagation (Rumelhart etal., 1988; Werbos, 1974), theyobtainedthegradientscorrespondingtoeachpixelinthein- putsuchthattheycouldvisualizewhatfeaturesthenetworkcon- sidersessential.Springenbergetal.(2015)arguedthatpositivegra- dientswitha largemagnitude indicatepixels ofhighimportance while negative gradients with a large magnitude indicate pixels whichthenetworkswantto suppress.Ifthesenegativegradients areincluded in thevisualization of importantpixels it might re- sultinnoisyvisualizationofdescriptivefeatures.Inordertoavoid noisyvisualizationsthe Guided Backpropagation procedure alters thebackward pass ofa neural networksuch that negativegradi- entsaresettozeroineachlayer,thusallowingonlypositivegradi- entstoflowbackwardthroughthenetworkandhighlightingpixels thatthesystemfindsimportant.

2.4. Montecarloguidedbackpropagation:Uncertaintyininput featureimportance

Todeterminetheuncertaintyassociatedwithaninputfeature’s importance for the prediction, we propose a novel approach in- spired byMonteCarlo DropoutcombinedwithGuided Backprop- agation. In Section 2.2 we discussed CNNs inability to produce any notion of uncertainty and described Monte Carlo Dropout, whichprovides amethod toobtain approximatemeasuresof un- certainty forCNNs by utilizingDropoutduringinference. Accom- panying a model’s prediction with an uncertainty estimate adds theoption toassess ifaparticular predictionishighlycertain or acasethatcouldrequirefurtheranalysisfromahumanexpert.In Section2.3wedescribedGuidedBackpropagation,atechniquede- velopedto visualize the relative importance ofinput features for CNNsby consideringthepositive gradientsfromabackward pass through thenetwork. But,determining theimportance of thein- putfeatures based ongradients froma singlebackward pass en- counters the same issue we discussed regarding decisions based onpredictions fromasingleforwardpass.Howconfident are we thatthesefeaturesareimportantforthedecisionofthenetwork?

Givenanewsamplex∗,wewanttofindthegradientsthatcor- respondtotheinputfeatures,denotedby

δ

^{0.Takinga similarap-}

proach asin Section 2.2, the approximate predictive distribution forthegradientsoftheinputfeaturesisgivenby

q

( δ

⁰

|

^x∗

)

^{= p}

( δ

⁰

|

^x∗,

θ )

^q

( θ )

^d

θ

^{. (3)}

Calculating p(

δ

^0|x∗,

θ

^{) is done throughthe} backpropagationalgo- rithm, i.e.computingthe gradientswith respectto theoutput of thenetwork andthenusingthe chainruleto workbackward to- wardtheinputgradients.Also,wemodifythebackwardpasssuch that negative gradients are canceled, following the Guided Back- propagation procedure.For clear notation, we denote thisproce- dure as

∇

_θ^fgb(x∗;

θ

^{), where}

∇

_θ ^{indicate finding the gradients of}

each layer with respect to the parameters of the network and f^gb(x∗;

θ

^{) is thepredictionof themodel withthemodified back-}

ward pass.The predictive distribution inEq. (1)can then be ap- proximatedusingMonteCarlointegrationasfollows:

q

( δ

⁰

|

^x∗

)

=1 T

T

t=1

∇

θf^gb

(

^x∗;W^∗_t

)

. (4)

In practice,this amounts to performing T forward and backward passeswithDropoutappliedandcomputingthestandarddeviation overthegradientsofeachinputpixeloverallTsamples.Werefer tothismethodofestimatinggradient uncertaintyasMonteCarlo GuidedBackpropagation.

3. Experiments 3.1. Experimentalsetup

We evaluate ourmethods on a recent benchmark dataset for polypsegmentation,namelytheEndoScenedataset(Vázquezetal., 2016), which consists of 912 RGB images obtained from colono- scopies of36patients. Eachinputimage hasa corresponding an- notated (labeled)image provided by physicians, wherepixels be- longingtoapolyparemarkedinwhiteandpixelsbelongingtothe colonare markedinblack. Weconsiderthe binarytaskofclassi- fyingeach pixelaspolyporpartofthe colon(background class).

Following the approachof Vázquez etal. (2016) we separate the dataset into a training, validation, and test set. The training set consistsof20patientsand547images,thevalidationsetconsists of8patientsand183images,andthetestsetconsistsof8patients and182images.AllRGBinputimagesarenormalizedtotherange [0,1].AllmodelsweretrainedusingADAM(KingmaandBa,2014)

Fig. 4. Figure displays the prediction, uncertainty map, and interpretability map for the FCN-8, SegNet and U-Net, for the input image shown in the leftmost column. Best viewed in color.

Table 1

Results on the EndoScene test dataset.

Model # Parameters(M) IoU background IoU polyp Mean IoU Global Accuracy

SDEM ( Bernal et al., 2014 ) - 0.799 0.221 0.412 0.756

U-Net 27.5 0.945 0.516 0.723 0.945

SegNet 29.5 0.933 0.522 0.727 0.935

FCN-8 ( Vázquez et al., 2016 ) 134.5 0.946 0.509 0.727 0.949

FCN-8 134.5 0.946 0.587 0.767 0.949

withabatch sizeof10 anda cross-entropyloss.Weusetheval- idation set to apply early stopping by monitoring the polyp IoU scorewithapatienceof30.Forperformanceevaluation,wecalcu- late the Intersectionover Union(IoU) metricandglobalaccuracy (per-pixelaccuracy) onthetestset.Foragivenclassc,prediction

ˆ

y_iandgroundtruthy_i,theIoUisdefinedas IoU

(

^c

)

=

i

(

^yˆi==c∧yi==c

)

i

(

^yˆi==c∨yi==c

)

⁽⁵⁾

where∧isthelogicalandoperationand∨isthelogicaloropera- tion.

Additionally,weevaluatedourproposedmethodforestimating uncertaintyininputfeatureimportanceonthe2015MICCAIpolyp detectionchallenge(Bernaletal.,2017).Asthetestimagesofthis dataset are ofhigh quality andour proposed approach is mostly avisualtechnique,assessingourmethodonthisdatawillprovide furthervalidationofourmethod.

3.2. Quantitativeandqualitativeresults

QuantitativeresultsInTable1wereportourresultsfortheFCN- 8,SegNet andU-Netalong withtheresultsofprevious workson polyp segmentation from both traditional machine learning and deep learning based approaches. The traditional machine learn- ing methodcomputes ahistogrambased onthe pixelvalues and uses peaks and valleys information from the histogram to per- formsegmentation.Itisreferred toastheSegmentationfromEn- ergy Maps (SDEM) algorithm (Bernal et al., 2014). For the deep learning approach, segmentation is performed using the FCN-8,

but without Batch Normalization or transfer learning. This ap- proachis referred to asFCN-8 in Table 1. The results show that all deep learning approaches significantly outperform the more traditionalmachine learningapproach, andthe difference inper- formance betweenour implementation ofthe FCN-8 and that of Vázquezetal.(2016)demonstratesthatincludingrecentadvances indeeplearningmethodologycanimproveperformance.

Qualitative results Fig. 2(b) and 4(b) displays some qualita- tive results on the test data for the FCN-8, SegNet and U-Net.

Fig.2showsatypicalexamplewherealarge,ellipticalpolypislo- catedwithhighprecisionbyallthreemodels.InFig.4wepresent a more challenging example where all models fail to locate the smallpolyppresentintheimage.Interestedreaderscanfindaddi- tionalresultsinAppendixsBandC.

3.3.Modelinguncertaintyinprediction

Figs.2(c)and4(c)presentexamples ofuncertainty estimation fortheFCN-8, SegNet andU-Net,respectively,using MonteCarlo Dropout.Theseuncertaintymapsareobtainedbysampling10pre- dictionsfromeachmodelwithadropoutrateof0.5andestimat- ingthestandarddeviationforeachpixel.Pixelsdisplayedinbright greenareassociated withhighuncertaintywhile pixelsdisplayed indarkblueareassociatedwithlowuncertainty.

TheexampleshowninFig.2 showsthat all modelshavehigh confidenceformostpixelsintheir prediction,withtheexception ofpixelsaroundtheborderofthepolypitself.Thisisreasonable, asit isdifficult to assessexactly where thepolyp startsand the colonends.IntheexampleshowninFig.4,whereallmodelsmake

Fig. 5. Figure displays input image (a), ground truth (b), prediction with uncertainty overlaid (c), input feature importance (d), and uncertainty in input feature importance (e). For the uncertainty in input feature importance results, pixels colored green indicate that the features are important for the prediction of polyps and that the model is certain of its importance. Pixels colored red indicate features that might be important for the prediction of polyps but the model is uncertain of its importance. Best viewed in color. Input image originated from the MICCAI dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 6. Figure displays input image (a), ground truth (b), prediction with uncertainty overlaid (c), input feature importance (d), and uncertainty in input feature importance (e). For the uncertainty in input feature importance results, pixels colored green indicate that the features are important for the prediction of polyps and that the model is certain of its importance. Pixels colored red indicate features that might be important for the prediction of polyps but the model is uncertain of its importance. Best viewed in color. Input image originated from the Endoscene dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 7. Figure displays input image (a), ground truth (b), prediction with uncertainty overlaid (c), input feature importance (d), and uncertainty in input feature importance (e). For the uncertainty in input feature importance results, pixels colored green indicate that the features are important for the prediction of polyps and that the model is certain of its importance. Pixels colored red indicate features that might be important for the prediction of polyps but the model is uncertain of its importance. Best viewed in color. Input image originated from the MICCAI dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

inaccuratepredictions,the uncertaintyestimateslooknotablydif- ferent,withlargeregions ofuncertaintyforall threemodels.The examplesshowninFigs.2and4demonstratehowseeminglysimi- larpredictionscanhavedifferentuncertaintyestimatesforthedif- ferent types of networks investigated in this work, and that er- roneous predictions show distinctively different uncertainty esti- matesthancorrectpredictions.

Fig. 3 displays how precision and recall is related to uncer- tainty inpredictions. Itshowsthe overallprecision andrecall for eachclassontheEndoscenetestdatasetwhenpixelwithamean- class uncertainty above acertain threshold are excluded. The es- timateduncertaintyforeachclasshavebeennormalizedintoval- ues between0 and1. Results in Fig.3 (a) display how precision decreases asmorepixelpredictionswithhighuncertaintyarein- cluded.Thisconnectionbetweenprecisionanduncertaintyagrees withthequalitativeexamplesinFigs.2and4discussedabove.Re- sultsinFig.3(b)showhowrecall slightlyincreasesforthepolyp classatalow uncertaintythreshold,butthenremains unchanged for both classes.The interestedreader can finda similar experi- mentontheMICCAIdatasetinAppendixC.

3.4. Modelinginputfeatureimportance

Figs.2(d)and4(d)showexampleswhereGuidedBackpropa- gationhasbeenusedtoanalyzetheFCN-8,SegNet andU-Net,re- spectively.Pixelsdisplayedinbrightgreenareassociatedwithpix- elsthat areimportanttothepredictionofthemodelwhilepixels displayedinblueareassociatedwithpixelsthatarelessimportant tothefinalprediction.

Fig.2indicatesthatallmodelsareconsideringtheedgesofthe polyptomaketheirprediction,whereparticularlytheleftmostand bottomedgeofthepolypishighlightedasimportantbyallmod- els.Fig.4,whereallmodelsfailtolocatethepolyp,displaysmore disagreementbetweenthemodelsastowhatpixelsareimportant.

3.5. Modelinguncertaintyininputfeatureimportance

In order to focus on the new methodology we only use one modeltoevaluateourproposedmethod.Theoverallbestperform- ing segmentation model, FCN-8,waschosen to evaluate the pro- posedmethodologyforestimatinguncertaintyininputfeatureim- portanceanddemonstrateitsmerit.Figs.5–7presentsexamplesof uncertaintyestimationforinput featureimportancefortheFCN-8 usingMonte Carlo GuidedBackpropagation.These resultsare ob- tainedbysampling10gradientestimatesfromeachmodelwitha dropout rate of0.5. The figures display: (a) the input image;(b) thegroundtruth; (c)predictionwithuncertaintyoverlaid;(d)in- put feature importance;and (e) uncertainty ininput feature im- portance.Forthe uncertaintyininput feature importanceresults, pixels colored green indicate that the features are important for thepredictionofpolypsandthatthemodeliscertainofitsimpor- tance.Pixelscoloredredindicatefeaturesthatmightbeimportant forthepredictionofpolyps butthemodelisuncertainofits im- portance.ExamplesshowninFigs.5and7arefromthetestsetof theMICCAIdatasetwhiletheexampleshowninFig.6isfromthe testsetoftheEndoscenedataset.Interestedreaderscanfindaddi- tionalexamplesofuncertaintyestimationforinputfeatureimpor- tanceinAppendixB.

Fig.5displaysanexamplewheretheFCN-8makesasuccessful segmentation. TheinterpretabilitymapinFig.5(d)indicates that there aretwo regions ofimportanceinthe inputimage, one cor- respondingtothepolypandoneregiontowardstheleftmostpart oftheimage.However,theuncertaintyintheinputfeatureimpor- tance mapin Fig.5(e) showsthat themodelis uncertainofthe leftmostfeature’simportance,whilethefeaturescorrespondingto thepolypitselfhaveahighdegreeofcertainty.

Fig.6 showsanother examplewhere the FCN-8makes a suc- cessful segmentation, but alsohighlight important input features towards the leftmostpart of theimage, in addition to the polyp itself.Fig.6(e) displaysthat theFCN-8ishighlyconfident inthe importance ofthe features corresponding to thepolyp itself, but indicate a highdegree of uncertainty forthe highlighted regions towardstheleftmostpartoftheimage.

Fig.7exhibitsanexamplefromtheMICCAIdatasetwherethe FCN-8fails tolocate the polyppresentin the image, butinstead segments a large portion ofthe colon as polyp.While the inter- pretabilitymapsinFig.7(d)show largeregions ofimportantpix- els, it is evident from Fig. 7 (e) that none of the regions have a high degree of importance. As the prediction with uncertainty overlayedinFig.7(e)alsoindicatesregionsofuncertainty,practi- tionerswouldbewarytotrustthemodel’spredictioninthiscase.

4. Conclusion

Inthis workwe havedemonstrated howDSSs based on deep learning can be interpretable and provide uncertainty estimates with their predictions. Moreover, we presented a novel method forestimatinguncertaintyininputfeatureimportanceanddemon- strated how this techniquecan be used to model uncertainty in input pixel importance. Ourresults demonstratethat themodels consideredintheseexperimentsexploit edgeandshapeinforma- tionofpolyps inorderto maketheir predictions andthat uncer- taintydifferssignificantlybetweenfalseandcorrectpredictions.

DeclarationofCompetingInterest

All authors declare that they haveno conflicts of interest re- gardingthepublicationofthispaper.

Acknowledgments

Wegratefullyacknowledge thesupport ofNVIDIACorporation withthedonationoftheGPUusedforthisresearch.

AppendixA. Networkdetails

In order to perform per pixel predictions, FCNs employ an encoder-decoderarchitectureandarecapableofend-to-endlearn- ing. Theencoder network extracts useful features froman image andmapsit toa low-resolutionrepresentation.The decoder net- work is tasked with mapping the low-resolution representation back intothesame resolutionasthe input image.Upsampling in FCNs is performed using a fixed upsampling approach, like bi- linearornearestneighborinterpolation,orbylearningtheupsam- plingprocedureaspartofthemodeloptimizationvia transposed convolutions.Learnedupsamplingfiltersaddadditionalparameters tothenetworkarchitecture,buttendtoprovidebetter overallre- sults(Shelhameretal.,2017).Upsamplingcanfurtherbeimproved byincludingskipconnections,whichcombinecoarselevelseman- ticinformationwithhigherresolutionsegmentationfromprevious networklayers.Duetothelackoffullyconnectedlayers,inference canbeperformedonimagesofarbitrarysize.

A1.FCN-8

TheFCN-8wasintroduced byShelhameretal.(2017)andcon- sistsofanencodernetworkandadecodernetwork,wheretheen- codernetworkisbasedontheVGG-16architecture(Simonyanand Zisserman,2015) andconsistsof five encoders. The decoder net- workconsistsofthreedecoders.Dropout(Srivastavaetal.,2014),a regularizationtechniquethatrandomlysetunitsinalayertozero, isincluded betweenalllayers ofthe firstdecoder. Upsampling is

Fig. A.8. An illustration of the FCN-8. Color codes description: Blue - Convolu- tion (3x3), Batch Normalization and ReLU; Yellow - Upsampling; Pink - Summing;

Red - Pooling (2x2); Green - Soft-max. Dropout was included as proposed by Simonyan and Zisserman (2015) (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.).

Fig. A.9. An illustration of the U-Net. Color codes description: Blue - Convolution (3x3), Batch Normalization and ReLU; Green - Soft-max; Yellow arrow - Upsam- pling; Black arrow - Concatenate; Red arrow - Pooling (2x2) (For interpretation of the references to colour in this figure legend, the reader is referred to the web ver- sion of this article.).

performedusing transposed convolutions atthe end ofeach en- coderandskipconnectionsareincludedbetweenthethreecentral encodersand thedecoders. Note that we haveadded BatchNor- malization (Ioffe and Szegedy, 2015) in our implementation and that the encoder weights are initialized with pretrainedweights from a VGG16 model (Simonyan and Zisserman, 2015) that was previouslytrainedontheImageNetdataset(Dengetal.,2009).

A2. U-Net

One of the first networks to build upon FCNs was the U- Net (Ronneberger etal., 2015), whichiscomprised ofan encoder network consisting offive encoders anda decoder network con- sisting offour decoders. U-Net introduced an alternative method torecovertheresolutionofthedatawherethefeaturemapspro- ducedinthefifth encoder isupsampledby afactor oftwousing transposed convolution and concatenated withthe feature maps producedbythefourthencoder.Thesecombinedfeaturemapsare passedintothefirstdecoder,whichinturnisupsampledandcon- catenatedwiththefeaturemapsofthethirdencoder.Thisprocess isrepeateduntiltheresolutionoftheinputfeaturemap isrecov- ered. The final decoder is followedby a 1 × 1convolutions that mapsthefeaturevector intothedesirednumberofclassesanda softmaxfunction. Dropoutis appliedafter each layer ofthefinal encoder.WeincludedBatchNormalizationaftereachlayer,except forlayersprecedingatransposedconvolutionandthefinallayer.

A3. SegNet

BoththeFCN-8andtheU-Netrelyontransposedconvolutions torecoverfeaturemapswiththesameresolutionastheinputfea- tures. SegNet (Badrinarayanan etal., 2017), instead, presents an- other option and is made up of a symmetrically structured en- coderdecodernetwork,wheretheencodernetworkconsistsoffive encoders based on the VGG-16 (Simonyan and Zisserman, 2015) andthedecoderconsistsoffivedecoders.Thedecodernetworkis identicaltotheencodernetworkbutwiththemax-poolingopera- tionreplacedbya max-unpoolingoperation.Whenafeature map is downsampled the max-pooling indices are stored andused at alater stage toperformnon-linear upsampling,aprocedurewith severaladvantages.Firstly,itproducessparsefeaturemapsthatare computationallyattractiveandimplicitfeatureselectors.Secondly, itremovestheneedtolearnadditionalfilterforupsampling,thus reducing the number of parameters in the model. Dropout was included after the three central encoders and decoders inspired byKendalletal.(2015).

Fig. A.10. An illustration of SegNet, originally obtained from Badrinarayanan et al. (2017) . Color codes description: Blue - Convolution (3x3), Batch Normalization and ReLU;

Green - Soft-max; Yellow arrow - Upsampling; Black arrow - Concatenate; Red arrow - Pooling (2x2) (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.).

AppendixB. additionalqualitativeresults

Figs. B.11–B.13 display additional results on test images from the Endoscene datasetfor the FCN-8,SegNet andU-Net, respectively.

Eachrowrepresents,fromtoptobottom,inputimage,groundtruth,prediction,uncertaintymap,andinterpretabilitymap.Results were obtainedusingthesameprocedureasdescribedinthemainpaper.

Figs. B.14–B.16 display additional results of estimatinguncertainty in input feature importance forthe FCN-8.These resultsare also obtainedfollowingthesameproceduredescribedinthemainpaper.

Fig. B.11. Figure displays FCN-8’s predictions, the uncertainty map associated with the predictions, and the input features the network deems important. Each row represents, from top to bottom, input image, ground truth, prediction, uncertainty map, and interpretability map. White pixels are classified as polyps and black pixels are classified as background class. For the uncertainty maps, dark blue pixels are associated with low uncertainty and bright green pixels are associated with high uncertainty. For the interpretability maps, bright green pixels are considered important to the prediction of the network. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. B.12. Figure displays SegNet’s predictions, the uncertainty map associated with the predictions, and the input features the network deems important. Each row rep- resents, from top to bottom, input image, ground truth, prediction, uncertainty map, and interpretability map. White pixels are classified as polyps and black pixels are classified as background class. For the uncertainty maps, dark blue pixels are associated with low uncertainty and bright green pixels are associated with high uncertainty.

For the interpretability maps, bright green pixels are considered important to the prediction of the network. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. B.13. Figure displays U-Net’s predictions, the uncertainty map associated with the predictions, and the input features the network deems important. Each row represents, from top to bottom, input image, ground truth, prediction, uncertainty map, and interpretability map. White pixels are classified as polyps and black pixels are classified as background class. For the uncertainty maps, dark blue pixels are associated with low uncertainty and bright green pixels are associated with high uncertainty. For the interpretability maps, bright green pixels are considered important to the prediction of the network. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. B.14. Figure displays input image (a), ground truth (b), prediction with uncertainty overlaid (c), input feature importance (d), and uncertainty in input feature importance (e). For the uncertainty in input feature importance results, pixels colored green indicate that the features are important for the prediction of polyps and that the model is certain of its importance. Pixels colored red indicate features that might be important for the prediction of polyps but the model is uncertain of its importance. Best viewed in color. Input image originated from the MICCAI dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. B.15. Figure displays input image (a), ground truth (b), prediction with uncertainty overlaid (c), input feature importance (d), and uncertainty in input feature importance (e). For the uncertainty in input feature importance results, pixels colored green indicate that the features are important for the prediction of polyps and that the model is certain of its importance. Pixels colored red indicate features that might be important for the prediction of polyps but the model is uncertain of its importance. Best viewed in color. Input image originated from the Endoscene dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. B.16. Figure displays input image (a), ground truth (b), prediction with uncertainty overlaid (c), input feature importance (d), and uncertainty in input feature importance (e). For the uncertainty in input feature importance results, pixels colored green indicate that the features are important for the prediction of polyps and that the model is certain of its importance. Pixels colored red indicate features that might be important for the prediction of polyps but the model is uncertain of its importance. Best viewed in color. Input image originated from the Endoscene dataset. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

AppendixC. AdditionalQualitativeResultsonMICCAIdataset

Fig. C.17and C.18 display additionalresults on test images fromthe MICCAI datasetfor theFCN-8, SegNet and U-Net,respectively.

Resultswere obtainedusingthesameprocedureasdescribedinthemainpaper.Fig.C.19displayshowprecisionandrecallisrelatedto uncertaintyinpredictionsontheMICCAItestdata,similartotheexperimentdescribedinSection3.3.

Fig. C.17. Figure displays the prediction, uncertainty map, and interpretability map for the FCN-8, SegNet and U-Net, for the input image from the MICCAI dataset shown in the leftmost column. Best viewed in color.

Fig. C.18. Figure displays the prediction, uncertainty map, and interpretability map for the FCN-8, SegNet and U-Net, for the input image from the MICCAI dataset shown in the leftmost column. Best viewed in color.

Fig. C.19. Precision and recall vs uncertainty plot for background and polyp class on the MICCAI test set.

References

Alain, G., Bengio, Y., 2017. Understanding intermediate layers using linear classifier probes. ArXiv: 1610.01644 .

Alexandre, L.A. , Casteleiro, J. , Nobreinst, N. , 2007. Polyp detection in endoscopic video using svms. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladeni ˇc, D., Skowron, A. (Eds.), Knowledge Discovery in Databases: PKDD 2007.

Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 358–365 .

Bach, S. , Binder, A. , Montavon, G. , Klauschen, F. , Müller, K.-R. , Samek, W. , 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise rele- vance propagation. PloS One 10 (7), e0130140 .

Badrinarayanan, V. , Kendall, A. , Cipolla, R. , 2017. Segnet: a deep convolutional en- coder-decoder architecture for image segmentation. IEEE TPAMI 2481–2495 . Bernal, J. , Núñez, J.M. , Sánchez, F.J. , Vilariño, F. , 2014. Polyp segmentation method in

colonoscopy videos by means of Msa-Dova energy maps calculation. In: Work- shop on Clinical Image-Based Procedures. Springer, pp. 41–49 .

Bernal, J. , Sánchez, F.J. , Fernández-Esparrach, G. , Gil, D. , Rodríguez, C. , Vilariño, F. , 2015. Wm-Dova maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Comput. Med. Imaging Graph. 43, 99–111 . Bernal, J., Tajkbaksh, N., Sánchez, F.J., Matuszewski, B.J., Chen, H., Yu, L., Anger-

mann, Q., Romain, O., Rustad, B., Balasingham, I., Pogorelov, K., Choi, S., De- bard, Q., Maier-Hein, L., Speidel, S., Stoyanov, D., Brandao, P., Córdova, H., Sánchez-Montes, C., Gurudu, S.R., Fernández-Esparrach, G., Dray, X., Liang, J., Histace, A., 2017. Comparative validation of polyp detection methods in video colonoscopy: results from the miccai 2015 endoscopic vision challenge. IEEE Trans. Med. Imaging 36 (6), 1231–1249. doi: 10.1109/TMI.2017.2664042 . Chen, W., Zheng, R., Baade, P.D., Zhang, S., Zeng, H., Bray, F., Jemal, A., Yu, X.Q., He, J.,

2016. Cancer statistics in china, 2015. CA: A Cancer J. Clinic. 66 (2), 115–132.

doi: 10.3322/caac.21338 .

Condessa, F. , Bioucas-Dias, J. , 2012. Segmentation and detection of colorectal polyps using local polynomial approximation. In: Campilho, A., Kamel, M. (Eds.), Image Analysis and Recognition. Springer Berlin Heidelberg, pp. 188–197 .

Deng, J. , Dong, W. , Socher, R. , Li, L.-J. , Li, K. , Fei-Fei, L. , 2009. ImageNet: A Large-Scale Hierarchical Image Database. In: Proceedings of the CVPR09, pp. 1097–1105 . Dubost, F., Adams, H., Bortsova, G., Ikram, M.A., Niessen, W., Vernooij, M., de Brui-

jne, M., 2019. 3D regression neural network for the quantification of enlarged perivascular spaces in brain mri. Med. Image Anal. 51, 89–100. doi: 10.1016/j.

media.2018.10.008 .

Gal, Y. , Ghahramani, Z. , 2016. Dropout as a bayesian approximation: Represent- ing model uncertainty in deep learning. In: Proceedings of the ICML. JMLR.org, pp. 1050–1059 .

Guo, S., Wang, K., Kang, H., Zhang, Y., Wang, K., Li, T., 2019. Bts-dsn: deeply super- vised neural network with short connections for retinal vessel segmentation.

Int. J. Med. Inf. doi: 10.1016/j.ijmedinf.2019.03.015 .

Havaei, M., Davy, A., Warde-Farley, D., Biard, A ., Courville, A ., Bengio, Y., Pal, C., Jodoin, P.-M., Larochelle, H., 2017. Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31. doi: 10.1016/j.media.2016.05.004 . Hwang, S., Oh, J., Tavanapong, W., Wong, J., de Groen, P.C., 2007. Polyp detection

in colonoscopy video using elliptical shape feature. In: Proceedings of the IEEE International Conference on Image Processing, 2. II–465–II–468. doi: 10.1109/

ICIP.2007.4379193 .

Häfner, M., Tamaki, T., Tanaka, S., Uhl, A., Wimmer, G., Yoshida, S., 2015. Local fractal dimension based approaches for colonic polyp classification. Med. Image Anal.

26 (1), 92–107. doi: 10.1016/j.media.2015.08.007 .

Ioffe, S. , Szegedy, C. , 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the ICML, pp. 448–456 . Kendall, A., Badrinarayanan, V., Cipolla, R., 2015. Bayesian segnet: model uncertainty

in deep convolutional encoder-decoder architectures for scene understanding.

arXiv: 1511.02680 .

Kendall, A. , Gal, Y. , 2017. What uncertainties do we need in bayesian deep learn- ing for computer vision? In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (Eds.) Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 5574–5584 .

Kingma, D.P., Ba, J., 2014. Adam: a method for stochastic optimization. arXiv: 1412.

6980 .

Larsen, I., 2016. Cancer in norway 2015 - cancer incidence, mortality, survival and prevalence in norway. oslo: Cancer registry of norway; 2016.

Liu, Q., 2017. Deep learning applied to automatic polyp detection in colonoscopy images : master thesis in system engineering with embedded systems.

Nida, N., Irtaza, A., Javed, A., Yousaf, M.H., Mahmood, M.T., 2019. Melanoma lesion detection and segmentation using deep region based convolutional neural net- work and fuzzy c-means clustering. Int. J. Med. Inf. 124, 37–48. doi: 10.1016/j.

ijmedinf.2019.01.005 .

Brandao, P., Mazomenos, P., Ciuti, G., Caliò, R., Bianchi, F., Menciassi, A., Dario, P., Koulaouzidis, A., Arezzo, A., Stoyanov, D., 2017. Fully convolutional neural net- works for polyp segmentation in colonoscopy. Proc. SPIE 10134. 10134–10134–

7. doi: 10.1117/12.2254361 .

Ribeiro, E., Uhl, A., Häfner, M., 2016. Colonic polyp classification with convolutional neural networks. In: Proceedings of the IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), pp. 253–258. doi: 10.1109/CBMS.2016.

39 .

Ronneberger, O. , Fischer, P. , Brox, T. , 2015. U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (Eds.), MICCAI. Springer International Publishing, Cham, pp. 234–241 . Rumelhart, D.E. , Hinton, G.E. , Williams, R.J. , 1988. Neurocomputing: foundations of

research. Nature 696–699 .

Sharma, N. , Ray, A. , Shukla, K. , Sharma, S. , Pradhan, S. , Srivastva, A. , Aggarwal, L. , 2010. Automated medical image segmentation techniques. J. Med. Phys. 35 (1), 3 .

Shelhamer, E. , Long, J. , Darrell, T. , 2017. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39 (4), 640–651 . Shwartz-Ziv, R., Tishby, N., 2017. Opening the black box of deep neural networks via

information. arXiv: 1703.00810 .

Siegel, R.L., Miller, K.D., Jemal, A., 2017. Cancer statistics, 2017. CA: A Cancer J. Clinic.

67 (1), 7–30. doi: 10.3322/caac.21387 .

Simonyan, K., Vedaldi, A., Zisserman, A., 2013. Deep inside convolutional networks:

visualising image classification models and saliency maps. arXiv: 1312.6034 . Simonyan, K. , Zisserman, A. , 2015. Very deep convolutional networks for large-scale

image recognition. ICLR .

Springenberg, J. , Dosovitskiy, A. , Brox, T. , Riedmiller, M. , 2015. Striving for simplicity:

The all convolutional net. In: Proceedings of the ICLR (Workshop track), p. N/A . Srivastava, N. , Hinton, G. , Krizhevsky, A. , Sutskever, I. , Salakhutdinov, R. , 2014.

Dropout: a simple way to prevent neural networks from overfitting. J. Mach.

Learn. Res. 15, 1929–1958 .

Tajbakhsh, N., Gurudu, S.R., Liang, J., 2016. Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans. Med.

Imaging 35 (2), 630–644. doi: 10.1109/TMI.2015.2487997 .

Urban, G., Tripathi, P., Alkayali, T., Mittal, M., Jalali, F., Karnes, W., Baldi, P., 2018.

Deep learning localizes and identifies polyps in real time with 96 percent ac- curacy in screening colonoscopy. Gastroenterology doi: 10.1053/j.gastro.2018.06.

037 .

Van Rijn, J.C. , Reitsma, J.B. , Stoker, J. , Bossuyt, P.M. , Van Deventer, S.J. , Dekker, E. , 2006. Polyp miss rate determined by tandem colonoscopy: a systematic review.

Am. J. Gastroenterol. 101 (2), 343 .

Vázquez, D. , Bernal, J. , Javier Sánchez, F. , Fernández-Esparrach, G. , López, A. , Romero, A. , Drozdzal, M. , Courville, A. , 2016. A benchmark for endoluminal scene segmentation of colonoscopy images. J. Healthcare Eng. 2017 .

Werbos, P. , 1974. Beyond regression: New tools for predicting and analysis in the behavioral sciences. Ph.D. thesis. Harvard University .

Wickstrøm, K., Kampffmeyer, M., Jenssen, R., 2018. Uncertainty modeling and in- terpretability in convolutional neural networks for polyp segmentation. In: Pro- ceedings of the IEEE (MLSP), pp. 1–6. doi: 10.1109/MLSP.2018.8516998 . Wimmer, G., Tamaki, T., Tischendorf, J., Häfner, M., Yoshida, S., Tanaka, S., Uhl, A.,

2016. Directional wavelet based features for colonic polyp classification. Med.

Image Anal. 31, 16–36. doi: 10.1016/j.media.2016.02.001 .

Yu, S., Príncipe, J.C., 2018. Understanding autoencoders with information theoretic concepts. arXiv: 1804.0 0 057 .

Zech, J.R., Badgeley, M.A., Liu, M., Costa, A.B., Titano, J.J., Oermann, E.K., 2018. Vari- able generalization performance of a deep learning model to detect pneumo- nia in chest radiographs: a cross-sectional study. PLOS Med. 15 (11), 1–17.

doi: 10.1371/journal.pmed.1002683 .

Zeiler, M.D. , Fergus, R. , 2014. Visualizing and understanding convolutional networks.

In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (Eds.), Proceedings o the ECCV.

Springer International Publishing, Cham, pp. 818–833 .