Contents lists available atScienceDirect
Heliyon
journal homepage:www.cell.com/heliyon
Research article
Evaluation of the quality indicators in dehazed images: Color, contrast, naturalness, and visual pleasingness
Laksmita Rahadianti
a,∗, Aruni Yasmin Azizah
a, Hilda Deborah
baFacultyofComputerScience,UniversitasIndonesia,Indonesia
bDepartmentofComputerScience,NorwegianUniversityofScienceandTechnology(NTNU),Norway
A R T I C L E I NF O A B S T R A C T
Keywords:
Dehazing Imagerestoration Hazyimages Imagequality Psychovisualexperiment
Hazyimagessufferfromlowqualityduetoblurring,veilingeffects,andlowcontrast.Toimprovetheirvisibility, dehazingmethods attempttorestorethem totheircorrespondingclear scenes,often byfocusingmoreon obtaininganaccurateestimatebasedonaknowngroundtruth.Theperceptualqualityofdehazedimages, whichcanbedescribedbymeansofobjectiveandsubjectivequalityassessments,isoftennotconsidered.This paperprovidesaqualityassessmentofdehazedimages,focusingonaspects,e.g.,color,imagestructure,and naturalness.Fourimagedehazingmethodsareconsidered,i.e.,ContrastLimitedAdaptedHistogramEqualization (CLAHE),Dark Channel Priorand Refinement(DCP-R),Perception InspiredDeep Dehazing Network with Refinement(PDR-Net)andConditionalGenerativeAdversarialNetwork(CGAN)Pix2pix.Thedehazingresults arethenputthroughobjectiveandsubjectiveassessments,foracomprehensiveevaluationonimagequality.
Overall,Pix2pixshowsthebestresults objectively,excellingin therecoveryof colorandimagestructure.
AlthoughitisoutperformedbyDCP-Rintermsofnaturalness,oursubjectiveassessmentshowsthatPix2pix isalsomostpreferredbyhumanobservers.
1. Introduction
Inadigitalimageacquisition,animagesensorcaptureslightrays reflected byobjects. Inclearconditions, these lightrays areableto travelunhinderedintheenvironment.However,inconditionssuchas fogandhaze,micro-particlesinthesurroundingmediainterferewith thepropagatinglightbychangingitsdirectionandintensity.Thistype ofmediaisoftencalledscatteringorparticipatingmediabecauseitplays anactiveroleinimagecapture.Animagecapturedinscatteringmedia, e.g.,hazyimages,willappearwithblurringeffects,lowcontrast,and low visibility.Theappearanceof hazyimagesmakesitverydifficult forobserverstoobtainusefulinformationof asceneandtheobjects init,makinginchallengingforimageunderstandingbybothhuman andcomputervision.Thus,itisoftennecessarytodehazeahazyimage toitscorrespondingclearscene [1]priortoanyfurtherprocessingor analysis.
Image dehazingmethodsseektorestorea hazyimagetoits esti- matedclearcounterpart,which canbe achievedby,e.g.,contrast or visibilityenhancementtechniques [2, 3].Otherapproachesstudythe physicalprocessofscatteringmodelandtheninvertitwiththehelpof
*
Correspondingauthor.E-mailaddress:laksmita@cs.ui.ac.id(L. Rahadianti).
additionalconstraints,e.g.,statisticalorspatialpriors [4,5].Deepnet- workshavealsobeenusedtomodelthetranslationbetweenhazyand clearimagedomains [6,7].
Thispaperis builton apreliminarystudyof thequalitydehazed images by Azizah, et al. [8]. In this paper, we evaluate theimage qualityofdehazedimagesusingfourdifferentdehazingmethods.Con- trastLimitedAdaptiveHistogramEqualization(CLAHE)[9]isanimage enhancementtechnique,asopposedtoarestorationtechnique,toim- provequalityandvisibility.Thedarkchannelprior(DCP) [5]usesthe physicalscatteringmodelandattemptstoreverseittoobtaintheorig- inalclearimage.Then,twodeeplearningbaseddehazingmethodsare alsoconsidered. Perception-inspiredSingle Image Dehazing Network withRefinement(PDR-Net)[10]isthestate-of-the-artgenerativemodel thatusesaperception-basedapproachtotranslatehazyimagestotheir clearcounterparts.Pix2pix [11]isaConditionalGenerativeAdversar- ialNetwork(CGAN)thathasbeenusedextensivelyforimagetoimage translation.Inourcase,wewilluseittotranslatehazyimagestotheir clearcounterparts.
Dehazingresultsareoftenevaluatedbyhowclosetheyaretotheir knowngroundtruth.However,hazyimagescapturedinnaturalandreal
https://doi.org/10.1016/j.heliyon.2021.e08038
Received8June2021;Receivedinrevisedform13August2021;Accepted16September2021
2405-8440/©2021TheAuthor(s). PublishedbyElsevierLtd. ThisisanopenaccessarticleundertheCCBY-NC-NDlicense(http://creativecommons.org/li- censes/by-nc-nd/4.0/).
Fig. 1.Image formation model in scattering media.
settingsoftendonotcomewithgroundtruthclearimages,resultingin thelackofstandardizedanduniversaldatasets.Additionally,themajor- ityofexistingworksevaluatedehazedimagesbyvariationsofobjective qualitymetrics,makingbenchmarkingofthequalityofdehazedimages verydifficulttodo [12].Inourwork,weaddressthechallengebyfram- ingtheevaluationofdehazingmethodsfromadifferentstandpoint.In additiontomeasuringaccuracywithvariousobjectivequalitymetrics, wealsocarryoutasubjectiveimagequalityassessment(IQA)basedon visualpleasingness.Wethenstudythecorrelationofbothobjectiveand subjectiveassessments,providingnotonlyamorecomprehensiveIQA ofdehazedimages,butalsoanalysesoftheirperceptualattributes.Re- gardingthedataset,aimingtoencompassdifferenttypesofhazyscenes, weuseacompilationofvariousdehazingdatasetsandalsogeneratea simulateddataset.Wehopethispaperwillsparkdiscussionandfurther researchon theconceptofqualityinimagerestorationtaskssuchas dehazing.
2. Onimagedehazing
Todehazehazyimagestotheirclearform,itisnecessarytostudy thephysical process thatcaptures a hazyimagefrom a clearscene.
This sectionprovidetheimageformationmodelinscatteringmedia, thestate-of-the-artimagedehazingmethods,andtheIQAapproaches fordehazedimages.
2.1. Hazyimages
Digitalimagesareformedwhenlightraysarecapturedonadigital sensorgridofacamera.Theselightraysusuallycomefromaprimary lightsource,andarereflectedbyasceneintothelensofacamera.The lightraysthatarecapturedatdifferentpixellocationsbuildupanim- agerepresentationofthescene.Thisprocessisquitestraightforwardin clearmedia,whichallowsthelightraystopassthroughunhindered.
However,thisprocessisnotpossibleinscatteringmediaenvironments asmicro-particlescontainedinitmayalterthedirectionofthetravel- ing raysandscattertheminmanydirections.Thiswillalsoresultina reducedlightintensitybecausethemediawillabsorbordiverttheorig- inallight.Themediaactivelyinterfereswithimageformationresulting in abelowstandardrepresentationofthereal scene.Thealteredin- tensitieswillthuscompromisethevisualfeaturesnecessaryforimage understanding[13].
Theimageformationmodelusedinthispaperisbasedonthegen- eralatmosphericscatteringmodel [14],seeillustrationinFig.1.This modelassumesthatthemicro-particlesinthemediaareverysmall,so thescatteringcanbeconsideredhomogeneousandhasrelativelylow density. These assumptionsholdin most naturalhazyconditions. In these naturalconditions,theappearance ofobjectsintheimagewill differbasedontheirdistancefromthecamera.Themicro-particlesin themedia coupledwiththeamountof distancetraveled, directlyaf- fecttheamountoflightthatisabletopenetratethemedia [15].The fractionoflightthatisabletopenetratethescatteringmediadecayex- ponentiallyproportionaltothedistancefromthecamera𝑑,andcanbe definedas:
𝑡=𝑒−𝛽⋅𝑑 (1)
where𝛽isthescatteringcoefficientofthemedia,anddisthedistance tothecamera.Inahomogeneousscatteringmedia,thecapturedhazy image𝐼ofanoriginalclearscene𝐽canbedefinedusingthetransmis- sion𝑡asfollows,where𝐴istheairlight:
𝐼=𝐽⋅𝑡+𝐴(1 −𝑡) (2)
The term airlight 𝐴 refers to the color of ambient light in the scene [4]. Thisconsistsoftheaccumulation ofscatteringeffectsand carriesnoinformationabouttheoriginalsceneitself.Airlightisscat- teredin the scene,creating aveiling effectthat subtlyobscures the entireimage[16].Theexactappearanceandhueoftheveilingeffect willdependhighlyonthescatteringmediaintheenvironment.Forim- agescaptured inenvironmentssuchasfogorhaze,theairlightoften presentsapalewhitishcolor.Asafinalresult,theimagescapturedin theseenvironmentswillnotbeabletoportraythesceneperfectly,due toobscureddetails,lossof intensityandlow contrast.Inthispaper, theseimageswillbereferredtoashazyimages.
2.2. Imagedehazing
Imagedehazingisanimagerestorationtask,mappinghazyimages totheirclearcounterpartswiththehighestpossibleaccuracy.However, there aremany cases in which a precise ground truth is not avail- able.Thus,werestatetheimagedehazingprocesstoworktowardsthe achievablegoalofobtainingavisuallypleasingimagewithbetterqual- ity.Theendresultshouldbejustgoodenoughsothatobserversareable tounderstand thescenethroughits visualfeatures.A visuallypleas- ingimagedoesnotnecessarilyneedtobeanexactmatchtoaknown standard,thusitcanbeachievableusingimageenhancementmethods.
Thesemethodshavetheadvantageofnotrequiringaknowncleartar- get,astheiraimistosolelyreducethehazyeffectssuchasdescribed inSection2.1.Thesemethodsattempttoimprovethevisibilityofthe imagethroughmethodssuchascolorcorrection [3,17],contrastcor- rection [2,18],contrastbalancing [9,19],andothers.
Itis alsopossible tomodel thephysical transformationbetween hazyandclearimages.Themodelcan thenbe usedtorestorehazy imagestotheirclearformusingsingleimagedehazingtechniques.Sin- gleimagedehazingconsidersthephysicalscatteringmodeldescribed inSection2.1asthe processinwhich ascenebecomes hazy.Then, singleimagedehazingfollowstheimageformationmodelofhazyim- ages in Eq. (2) and uses its inverse to extract the original scene𝐽 fromthehazeimage𝐼followingEq. (3).Thisequationinvolvesmany unknowns.Thus,singleimagedehazingmethodsmustestimatetheun- knownvariablesinphases,i.e.,estimatingtheambientlightorairlight 𝐴,̂ predictingtransmissionmap̂𝑡,thenrecoveringtheoriginalscene ̂𝐽 (Fig.2).
̂𝐽=𝐼−𝐴
𝑡 +𝐴 (3)
Unfortunately,thetransmission𝑡orthedistance𝑑 arerarelyknown.
Furthermore,thescatteringcoefficientofthemedia𝛽canbedifferent foreveryscene,hinderingaone-fits-allestimateforallimages.Tosolve thisissue,itisnecessarytoemployadditionalconstraintstothemodel.
Onepossibleapproachistousestatisticalpriors,i.e.,theDarkChannel
Fig. 2.General step-by-step process of single image dehazing.
Fig. 3.An example of a clear scene (a) and its depth map (b) from the NYU Depth Dataset [21], and the (c) synthesized hazy image generated based on Eq. (2).
Prior(DCP) [5]orAdaptiveDarkandBrightChannelPrior [20].These priorobservationsof naturalhazyimagesformpre-determinedinfor- mationtohelpestimatingvariablesinvolvedinanimageformation.
Theproblemofimagedehazingcanbemodeled asanimagetoim- agetranslationproblem,e.g.,fromthesetofhazyimagestotheirclear counterparts. There havebeen variousdeep networks that proposed thispurpose,usingvariousstructuresofconvolutions [6,22],attention cues [23]orevenhumanperceptioncues [10].Inrecentyears,these deepnetworkbasedapproacheshavebeenpopularforotherimageto imagetranslationproblemsaswell.Imagetranslationcanbeusefulfor variousapplications,e.g.,semanticsegmentation [24],domainadapta- tion [25],orimagetodepthestimation [26].
2.3. Imagequalityassessment
Imagequalityassessment(IQA)triestomeasurethequalityofdig- ital images in depicting the original scene, andin turn, in relaying information.Itcanbedividedintosubjectiveandobjectiveapproaches.
ObjectiveIQAinvolvesimagequalitymetricstocomputeanddepict imagequalityquantitatively.Thesemetricscanbedividedintofull-,re- duced,orno-referencemetrics.Full-reference(FR)metricsneedaground truthimage,suchthatthedifferenceorerrorfromthetargetimagecan be computed [27].Reduced-reference (RR) metricscan be used when onlyincompletereferencesareavailable,andno-reference(NR)metrics whentherearenoneatall.NRmetricsarealsoreferredtoasblindimage qualitymetrics[28].SubjectiveIQAgivesthetasktohumanobservers, toevaluatethequalityofasetofimagesbasedonacertaincriteriaby, typically,indicatingtheirpreferences.Thehumanvisualsystemiscon- sideredthemostreliablebiologicalimagecapturedeviceabletojudge visualimagequalityinaconsistentmanner [29].
ThemainissuefortheIQAofdehazedimagesisthelackofstan- dardizedevaluationprotocol.Firstofall,hazyimagesthemselvesare difficulttocomeby,andthereisalackofdatasetsthatcanbe used forqualitybenchmarking.Furthermore,itisverydifficulttoobtaina referencesetofhazyimagesandtheirexactclearcounterpart.Natural hazeoccursinoutdoorenvironmentswheremanyfactorsareinvolved inimagecapture.Mostofthesefactors areuncontrollableforexperi- mentaldatacaptureandthesceneisneverreliableenoughtocapture
inboth hazyandclearconditions.The nextbest optionis capturing imagesusingsynthetic haze,butitisstilldifficulttoobtainthe ex- actsameconditiontocapturethesceneinbothconditions.Thiswork wouldalsohighlighttheneedtoconsiderIQAfordehazedimages.Nat- uralatmosphericscattering [14]occursoutdoors evenintheclearest ofconditions,thushumanobserversexpectsomehazeinanynatural image.Meanwhile,objectivemetricswouldseeanyremaininghazeas anindicationoflowquality.Thus,itisunrealistictorelyonaccuracy alone,sinceitisimportanttoconsidervariousfactorsofimagequality dependingonthecontextoraimofthetask.
3.Materialsandmethods
This section will describe the image dehazing experiments con- ductedforatestsetof50images,followedbyathoroughexplanation oftheusedIQAmethods.
3.1. Dataset
Inourexperiments,acombineddatasetofhazyimageswithground truthclearimagesfrom3differentsourceswereused.Theentiredataset iscomprisedof600hazy-clearimagepairs,whichthendividedintoa trainingsetof550imagestotrainthePDR-Net(Section3.2.3),anda testsetof50images.OnlythetestsetwasusedtoperformIQAonthe dehazingresults.
3.1.1. Synthetichazyimages
TheNewYorkUniversity(NYU)depthdatasetiscomprisedof1449 pairsofRGB-Depth(RGBD)data,i.e.,indoorscenesandtheircorre- spondingdepthmaps [21],seeanexampleinFig.3aand3b.Theyare capturedinvariouscommercialandresidentialbuildings,withdepth mapsobtainedusingtheMicrosoftKinect.
Forourexperiment,wegeneratedsynthetichazyimagesfrom210 randomlyselectedRGBDpairsfromtheNYUdatasetfollowingEq. (2), seeexampleinFig.3c.First,thepixel-wisedepthmap𝑑wereusedto createadensetransmissionmap𝑡usingseveralscatteringcoefficient values𝛽∈ {0.1,0.2,0.3,0.4}.Then,𝑡wasusedtosynthesizehazyimages 𝐼usingdifferentairlightvalues𝐴∈ [0.7,1][26,30].
3.1.2. Hazyseriesdataset
FourdatasetsareavailablefromTheNewTrendsInImageRestora- tion And Enhancement Workshop And Challenges (NTIRE) chal- lenge [31].TheIndoorHaze(I-Haze)dataset [32]consistsofhazy-clear imagepairscapturedinindoorconditions,whiletheOutdoorHaze(O- Haze)dataset [33]wascapturedinoutdoorconditions.Inbothdatasets, hazewasgeneratedusingtwoprofessionalfogmachines(LSM1500PRI) which was dispersed using fanstoensure homogeneity. The Dense- Haze [34]andNon-Homogeneous Haze(NH-Haze) [35]datasetsare bothextensionsoftheO-Hazedataset.Thehazefromthefogmachines was releasedforalongerdurationtocreateadenserhazeeffect for theDense-Hazedataset.ForNH-Haze,thegeneratedhazewasnotdis- persed evenlythrough the scene ensuringa non-homogeneous haze condition.Atotalsetof180hazyimagesandtheirclearcounterparts wascollectedfromallfourofthesedatasets.
3.1.3. REalisticsingle-imageDEhazingdataset
Li,etal. [12]createdtheREalisticSingle-ImageDEhazing(RESIDE) datasetwiththeintentiontoprovideastandardizedlarge-scaledataset of hazy-clearimagepairstofacilitate thebenchmarkingofdehazing methods.RESIDEconsistsofrealisticallysimulatedhazyimagesandis organizedinto3subsets,i.e.,theIndoorTrainingSet(ITS),Synthetic Outdoor TrainingSet (SOTS),andtheHybridSubjective Testing Set (HSTS).Amongallthreesubsets,SOTSisthemostsuitableforourneeds asithasmanyimagesof urbanoutdoorhazyscenes.Forourexperi- ment,210outdoorimagepairsfromtheSOTSsubsetwererandomly selected.
3.2. Dehazingmethods
Fourmethodsareusedinourexperiments.Theyareanimageen- hancementapproach,i.e.,CLAHE [9],asingleimagedehazingmodel usingdarkchannelprior(DCP) [5],andtwodeeplearningbasedde- hazingmodels.Thedeeplearningbasedmodelsselectedencompassa generativemodelPDR-Net [10]andanadversarialmodelPix2pix [11].
3.2.1. Contrastlimitedadaptivehistogramequalization(CLAHE) Oneoftheeffectsthatoccurinhazyimagesislowcontrastandloss ofdetail.Thisiswhyhazyimagesaresodifficulttoperceive,because contrast carries significant information about image structure [36].
Thus,toimprovevisibility,wecanattempttoimproveorcorrectthe contrast.Contrastcorrectioncanbedonebybalancingthedistribution ofintensityvaluesinanimage.Thehistogramofalowcontrastimage willbeconcentratedinanarrowrangeofvalues,makingdifferentiating intensityvaluesdifficult.
Wecanimproveimagecontrastbymappingtheintensityvaluesto awiderrangethroughhistogramequalization.Theincreaseofdiffer- encebetweenindividualintensitieswillamplifyedgesandimagestruc- tures,herebyimprovingvisibility.Therehavebeenvariousproposals towardshistogramequalization,andinthispaper,weselectthewell- knownandeasilyimplementedCLAHE [9].CLAHEseparatesanimage intodifferentcontextualregions,andperformsanadaptivehistogram equalizationoneach regionseparately. Additionally,italsoclipsthe intensitydistributionatacertainlimittopreventover-saturationinho- mogeneous areas.Our experiments involvecolorimages, sowe first transformedtheRGBcolorimagetotheCIELABcolorspace[37]and CLAHEwasperformedonthebrightnesschannel𝐿∗[9].
3.2.2. Darkchannelpriorandrefinement(DCP-R)
The imageformation model in Eq. (2) shows thetransformation fromaclearimage𝐽toahazyimage𝐼,whichwewouldliketoreverse.
However,thisequation involvesmany unknownvariables,makingit difficulttosolve.Additionalconstraintsareneededtoallowreversing theequationtoestimate𝐼 from𝐽. Inthis paper,weselecttheDark ChannelPrior(DCP) [5]whichiswidelyacknowledgedandusedasa basisofmanydehazingworks.
DCPisastatisticalpriorbasedontheobservationofnaturaloutdoor images.Itstipulatesthatforlocalpatchesofnon-skyareasinnatural images,thereisalwaysonecolorchannelthathasaverylowvalueor min(𝐼𝑠)≈ 0,𝐼𝑠∈ [𝑅,𝐺,𝐵].Thesedarkpixelsoccurduetonaturaldark objects,lighting,anddistinctlycoloredobjects.Insuchcases,thereis oftenonechannelwithlowintensity,whileotherchannelscontainthe colorinformation,thuscreatingtheaforementioneddarkchannel.
Darkchanneltypicallyoccursinnaturalclearimages.Inhazyim- ages,adarkchannelbecomessaturatedwiththeeffectofhaze,which canbedenotedasaproportionofairlightintensity𝐴fromtheimage formationmodelinEq. (2).Basedon theconceptoftransmission in Eq. (1),itisknownthatairlightwillaggregate in thedarkchannel, withhigherquantityindicatingfurtherdistance [5,38].Consequently, thedistancevaluewillalsogrowproportionallywiththevalueofDCP, allowingittobeusedasanindicatorofrelativedepth.He,etal. [5]
proposedamethodtoestimatetransmissionbasedonDCPasdescribed in Eq. (4). Assuming anRGB hazyimage 𝐼 in theRGB colorspace (𝑠∈ [𝑅,𝐺,𝐵]),thetransmission ̂𝑡can be estimated foreverypixel 𝑥 inthehazyimage𝐼 consideringeverypixel𝑦inthelocalareaΩ(𝑥) surrounding𝑥.
̂𝑡𝐷𝐶𝑃(𝑥) = 1 − min
𝑠∈{𝑅,𝐺,𝐵}
(
𝑦∈Ω(𝑥)min (𝐼𝑠(𝑦)
𝐴𝑠
)) (4)
Note that the value of airlight 𝐴 in the RGB color space (𝑠∈ [𝑅,𝐺,𝐵])orits estimationisneededtocompute ̂𝑡.Therearevarious approachesthatcanbeusedtoestimate𝐴,thesimplestofwhichisby usingtheDCPvalueitself.He,etal. [5]setthevalueofairlightwith thepixelvalueatthelocationofthemaximumDCP.Airlight,orambi- entlight,isoftenequatedwiththecolorofthesky,whichcanbefound atamaximumdistance𝑑≈ ∞.SincealargerDCPvaluewouldindicate afurtherdistance,themaximumvalueofDCPshouldindicatethefur- thestpointaswell,atwhichpointcanbeusedasanestimateof ̂𝐴.With theestimatedtransmission̂𝑡basedonEq. (4),theoriginalscene𝐽 can berecoveredforeverypixel𝑥basedontheprinciplesofDCPaccording toEq. (5).Toavoidadivisionbyzero,itisnecessarytolimitthevalues of̂𝑡byaminimumvalueof𝑡0.
̂𝐽𝑠(𝑥) = 𝐼𝑠(𝑥) −𝐴𝑠 max(̂𝑡(𝑥), 𝑡0
)+𝐴𝑠 (5)
Inourexperiments,imagedehazingusingDCPwasperformedbya two-stepapproach.Thefirststepresultsinanapproximatedclearim- ageofthescene.However,sinceDCPassumesconstanttransmission inlocalareas,theseimagessufferfromthelossofdetailsorappearing tobepatchy.Thus,arefinementstepwasaddedusingtheguidedfil- ter [39],aimingtofurtherimproveandsmooththedehazedimages.
Withanedge-preservingsmoothingproperty,theguidedfilterusesa referenceimagetoguidetherefinementprocess,resultinginalinear transformoftheguidanceimage.Finally,thecompletesingleimagede- hazingapproachinthispaperisacombinationofDCPandrefinement usingguidedfilter,furtherreferredtoasDCP-R,anditwasusedtode- hazethe50hazyimagesfromthetestset.
3.2.3. Perception-inspireddehazingnetworkwithrefinement(PDR-Net) Li,etal. [10]proposedPDR-Net,aperception-baseddeeplearning architecturetomodelend-to-endimagetoimagetranslationbetween hazyimagesandtheir clearcounterparts.Similar toDCP-R,ittreats theprocessofdehazingastwosteps,i.e.,hazeremovalandrefinement, eachimplementedastwoseparatesub-networks.PDR-Net employsa perceptuallossin thehazeremovalsub-network,todescribehigher- levelperceptualcues.Meanwhile,therefinementsub-networkemploys amulti-term losstorecover colordistortionand enhancethevisual qualityofthedehazedimage.Throughthesub-networks,PDR-Netde- hazesimagesintwosteps,resultinginnotonlyanaccuratelydehazed image,butalsoavisuallypleasingone.
Inourexperiments,PDR-Netwastrainedusingthetrainingsetof 550hazy-clearimagepairs.Sincedeeparchitecturesoftenbenefitfrom
moresamples,thetrainingsetwasaugmentedbyflippingeachimage horizontally, creatingatotal of1100 trainingimages.Followingthe proceduredetailed in [10],each sub-networkwastrainedseparately, with700trainingimagestotrainthehazeremovalsub-network,and 400fortherefinementsub-network.Eachsub-networkwastrainedfor 200epochs.AswithourDCP-R,PDR-Netwasalsousedtodehaze50 hazyimagesofthetestset.
3.2.4. Pix2pix
Commongenerative networks mayexperience difficulty in train- ingimagetoimagetranslationsduetothelarge numberofvariables andprobabilisticcomputationstoestimate.Thus,theGenerativeAd- versarialNetworks(GAN) [40]frameworkmaybeusedtoassistwith trainingthesegenerativenetworks.AGANsubjectsagenerativemodel toadiscriminativemodel,trainedtoworkasopponents.Thegenera- toristrainedtogeneraterealisticrenderingsoftheintendedoutputs, whilethediscriminatoraimstodetectthese reconstructionsfromthe groundtruth output [40]. AConditionalGenerativeAdversarialNet- work(cGAN)extendstheconventionalGANframework,i.e.thegener- atoranddiscriminatorareconditionedwithextrainformationtoguide thegenerativeprocess [41].
Pix2pixisacGANmodelproposedforgeneralnon-specificimageto imagetranslationtask [11].Pix2pixusesamodifiedU-Net [24]asits generativemodel,guidedbyaPatchGANdiscriminator [42].Pix2pix was thoroughly examined using various image to image translation tasks, suchasgrayscaletocolorimagesordaytonightimages.The resultsshowitsabilitytoreproduceavisuallysimilaroutputtothetar- getforseeminglyarbitraryimagepairs.Consideringadehazingtask, weareoptimisticthatthediscriminatorcomponentcanimprovethevi- sualpleasingnessoftheresultingimage.Sincethediscriminatorjudges thegeneratedoutputbasedonoverallsimilarity,notonlyonamini- mizederror,webelievethismaymimictheobservationorjudgement ofahumanobserver.Inourexperiments,wealsousedPix2pixtotrain image toimage translationfromhazyimagestotheir clearcounter- parts.FollowingthetrainingandtestingsetupofPDR-Net,Pix2pixwas alsotrainedfor200epochsusingthesameaugmentedhazy-clearim- agepairs,with1100trainingimages.Thefinaltrainedmodelwasthen usedtodehaze50hazyimagesofthetestset.
3.3. IQAofdehazedimages
Thedehazingresultswerethenassessedbymeansofobjectiveand subjectiveIQAs.Theimagequalitymetricsandthedesignofthepsy- chovisualexperimentforthesubjectiveIQAaredetailedinthissection.
3.3.1. Objectivemetrics
Inthis study,fourIQA metricswereselectedtoevaluatevarious quality aspectsofan image.Three metricsarefull-referencemetrics thatassignaquantitativevaluetotheaccuracyofdehazingcompared tothegroundtruth,basedonthequalitycuesfocusedoninthispaper, namelycolor,imagestructure,andpixelintensity.
First, we evaluate color using Δ𝐸𝑎𝑏∗, a color difference function defined for the CIELAB color space [37], where perceptual non- uniformitiesareaccountedfor.Ahazyimage𝐼anditsdehazedversion 𝐽 willbeconvertedtotheCIELABcolorspacebeforethecomputation oftheircolordifferenceasinEq. (6).ImagesintheCIELABcolorspace havethreecomponentsorchannels,i.e.,𝐿∗(lightness),𝑎∗(green-red chromaticity),and 𝑏∗ (blue-yellow chromaticity).Therangeof color differencevaluesfromΔ𝐸∗𝑎𝑏is[0,100],whichwerescaleto[0,1],with alowervalueindicatingbetterquality.
Δ𝐸𝑎𝑏∗(𝐼, 𝐽) =
√
(𝐿∗𝐽−𝐿∗𝐼)2+ (𝑎∗𝐽−𝑎∗𝐼)2+ (𝑏∗𝐽−𝑏∗𝐼)2 (6) Toevaluatetheaccuracyofpixelwiseimageintensities,weusethe rootmeansquareerror(RMSE),showninEq. (7).RMSEcomputesthe differencesinimageintensityorbrightnesslevels[43].Assumingpixel
valuesintherangeof[0,1],theRMSEoftheentireimageisaveraged overthetotalnumberofpixelsintheimage(𝑛).Thus,theRMSEwill alsorangebetween0and1,withlowervalueindicatingabetteresti- mationoftheclearimage.
RMSE(𝐼, 𝐽) =
√√
√√1 𝑛
∑𝑛 𝑖=1
(𝐼𝑖−𝐽𝑖)2 (7)
Theaccuracyofthepixelintensitiesthemselves,doesnotensurea cleardistinctstructureinanimage.Thus,wealsousedthestructural similarityindexmeasure(SSIM)[44]toevaluatetheimagestructure recovered.SSIMisusedtodefinetheperceptualdifferenceoftwoim- agesbymeansoftheirluminance(𝑙),contrast(𝑐),andstructural(𝑠) components.TheformulaofSSIMisprovidedinEq. (8),with𝜇, 𝜎𝐼2, and𝜎𝐼𝐽 asthemeasureofaverage,variance,andcovariancebetween 𝐼and𝐽,respectively.Otherparametersintheformulaare𝑐1= (𝑘1𝐿)2, 𝑐2= (𝑘2𝐿)2,𝑐3=𝑐2∕2,𝐿= 2𝑏− 1,𝑘1= 0.01,𝑘2= 0.03,(𝛼,𝛽,𝛾)asadapt- ableweights,and𝑏asthenumberofbitsperpixelinanimagewhich willtypicallybe8.
SSIM(𝐼, 𝐽) = [𝑙(𝐼, 𝐽)𝛼⋅𝑐(𝐼, 𝐽)𝛽⋅𝑠(𝐼, 𝐽)𝛾],where 𝑙(𝐼, 𝐽) = 2𝜇𝐼𝜇𝐽+𝑐1
𝜇2𝐼+𝜇2𝐽+𝑐1
, 𝑐(𝐼, 𝐽) = 2𝜎𝐼𝜎𝐽+𝑐2
𝜎𝐼2+𝜎𝐽2+𝑐2
, 𝑠(𝐼, 𝐽) = 𝜎𝐼𝐽+𝑐3
𝜎𝑖𝜎𝐽+𝑐3
. (8)
Therangeof values that theSSIMcan take on is [−1,1], witha largervalueindicatingbetterquality.Meanwhile,withalloftheother metricsusedinthisstudy,betterqualityisindicatedwithalowervalue.
Furthermore,SSIMisalsotheonlymetricthatcanberepresentedwith anegativenumber.Thus,theSSIMisconvertedintoaninvertedSSIM whichwillbedenoted𝑆𝑆𝐼𝑀−1withamodifiedrangeof[0,1],witha lowervalueindicatingbetterqualitybasedonEq. (9).
SSIM−1(𝐼, 𝐽) = 1 −
(SSIM(𝐼, 𝐽) + 1 2
)
(9) BelongingtothecategoryofFRmetrics,Δ𝐸∗𝑎𝑏,RMSE,andSSIM−1 allrequiretwoimagesasinput,i.e.,groundtruthorreferenceimage andthetargetof evaluation.Notethattheimagedimensionofboth imageshavetobeexactlythesame.Thesethreemetricsmeasurethe accuracyofthedehazedimagecomparedtotheknowngroundtruth.
Wehavealreadysuggestedthatoften,accurateimagesdonotnec- essarilyhavebettervisualquality.Frequently,humanperceptionwill considerabstractconceptssuchasnaturalness. Thus,wealsouseda metricthatwasproposedtomeasurenaturalness,TheNaturalnessIm- ageQualityEvaluator(NIQE)[45].NIQEisanNRmetricthatcomputes thescoreofanimage againstamodelof naturalsceneimages.This modelisrepresentedasmultidimensionalGaussiandistributions.The rangeofvaluesthatNIQEcantakeonis[0,∞),withalowervaluein- dicatingbetterimagequality.
3.3.2. Psychovisualexperiment
BasedontheobjectiveevaluationresultinSection3.3.1,wecould naivelyinferthatadehazedimagewithagoodobjectiveresultindicates thatitisvisuallypleasingtothehumaneye.Thesameprincipleshould intuitivelyapplytotheoppositecase.However,subjectiveandobjective measuresdonotalwaysalign [44].Thus,subjectiveIQAbymeansof apsychovisualexperimentwasalsocarriedoutforthetestsetof50 images.
Dehazingresultsofthefourdehazingmethodsweresentoutina surveyinwhichobserverswereaskedtoidentifytheirpreferredresult.
ThedisplayofthepsychovisualexperimentcanbeseeninFig.4.Each imageisshownonanindividualslide,withaneutralgraybackground andtheslide numberwritten atthetop-left of eachslide indicating whichimageiscurrentlydisplayed.Oneachslide,fiveimagesaredis- played,i.e.theoriginalhazyimageinthefirstrowandfourdehazed imagesonthesecondrow,eachobtainedusingCLAHE,DCP-R,Pix2pix, andPDR-Net,respectively.Observerswerethenaskedtorankthefour dehazedimagesfromtheimagethatis mostvisuallypleasingtothe
Fig. 4.Image arrangement for the psychovisual experiment.
least.Consideringobserverfatigue,eachobserverwasonlyshown25 randomlyselected imagesfromtheentiretestsetof 50images. Ob- serverswerefreetouseanycriteriatheydeemimportanttodetermine theirpreference.
Responses from observers were then populated using the Mean OpinionScore(MOS) [46].TheMOSofimage𝐼isdeterminedbythe scores𝑆assignedtoitby𝑁observers,whichcanbecomputedbased onEq. (10).
MOS(𝐼) = 1 𝑁
∑𝑁 𝑛=1
𝑆𝑛 (10)
Inourexperiment,thescoresweregivenbasedontherankordergiven byhumanobservers.Foreachresponse,ascoreof1wouldbeassigned tothemostpreferredresult,2tothenext,3tothethird,and4tothe leastpreferredresultforthatparticularimage𝐼𝑖.Thus,alowerscore willindicatebetterperceivedquality,whichisconsistentwiththeob- jective metricsin Section3.3.1. Thescore for each image wasthen averagedoverthenumberofitsobservers𝑁.Theaveragescorewas thenfurtherconvertedtoamodifiedrangeof[0,1].Thiswascomputed forall50testsetimages𝐼𝑖,for𝑖∈ [1,50].Attheendoftheexperiment, observerswerealsoaskedtogiveopen-endedcommentstoelaborate furtherabouttheirobservationsaboutthedehazedimages.Thesecom- mentsmaygive anindication aboutthefactorsthatinfluenced their preferenceintermsofcolor,structure,clarity,ornaturalness.
4. Resultsanddiscussion
Thissectionwilldisplay,evaluate,andanalyzetheresultsusingthe fourdehazingmethodsinourexperimentsonthetestsetof50images describedinSection3.2.Arandomlyselectedsubsetoftheresultsare showninFig.5.Fromtheresults,itisvisiblethatallmethodsperformed wellinlightandsimulatedhazesuchasintheO-Haze,Synthetic,and RESIDEimages.However,itismorechallengingtohandleimagesfrom theDense-HazeandNH-Hazedatasets.
Inthecaseoftestimageswithdensehaze,noneofthefourdehaz- ingmethodswereabletorecoverthescenecorrectly,asshowninthe Dense-HazecolumninFig.5.Fig.6showsmoredetailsoftherecovered Denze-Hazeimages,whicharestillshroudedinaveilinghaze.CLAHE wastheleastsuccessful,duetotheover-saturatedhazethatdominates theimageandhenceitshistogram.DCP-Rwasmoresuccessfulinrecov- eringdetailsofthesceneandtheresultsshowsmoresaturatedcolors.
However,thesecolorsarefarfromcolorsinthegroundtruthimage.
PDR-Netproducesadarkerimagewithlesssaturatedcolors.Italsoap- pearstohavealotoftexturesimilar tothatof thetreeobjectinthe groundtruthimage.However,notmuchinformationwassuccessfully restoredasidefromthewhitestructuresattoprightcorneroftheim- age.ThedehazedimageusingPix2pixwasabletoreconstructthearea
Table 1.AverageobjectiveandsubjectiveIQAmetricresults onthetestsetusing4dehazingmethods.Allmetricshavebeen convertedandscaledsothatthelowervaluewillindicatebetter quality.
Objective IQA
Metric Range↓ CLAHE DCP-R PDR-Net Pix2pix RMSE [0,1] 0.4221 0.3361 0.3729 0.2367 𝑆𝑆𝐼𝑀−1 [0,1] 0.2206 0.2425 0.2292 0.1919 Δ𝐸𝑎𝑏∗ [0,1] 0.2391 0.2291 0.2194 0.1514 NIQE [0,∞) 3.1521 2.8290 4.7216 2.8626 Subjective IQA
Metric Range↓ CLAHE DCP-R PDR-Net Pix2pix MOS [0,1] 0.4461 0.4877 0.8447 0.2220
oftheleaves,butalsoproducedcolorartefactsinthehazeareasofthe image.
Section2.1showsthatthebasichazyimageformationmodelitself isbuiltontheassumptionthatscatteringoccurshomogeneously.Thus, imageswithnon-homogeneoushazeposeachallengetocommonde- hazingmethods.SeeNH-HazecolumninFig.5,whereCLAHE,DCP-R, andPix2pixwereallunabletofullyremovethehaze,andthedehazed imagesstillcontainhazeinareaswherethehazeisthicker.Meanwhile, theresultofPDR-Netwassuccessfullyvoidofhazeeffects,albeitwith lowcolorsaturation.DetailedobservationsoftheresultsonNH-Haze canbeseeninFig.7.
Objectiveevaluationofthedehazingresultsforthefourdehazing methodsispresentedinTable1.Notethatallobjectivemetricshave beenconvertedandscaledsothatalowervalueindicatesabetterre- construction.Overall,Pix2pixisabletoobtainthebestscoreinRMSE, 𝑆𝑆𝐼𝑀−1,andΔ𝐸𝑎𝑏∗.ThebestNIQEscoreisachievedbyDCP-R,show- ingitssuperiorityintermsofnaturalness.Furthermore,allothermeth- odscomeinsecondindifferentaspects,whereCLAHEhasthesecond best𝑆𝑆𝐼𝑀−1score,DCP-RinRMSE,andPDR-NetinΔ𝐸𝑎𝑏∗.CLAHEis amethodthatmodifiestheimageveryminimally,explainingthegood 𝑆𝑆𝐼𝑀−1score.DCP-RhasagoodRMSE,indicatingasuperiorpixel intensityrecovery.However,DCP-Rdoessufferfromcolordistortion, explainingalowerΔ𝐸𝑎𝑏∗ scorecomparedtoPDR-Net.
Thepsychovisualexperimentofthe50testimageswasconducted withatotalof40humanobservers.TheaverageMOSforeachmethod is shown in thebottom rowof Table 1, indicating a preferencefor dehazingbyPix2pixreflectedin itsloweraverageMOS.CLAHEand DCP-Rcomeinsecondandthird,withPDR-Netastheleastpreferred dehazingmethod.Thepsychovisualexperimentalsoprovidedanoppor- tunityforobserverstoaddopen-endedcomments,whichmayfurther provideinsightsonthepossibleperceptualattributesfortheperception ofdehazedimages.Theywillbevaluableforadeeperanalysisofvisual cuesaffectingtheperceivedimagequalityandpotentialimprovements
Fig. 5.Examplesofdehazingresultsoftestimagesfromvariousdatasets,i.e.,Dense-Haze,NH-Haze,O-Haze,Synthetic,andRESIDEdatasets.Theoriginalhazy imageisshowninthefirstrow,followedbythegroundtruthclearimageinthesecond.ThedehazedimagesobtainedusingCLAHE,DCP-R,PDR-Net,andPix2pix areshowninrow3-6.
Fig. 6.Detailedcomparisonofaresultof(a)CLAHE,(b)DCP-R,(c)PDR-Net,and(d)Pix2pixforanimagewithdensehaze.Seethecorrespondingoriginaland groundtruthimagesinDense-HazecolumninFig.5.
Fig. 7.Detailed comparison of a result of(a)CLAHE,(b)DCP-R,(c)PDR-Net, and(d)Pix2pix for an image with non-homogeneous haze.
Table 2.Insightsfromtheopen-endedcommentsprovidedbyobserversinthe psychovisualexperiment.
No. Comment
1 ImagesdehazedwithCLAHEstillcontainaveryvisiblehaze,indicatinga lesssuccessfuldehazingprocess.
2 ImagesdehazedwithCLAHEarestillmorevisuallypleasingregardlessof thehaze,becausetheydonotcontainnoiseorartifacts.
3 TheleftoverhazeinCLAHEimagesactuallycontributestothenaturalness oftheimage.
4 ImagesdehazedwithDCP-Raregenerallybright,smoothandsharp.DCP-R seemstoperformverywellwhenthehazeintheoriginalimageislight.
5 ImagesdehazedwithDCP-Roftenhavecolordistortionsmakingitless visuallypleasing.Thisisespeciallyvisibleinoutdoorsceneswithhave unnaturalskycolor.
6 ImagesdehazedwithDCP-Rsometimesappeardullanddarkwithsome scenes.
7 SeveralimagesdehazedwithPDR-Nethavespots,artifacts,ornoise.
Althoughsomeproducecolorsthataresimilartorealobjectsinthescene buttheartifactsmakesPDR-Nettheleastvisuallypleasing.
8 PDR-Netresultsseemtohavelostthesenseofdepthofthescene,duetoa veryhighcontrast.
9 PDR-NetandPix2pixbothareabletoeliminatemorehaze,butarevisibly unnatural,makingobserversuncomfortable.
10 ImagesdehazedwithPix2pixtendtohavesoftercolorswhichmakeitmore visuallypleasing.
11 ImagesdehazedwithPix2pixaremostoftenmorevisuallypleasing comparedtotheothermethods,asidefromsomeunnaturalcolors.
12 Clearobjectsandminimalartifacts/distortionarealargefactorin evaluatingvisuallypleasingimages.ThatiswhyCLAHEandPix2piximages werefrequentlyplacedatthetoprank.
fordehazingmethods.Someinterestingpointsabouttheobservations thatcanbeconcludedfromthesecommentsareshowninTable2.
Asdescribedbefore,theconceptofimagequalityisverycomplex.
Thequantitativemetricsthatarecommonlyusedtodescribeaccuracy ofdehazedimages,arenotalwaysadequatetodenoteitsquality.This isapparentintheobjectiveandsubjectiveresultspresentedinthissec- tion.Fig.8showsthescatterplotofMOSagainstall4objectivemetrics, i.e.RMSE,𝑆𝑆𝐼𝑀−1,Δ𝐸∗𝑎𝑏,andNIQE.Forvisualizationpurposes,each objective metricis scaledtotherange of [0,1]. We thenattemptto infer thecorrelationbetween MOS andeach objective metric, using curvefittingwithapolynomialregressionoforder1.Theplotshows weakcorrelationofMOSwithΔ𝐸∗𝑎𝑏and𝑆𝑆𝐼𝑀−1,whoselineplotsare almosthorizontal,indicatingnorelation.Meanwhile,theMOSscores
Fig. 8.ScatterplotbetweenthesubjectiveMOSandRMSEinred,𝑆𝑆𝐼𝑀−1in blue,Δ𝐸𝑎𝑏∗ inyellow,andNIQEingreen.Thecorrelationlineforeachmetricis obtainedthroughcurvefittingwithpolynomialregressionorder1.
withRMSEandNIQEshow astrongercorrelationthroughlinesthat inclinevisibly.
Tomeasurethe correlationquantitatively, wealso computed the Pearson𝑟correlationcoefficientbetweenMOSandeachobjectivemet- ric [47]. The Pearson𝑟 correlationcoefficient is commonly used to quantitativelyrepresent therelationshipbetweenvariables [47].The Pearson𝑟coefficientofMOSwithRMSE,𝑆𝑆𝐼𝑀−1,Δ𝐸∗𝑎𝑏,andNIQEare showninTable3.Fromtheresults,weareabletoconfirmtheinsights weobtainedfromFig.8.All4objectivemetricsarepositivelycorre- latedwithMOS,todifferentdegreesofstrength.Amongthem,NIQEis theobjectivemetricthathasthestrongestcorrelationwithsubjective MOSwithaPearson𝑟coefficientof0.4486.
4.1. Coloranalysis
Thecolorqualityofthedehazedimagescanbeobjectivelymeasured usingthecolordifferencemetricΔ𝐸𝑎𝑏∗.FromTable1,Pix2pixisclearly
Fig. 9.Detailedareasofthedehazedimagesusing(a)CLAHE,(b)DCP-R,(c)PDR-Net,and(d)Pix2pixintermsofcolorandcolorrecovery,contrast,andsmoothness.
Table 3. Pearson 𝑟 correlation coefficient between MOSand4objectivemetrics.
Metric paired with MOS
RMSE 𝑆𝑆𝐼𝑀−1 Δ𝐸∗𝑎𝑏 NIQE Pearson𝑟 0.2284 0.0914 0.1856 0.4486
aheadwithanaverageΔ𝐸∗𝑎𝑏of0.1514followedbyPDR-Net,DCP-R,and CLAHE,inthatorder.Thiscanalsobeconfirmedbyvisuallyobserving thedehazingresultsshowninFig.9.Thetwodeeplearningmethodsare intheleadinthiscriterion,astheybothlearncomplexmappingsfor pixel-basedregressionsfrominputtooutputimageswithlittleexternal factors.
CLAHEenhancestheimagebyre-distributingtheintensitieswith- out correctingthe colorwhich is apparent Fig. 9a andFig. 10a. In the two first images of Fig. 9d, where Macbeth ColorCheckers are present,Pix2pixshowsitscapabilitytorecoverthecolorsonthecolor checkers.DCP-R andPDR-Netalsodowellrestoring thevividnessof thecolor in the color checkers in the first two images. Forthe se- lectedareasin imagesin columns 3-5of Fig.9b,DCP-R showsthat its recoveryof colors results in higher saturation while maintaining smoothness.However, DCP-R ishighly relianton the airlightof the scene,whichisfrequentlyinaccuratelyestimated.Thiserrormanifests intheunnaturalskycolors,whichwehighlightinthefirsttwoimages ofFig.10bForthesimulateddatasetinvolvingindoorimagesincol- umn3,thesecolordistortionsresultincolorartifactssuchasin the
Fig. 10.Detailed areas of the dehazed images using(a)CLAHE,(b)DCP-R,(c)PDR-Net, and(d)Pix2pix, specifically the artifacts and color distortion.
rightmostimageofFig.10b.Whilethiscolordistortiondoesnotoc- curintheresultofPDR-Net,artifactsareintroduced,suchasshownin Fig.10c.
Finally,thepsychovisualexperimentresultsalsoshowtheobservers’
generalpreferencetowardsthedehazedimagesobtained byPix2pix.
Basedon theMOS in Table 1, theorder of preferenceafterPix2pix isCLAHE,thenDCP-R,andlastlyPDR-Net.Mostobserverstakenote of thecolor recovery asan major factor todetermine visual pleas- ingness.DCP-Respeciallywassingledoutfrequentlyaboutitscolors, due to thecolor distortions that occur. Manyobservers commented particularlyon the sky areas,that thecolorswere “uncomfortable”,
“unnerving”,and“unnatural”.BasedonMOS,wecanseethatCLAHE isthusmorepreferred,althoughitclearlydoesnotrecovercolorcor- rectly.
4.2. Contrastandclarity
Thecontrast andclarityof dehazedimagescan be inferredfrom RMSEor𝑆𝑆𝐼𝑀−1inTable1.Basedonthosetwometrics,onceagain Pix2pixis superiortotheothermetrics.Interms ofimagestructure, CLAHEisabletokeepitintactsinceitonlymodifiesimageintensities withoutchangingthestructure atall.Thisis shownbytheCLAHE’s
𝑆𝑆𝐼𝑀−1scorethatissecondbesttoPix2pix.Lastly,wehavePDR-Net followedbyDCP-R.Therecoveryofstructureanddetailinanimageis difficult,particularlyifpixelintensitiesaresaturatedwithhaze.
However,indensehazeimages,suchasinthefirstcolumnofFig.9, theresultsofDCP-Rshow abilitytorecovercontrastanddetail,par- tiallyduetothehelpoftheguidedfilterintherefinementstep,which guarantees asmoothimage.Inthese conditions,CLAHEisonly able toenhancetheedgesthatwerealreadythere,butisincapableofhan- dlingover-saturatedhazeareas.Meanwhile,dehazingbyPDR-Netand Pix2pixisdeeplearningbased,ensuringahighlyaccuratereconstruc- tionasprojectedbyalowerRMSE.Despitebeingthebest-performing methodintermsofRMSEor𝑆𝑆𝐼𝑀−1,Pix2pixalsostrugglestorecover imagesindensehaze.Additionally,sincedeeplearningmethodsusea globalaccuracyovertheentireimage,localartifactsarestillpresent, e.g.,thefirstandthirdimagesofFig.10.
The resultsof the subjectiveevaluationshow thatobservers also gravitatetowardsPix2pixdehazing.Manyobserverstakenoteofedges, visibility,anddistinctionofobjectswhenevaluatingthedehazedim- ages. CLAHEisoften notedforbeingunabletohandlevarioushazy images, especiallyindense haze.Observershave takennotice about theabilityofDCP-RandPix2pixtorecoverobjectsclearlywithdistinct edges.Next,imagesdehazedwithCLAHEoftenstillcontainhazeob- scuringtheobjectsandedgesofthescene.Althoughedgesandcontrast is importantin ascene,some observersmentioned thatthecontrast andedgesinimagesdehazedbyPDR-Netwereextreme.PDR-Netcre- atedunnecessaryedgesandartifactstahtwerepointedoutexplicitlyas reasonstorejectthedehazingresultsofPDR-Net.
4.3. Naturalness
Naturalnessisadifficultconcepttoquantifybecauseitisalargely subjectiveconcept.This involvesmany immeasurablefactorssuchas preference,perception,andfamiliarity.Thispaperattemptstomeasure naturalnessquantitativelyusingNIQE.BasedontheresultsinTable1, DCP-RobtainedthebestNIQEof2.8290,followedbyPix2pix,CLAHE, andlastlyPDR-Net.Amongthetoptwo,DCP-RandPix2pix,thesub- jectiveevaluationshowsapreferencetowardsPix2pixdehazedimages.
CLAHE,ontheotherhand,mayobtainnaturalresultsbutis unableto completethetaskofdehazing.Finally,PDR-Netperformstheworstin reconstructinganaturaldehazedimage.Avisualobservationofsome oftheresultscanbedonethroughFig.10.
PDR-Netdehazedimagesaregeneratedbyadeep-learningnetwork throughconvolutions,allowingahigh-levelabstractionthatdoesnot necessarily matchwith howthe humanvisual system or perception works.Thisprocessiscomputedbasedonaccuracyanderror,bothof which areobjectivemeasuresthatdonot reflectoraccount forcues thatareimportantforhumanperception.PDR-Netwasdesignedtouse amodifiederrorthatconsidersbothaccuracyandvisualperception,but theresultshereshowthatitstillhasthislimitation.Meanwhile,while Pix2pixisalsoadeepgenerativenetwork,itistrainedusinganadver- sarialdiscriminator,notjustbyasimpleerrororlosscomputation.The discriminatorcomponentofPix2pixactsasifitisperceivingtheimage, tryingtoidentifythegeneratedimagefromthegroundtruth.Thus,the generatortriestogenerateanoutputthatcantrickthediscriminator, resultinginavisuallypleasingandperhapsmorenaturalimage.
ImagesdehazedwithPix2pixaremostpreferredbyobserversbased ontheMOSinTable1.However,althoughthecommentsinTable2 aregenerallypositivetowardsthePix2pixresults,theystillpointout Pix2piximagestobeunnatural.Intermsofnaturalness,weinferthat thecolordistortionscreatedbyDCP-Rskewedthesubjectivescoresin favorofPix2pix,eventhoughDCP-Rhassharperedgesandsmoother regions.SomecommentsalsofavoredCLAHEintermsofnaturalness,al- thoughobserversacknowledgethehazewasnotsuccessfullyremoved.
Infact,manycommentedthatthepresenceofhazeactuallyaddstothe naturalnessoftheimage,asitiscommontoseesuchscenesinthereal world.ManyobserversagreethatthedehazedimagesbyPDR-Netare
theleastvisuallypleasing,consistentwiththeNIQEscores.Manycite theroughedges,artifacts,noise,andlossofdepthasthereasonforthis judgement.
5.Conclusion
Inourexperiments,weconductedimagedehazingonauniformset imagesusingfourmethods,i.e.,CLAHE,DCP-R,PDR-NetandPix2pix.
Thetwo deep learningapproaches,PDR-Net and Pix2pixwereboth trainedusingthesamesetoftrainingimages.Inthiswork,weevaluate dehazingresultsnotonlybyhowsimilartheyaretothegroundtruth, butalsobyimagequalityusingobjectiveandsubjectiveIQA.Theobjec- tivemetricswereselectedtomeasurecolordifference(Δ𝐸∗𝑎𝑏),contrast andimagestructure(RMSE,SSIM),andnaturalness(NIQE).Thesub- jectiveresultismeasuredwiththemeanopinionscores(MOS).Among allfourmethods,Pix2pixissuperiorintermsofcolordifference,con- trastandimagestructure.TheMOSalsoshowsthatPix2pixisthemost preferredbyhumanobservers.AlthoughPix2pixissurpassedbyDCP-R intermsofobjectivenaturalness,DCP-Rdehazingobtainedthesecond toworstMOSduetothecolordistortionthatsometimesoccurs.CLAHE isunabletoremovehazeatallasreflectedinitsbadobjectivescores.
Interestingly,CLAHEobtainsthesecondbestMOSafterPix2pix,indi- catingthatthepresenceofhazeisnotaproblemforhumanobservers.
Finally,PDR-Netdehazingdoesnotexcelatanyoftheobjectivecrite- ria,althoughitis secondplaceforcolorrecovery.PDR-Net dehazing alsofallsinlastplacefornaturalnessandMOS.
Thecontextofthedehazingbecomesveryrelevanttothediscussion.
Incaseswheretheimagesareintendedtobeviewedbyhumans,CLAHE issimpleandeasytoimplement.However,althoughCLAHEisableto enhancethevisibilityofahazyimage,itwouldbemisleadingtoclaim CLAHEasadehazingmethodsincetheresultsareoftenstillhazy.For afullyautomatedmachinebasedapproach,PDR-Netisstraightforward andsucceedstoobtaingoodreconstructionwithminimalhumaninter- vention.Alternatively,DCP-Risabletodehazeanimagemostnaturally, butinordertoavoidthecolordistortions,itisnecessarytoimprove airlightestimation.Itisnecessarytolookintotherelevantqualitycues ofeachdehazingmethod,anddeterminewhichonesaremoreimpor- tantinthecontextof acertaintask.This isparticularlyrelevantfor largercomputervisionapplications, inwhich higher-levelimageun- derstandingisnecessary.Often,theseapplicationsassumeclearimages asinputs,soanyhazyinputsmustbedehazedpriortoprocessing.In thispaper,weprovidedageneralevaluationofimagequality,focusing oncomprehensiveanalysisanddiscussiononthecolor,imagestructure, andnaturalnessofthedehazedimages.
Inclosing,theabsenceofastandardizedhazydatasetisacommon obstacleinhazyimages.Assuch,ourexperimentwasconductedona fairlylimiteddatasetofhazyimages.Thus,thereismuchtobedesired inregardstothegeneralizationabilityofdehazingmethods.Forfuture work,ageneralapproachwouldbeinstrumentalfordehazingandits subsequentapplications.Inlinewiththistarget,theestablishmentofa largestandardizeddatasetwithhazy,clear,anddepthinformationwill beverybeneficialtoscatteringmediaimageunderstandingingeneral, e.g.,forunderwaterenvironments.
Declarations
Authorcontributionstatement
L.Rahadianti, A. Y. Azizah: Conceivedand designedthe experi- ments;Performedtheexperiments;Analyzedandinterpretedthedata;
Wrotethepaper.H.Deborah:Conceivedanddesignedtheexperiments;
Analyzedandinterpretedthedata;Wrotethepaper.
Fundingstatement
This research was funded by Universitas Indonesia through Hi- bahPublikasi TerindeksInternasional(PUTI) Q3grantnumberNKB-
