JournalofInformetrics10(2016)664–666
ContentslistsavailableatScienceDirect
Journal of Informetrics
jou rn al h om ep a ge : w w w . e l s e v i e r . c o m / l o c a t e / j o i
Awelcometomethodologicalpragmatism 1. Introduction
IntheircontributiontothisissueofJournalofInformetrics,‘AfarewelltotheMNCSandlikesize-independentindicators’, GiovanniAbramoandCiriacoAndreaD’Angeloopenupnewandinterestingviaeforthestudyofscientificproductivity.They combinepublicationandcitationfrequencieswithdatarepresentingeconomicresourcesforresearchinaproposedsingular indicator,the‘FractionalScientificStrength(FSS)’.Atthesametime,theycallonallofustofleefromTroyandabandon everytraditionalindicatorthatmeasurestheaveragecitationimpactrelativetothenumberofpublications.
Iwelcomethedevelopmentofindicatorsthatcombineinputandoutputinresearch.However,Idonotagreetothatwe shouldclosethedoorbehindustothetraditionalindicators.IwillexplainmydisagreementbeforeIreturntoaconstructive discussionofthecomplicatedtaskofmeasuringscientificproductivity.
2. Methodologicalpragmatism
Theideathatasingleindicatorcanreplace,orberegardedassupremeto,otherindicatorsisagainstatraditionfor methodologicalpragmatisminscientometrics.Iwouldliketoseepragmatismcontinue.Consequently,Iwelcomethe‘FSS’
indicatorwithoutabandoningthealternatives.Aspragmatists,weusuallyagreethattheuseofproperscientometricindica- torsdependsonthepurposeoftheassessment,theleveloftheiruseandtheavailabledata.Morethanoneindicatororone datasetmaybeneededtogiveamorecomprehensiveandmultidimensionalviewoftheresearchperformance.Theseideas haverecentlybeenexpressedinprinciples2,4,6,8and10intheso-called‘LeidenManifesto’(Hicks,Wouters,Waltman,de Rijcke,&Rafols,2015).
Methodologicalpragmatisminscientometricsisprobably,Iwouldthink,inunderstandingwithamorebasicphilosophical pragmatism:whatweseeormeasure,isnot“reality”.Itisnomorethanwhatourchoseninstrumentscanachieveforus.
Philosophicalandmethodologicalpragmatismseemsonlynaturalinthequitenewresearchtraditionofscientometrics, sinceitmainlyworkswith‘secondhand’,incompleteandcommerciallyrestricteddatathatareproducedfortheprimary purposeofbibliographicinformationretrievalinlibrarysystems.Ifindmethodologicalpragmatismpresentinalmostall pioneeringworksinscientometrics.MyownfirstlessonscamefromNarin’sandCarpenter’s(1975,1985)detailedaccounts ofhowtheyconstructedthescientometricindicatorsfortheNationalScienceFoundation’searliestScienceandEngineering IndicatorReports.Theyleftnodoubtaboutthelimitationsofthedataandhowdifferentjournalsetsandmeasurements couldchangethepicture.
Anybrandingofindicatorsisagainstpragmatism.Nonetheless,brandingoccursfromtimetotimeinthissemi-commercial fieldofscientometrics.Themostrecentexampleisthebrandingof‘SnowballMetrics’byElsevier.Ithasnowbeenofficially adoptedbyEurocris,theInternationalOrganisationforResearchInformation,inspiteofthefactthat‘SnowballMetrics’
recommendstheuseofanindividuallevelh-index,whichisinconflictwithprinciple7oftheLeidenManifesto.Another well-knownexampleofbrandingisthe‘CrownIndicator’.YearsaftertheCentreforScienceandTechnologyStudiesatLeiden Universityabandoneditthemselves,thenamecontinuestoindicate‘advanced’and‘correct’useofscientometricsamong researchadministratorsandpolicymakers.Itsmorerecentreplacement,the‘MNCS’,shouldnowbereplacedagainbythe
‘FSS’,argueAbramoandD’Angelo.
Theirargumentisworthlisteningtobecauseittakesplaceinanopendiscussioninascientificjournalandisfarfrom beingcommerciallymotivated.Indeed,boththeirdiscussionandthe‘FSS’itselfarecertainlywelcomefromapragmatist pointofview.WhatIfindtobeagainstpragmatism,however,istheirviewthatthereissuchathingas‘truemeasurement ofresearchefficiency’.Moreover,Icannotseewhyotherindicatorsshouldbeabandoned.
http://dx.doi.org/10.1016/j.joi.2016.04.005 1751-1577/©2016ElsevierLtd.Allrightsreserved.
Awelcometomethodologicalpragmatism/JournalofInformetrics10(2016)664–666 665
3. Achallengetoproductiontheory
AbramoandD’Angeloargueconvincinglythatcitationindicatorsbasedontheratiotopublications“violateanaxiom ofproductiontheory”:iftheoutputincreasesunderequalinputs,performancecannotbeconsideredtodiminish.More specifically,itisa‘paradox’,asAbramoandD’Angelo(2014)sayinanearlierpublication,that‘anorganizationmayworsen itsMNCSrankingifitproducesanadditionalarticlewhosenormalizedimpactisbelowthepreviousMNCSvalue’.This argumentisnicelyillustratedandseemslogicalinbothversions.TheMNCSandsimilarindicatorsofcitationimpactcertainly failtorepresentameasurementof‘productivity’.Still,onecouldquestionwhethertheywereevermeanttodoso?
Turningaroundtheirargumentforamoment,IwilladdanotherquotefromSenecatotheoneusedbyAbramoand D’Angelointheirpaper:‘Nonrefertquammultos[libros],sedquambonoshabeas’,or‘Itdoesnotmatterhowmanybooksyou have,buthowgoodtheyare’(EpistulaemoralesadLucilium,45,I).Inotherwords:theMNCSandsimilarindicatorsofcitation impactcanbeseenaschallengingeconomicproductiontheorybyvaluing‘quality’oroutcomehigherthanproductivity.For manyyears,therehasbeenaworldwideconcernaboutover-productionofpublicationsinthesciences.Theargumentisthat toomanypublicationsgouncited–andseemtobeunnecessary–intheoverwhelmingboomofscientificpublishing.Inthis perspective,itcanbeseenasaqualityoftheMNCSthatthevaluedecreasesifanotherpublicationwithacitationratebelow averageisadded.Itfollowsfromtheskeweddistributionofcitationsthattobecitedbelowaverageisthesameasneveror seldombeingcited.Mypragmatistconclusionisthatwemayneedsize-independentindicatorsofimpactinadditiontothe moreadvancedmeasurementofproductivitythatAbramoandD’Angelopropose.
4. Advancingthemeasurementofresearchproductivity
Theproposed‘FSS’indicatorisnotamodificationofthe‘MNCS’indicatorinthesamesenseasthe‘MNCS’indicator wasamodificationofthe‘CrownIndicator’.Thelattermodificationwasarepresentationofthesamedatainamodified mathematicalformula.The‘FSS’,however,combinesbibliometricdatarepresentingpublicationsandtheircitationswith economicdatarepresentingtheresourcesforresearchatthesameunit.AbramoandD’Angelo(2014)usethenumberof researchers(fulltimeequivalents)andtheirdifferentiatedsalariestorepresenttheeconomicresourcesmadeavailablefor researchatunitsofassessmentsatdifferentlevelsintheresearchsystem,fromtheindividualleveluptotheinstitutional.
ItisonlywiththeextensiontoeconomicdatathatAbramoandD’Angeloareabletobreakoutofthetraditionofmeasuring citationimpactontheratioofpublications.Thetraditionistostaywithinthescientometricdataandcomparecitationswith publications,toputitsimple.Bystayinginside,thereisatleastalittlebitofhopeforconsistentandcomparableindicators acrossfields,institutionsandcountriesaslongastheindicatorsarenormalizedandthejournalsetsarecautiouslymonitored.
Istherehopeforthesamedegreesofconsistencyandcomparabilityaseconomicdataareaddedtothemeasurement?
AbramoandD’Angelo(2014)demonstratethattheproductivityofItalianuniversitiescanbecomparedbyusingthe‘FSS’, butitremainscomplicatedanddependsoneconomicdataandfieldclassificationsofresearchersthatareonlyavailable withinthecountry.Thisavailabilityispartlyduetoasituationwithanationallyunifiedsystemofpubliclyfundeduniversities, partlytoanationalperformance-basedinstitutionalfundingmodelthatcreatescomparableandquality-assureddataina currentresearchinformationsystem(CRIS).TheScandinaviancountriesareinthesamesituation,whichseemspromising withregardtocombiningscientometricdatawithotherdatathatrepresenttheresources,activitiesandoutcomesofresearch (Sivertsen,2010,2016).Still,severalchallengesremaininthisnewlyopenedfieldofusingnationalCRISdataincombination withscientometrics(Piro,Aksnes,&Rørstad,2013).
Ontheinternationallevel,EuropeaninitiativessuchastheU-MultirankandEurostatarehavinglargerproblemswith consistencyandcomparabilityofeconomicdataandindicators.DespitefiftyyearsofdevelopingandapplyingtheFrascati Manual,theOECDisstillfarfromrepresentingthenationalR&Deconomiesinacomparableway(Wendt,Aksnes,Sivertsen,
&Karlsson,2011).Inthissituation,myconclusionswillbepragmatic.
5. Conclusions
AlthoughIdisagreewiththeircallforabandoningthealternatives,Iwelcometheintroductionofthe‘FSS’indicatorby AbramoandD’Angeloasanimportantsteptowardsadvancingthemeasurementofresearchproductivity.Ihopetheindicator willbecomeastrongdriverforeconomiststoinfluencetheircountriesinproducingmoreconsistentandcomparableR&D resourcedataattheinstitutional,nationalandinternationallevel.
References
Abramo,G.,&D’Angelo,C.A.(2014).Howdoyoudefineandmeasureresearchproductivity?Scientometrics,101,1129–1144.
Carpenter,M.P.(1985).Updatingandmaintainingthirteenbibliometricdataseriesthrough1982.CherryHill,N.J:ComputerHorizons,Inc.
Hicks,D.,Wouters,P.,Waltman,L.,deRijcke,S.,&Rafols,I.(2015).Bibliometrics:theLeidenManifestoforresearchmetrics.Nature,520(7548),429–431.
Narin,F.,&Carpenter,M.P.(1975).Nationalpublicationandcitationcomparisons.JournaloftheAmericanSocietyforInformationScience,26(2),80–93.
Piro,F.,Aksnes,D.,&Rørstad,K.(2013).Amacroanalysisofproductivitydifferencesacrossfields:challengesinthemeasurementofscientificpublishing.
JournaloftheAmericanSocietyforInformationScienceandTechnology,64(2),307–320.
Seneca,L.A.(65AD)EpistulaemoralesadLucilium.Rome.
Sivertsen,G.(2010).Aperformanceindicatorbasedoncompletedataforthescientificpublicationoutputatresearchinstitutions.ISSINewsletter,6(1), 22–28.
666 Awelcometomethodologicalpragmatism/JournalofInformetrics10(2016)664–666
Sivertsen,G.(2016).DataintegrationinScandinavia.Scientometrics,106(2),849–855.
Wendt,K.,Aksnes,D.W.,Sivertsen,G.,&Karlsson,S.(2011).Challengesincross-nationalcomparisonsofR&Dexpenditureandpublicationoutput.InE.
Archambault,Y.Gingras,&V.Lariviére(Eds.),ProceedingsofSTI2012Montréal(vol.II)(pp.826–834).Montréal:ScienceMetricsandtheObservatoire dessciencesettechnologies.
GunnarSivertsen NordicInstituteforStudiesinInnovation,ResearchandEducation(NIFU),P.O.Box2815Tøyen,N-0608Oslo,
Norway E-mailaddress:gunnar.sivertsen@nifu.no
9December2015 11December2015 Availableonline4May2016
