Modeling Effective Cybersecurity Training Frameworks: a Delphi Method-based Study

ContentslistsavailableatScienceDirect

Computers & Security

journalhomepage:www.elsevier.com/locate/cose

TC 11 Briefing Papers

Modeling effective cybersecurity training frameworks: A delphi method-based study

Nabin Chowdhury

, Sokratis Katsikas , Vasileios Gkioulos

NTNU, Teknologivegen 22, 2815 Gjøvik, Norway

a rt i c l e i nf o

Article history:

Received 3 September 2021 Revised 29 October 2021 Accepted 15 November 2021 Available online 18 November 2021 Keywords:

Cyber-security Training framework Delphi method Learning theory Personalizedp learning

a b s t r a c t

Today,cybersecuritytrainingiscommonplaceinbothlargecompaniesand Small&MediumEnterprise (SME).Nonetheless,theeffectivenessofmanyofthecurrenttrainingofferingsisputintoquestionbyre- portsofincreasingsuccessfulcyber-attacks.WhileanumberofmodelsfordevelopingCybersecurity(CS) trainingframeworksforindustrialpersonnelorgeneralaudiencehavebeenproposed,thesemodelsof- tenlackconsiderationforhumansaspectsoflearning(cognitiveabilities,learningstyles,meta-cognition amongothers) duringdevelopment.Additionally,the successofaCStraining programhighlydepends onitsabilityto engageparticipants. TodevelopaCStraining frameworkthatisabletomotivatepar- ticipants,wemustconsiderindividual-specific factorsthatcanaffecttheresultoftraining,besideses- tablishing optimal trainingdelivery methods and assessment. Forthis, inthiswork we proposeaCS trainingframework basedonarevisedversionoftheADDIE modeland morerecentresearchperson- alisedlearningtheory.TheDelphimethodwasusedtobothdevelopandvalidateourdecisionsduring thedevelopmentofthetrainingframeworkmodel.TheresultsofthedecisionoftheDelphimethodhave laterbeencomparedtorecommendationsintheliteraturetocreatethefinalisedframework.Thiswork presentstwomajordistinctionsfromotherCStraining frameworksmodelsdescribedintheliterature.

First,thedevelopedmodelisstronglybasedinlearningtheoryfoundationsandtakesintoconsideration differencesinlearningstyles,cognitiveabilitiesandmetacognitionofindividuals,tooffertailoredsolu- tionsoptimizedforeachgroupofemployeesandsingleindividual.Second,theuseoftheDelphimethod andthe involvementofexpertsstakeholdersfromvarioussidesofacademiaandindustrygaveawide insightintocurrentneedsandrecommendationsforCStraining,aswellasformalvalidationforthefinal development.

© 2021TheAuthors.PublishedbyElsevierLtd.

ThisisanopenaccessarticleundertheCCBYlicense(http://creativecommons.org/licenses/by/4.0/)

1. Introduction

The current landscape of threats in the cyberspace often in- volves attack vectors that exploit lack of readiness and human preparedness to accessconfidential data orcompromise systems.

In the private sector, this has been causing significant economic lossesforcompaniesbyjeopardizingsystems’regularfunctionality orduetoattackersaskingforsubstantialransoms.

Recentexamplesofthesetypeofoccurrencesarethe2020ran- somwareattacks againsttheTollGroup,whichhappenedsequen- tially ata distanceofthree months (Osbourne, 2020). The attack forcedthe logisticscompanyto switchmanyofthe servicesded- icated totheir clients offline. Interestingly,the malware used for

∗Corresponding author.

E-mail addresses: [email protected] (N. Chowdhury), [email protected] (S. Katsikas), [email protected] (V. Gkioulos).

the attack knownas theMailTo ransomware is notoriousfor not beingastealthymalware,whichsuggeststhatimprovedpersonnel securityawarenessmayhaveallowedtodetectandpreventtheat- tacks.Anothersimilarincidentinvolvedthehealthinsurancecom- pany Magellan, which fell victim to a ransomware attack in the first quarterof 2020(Davids, 2020). Inthis occurrence, attackers usedphishingmailstoimpersonateoneclientofthecompany,and afterobtaining accessto thesystem,stealconfidentialdataabout otherclients.

Asitcanbenotedfromtheseexamples,humannegligenceand unpreparedness are often the contributingfactors to the success ofcyberattacks.Humansareindeedoftendescribedastheweak- estlink inCybersecurity(CS) assurance.This characterizationhas been motivated by researchers because of human tendencies to- wardsnegligence,eitherbecauseoflackofknowledge(Goh,2021), butalso because ofpsychological attributes andcognitive biases,

https://doi.org/10.1016/j.cose.2021.102551

0167-4048/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ )

whichcanaffectanindividuals’judgementwhenitcomestotrust management(Hai-Jew,2019;Wiederhold,2014).

To counter this issue, research has been focused to develop methodsthatwouldallowtomovefromthecurrentperspectiveof humansasaproblemtobecomingaresourceandagentagainstcy- berthreats.ZimmermannandRenaud(2019)suggeststhatforsuch a transitiontohappen,thereneeds tobea shiftfromthecurrent mindset of control & prevention asthe basis for currentpolicies andtraininginattitudesthatencourageactivelearning,communi- cationandcollaboration.

Currently,oneofthemostprevalentmethodsforimprovinghu- man capabilities forCS in the private sector consistsof institut- ing internal CS awareness andtraining programs to educate and train staff against common attack vectors and prepare them in emergency scenarios. Although such forms of training have be- come commonplace in the private sector for many years, much criticismhasbeenraisedontheireffectiveness.Thecontinuedin- creaseincyber-attacksagainstcompaniesinrecentyearshasbeen one ofthemotivesforthequestioningoftheeffectivenessofcur- renttrainingofferings.

It has been reported that over the last 10 years, the num- berofsuccessfulmalwareattackshassteadilyincreasedyear-over- year(PurpleSec, 2021), with damage caused by ransomware ex- ceeding $7.5billion in2019 alone(atLast,2021). Additional crit- icisms raised against modern CS training offerings include their poor ability in changing users’ risk perception (Malmedal and Røislien,2016),lackofagreementonmosteffectivetrainingdeliv- ery methods, aswell asestablished evaluationcriteriaand tech- niques for these programs, which also contributes to their lack- lusterperformance(ChowdhuryandGkioulos,2021a).Finally,many CS trainingprograms stillfail toengageparticipantsandmotivate them towards learning. Lack of user engagement has been indi- cated in the literature as one of the main detractors to the ef- fectiveness of CS training programs (Bada etal., 2019). This lack of engagement has been justified by CS training and awareness campaignsoftenbeingperceived astediousactivities (Badaetal., 2019), or dueto not considering preferences in content delivery, participants learning styles and other individual-specific factors that caninfluencetrainingeffectiveness(Pashleretal.,2008;Pat- tinsonetal.,2019).

Thesefactorsmotivatedresearcherstofocusondeliveringtrain- ing that is more engaging, often by adopting more captivating training delivery methods such as game-based and simulation- basedtraining(Beuranetal.,2017;Hendrixetal.,2016;Nagarajan etal.,2012).

Inrecentyears,significantprogresshasalsobeenmadeinthe area of Personalized Learning Theory (PLT), which refers to pro- viding training that is tailored to a specific individual, based on their learningobjectives, learner’s profile andoverall preferences inlearning(Morin,2020).Additionally,researchershavealsobeen adapting established learningtaxonomies to CS education to im- prove the overall learning and evaluation process (Harris et al., 2015). Unfortunately, the same considerations have yet to be adaptedforCStrainingorincorporatedincurrentCStrainingpro- grams. Additionally, differences in prioritization of objectives by differentgroupsofstakeholdersinvolvedintheCS trainingdevel- opmentandemploymentarealsooftencauseofadecreaseinthe performanceoftheseprograms.

Totackleall theseissues,inthiswork, weproposea novelCS trainingframeworkmodelforCSpersonnelthattakesintoconsid- eration the aforementioned individual-specificfactors oflearning.

TheCStrainingframeworkhasbeendevelopedbasedonaninitial consultationbetweena panel ofexpertsinCS ondifferent topics relatingtoCS trainingframework development.ByusingtheDel- phimethod,wewereabletoconsultwithapanelofexpertsinCS from differentfields ofacademia andindustry and allow fordy-

namic andactive discussion betweenparticipants. After reaching an agreementon all raised topics,theresults ofthe Delphi were utilizedtodevelopaCStrainingframeworkmodelthatistheoret- icallyfounded.

The main novelty of this work comes from developing a CS trainingframework relianton thecurrentprogress inresearch in learningtheoryandits applicationindigitaleducationandtrain- ing.Specifically,veryrecenteducationalconcepts proposedinPLT wereincorporatedinthedesign ofthe model.Anothernoveltyof thiswork is its useof the Delphi process asan initial validation methodforthelaterdevelopedmodel.Byinvolvingaveryhetero- geneous panelofexperts whenitcomes torolescovered inboth industryandacademia,wewereabletoconsiderdifferencesinob- jectives and prioritization betweendifferent roles andagents in- volvedinCSandreachamajorityagreementoncriticalaspectsof CStraining.

The remainder of the work is organized as it follows. In Section2 wediscussrelatedworkfound intheliteraturefocused onproposalsforCSframeworks.InSection3wedescribetheover- allmethodologyusedinthisstudy,byillustratingindetailallthe stepsoftheDelphiprocessandhowtheresultsoftheDelphiwere utilizedto develop the final CS trainingframework. In Section 4, we describe thefindings of thediscussion betweenthe panel on selected topicsrelatingto CS training. These includetraining de- velopment methodology, training components and desirable at- tributes,trainingevaluation andthehuman factorinCS training, whichisfurtherdefinedintherelatedsection.Additionallytothe findings fromthe discussionsconductedduringtheDelphi, these sectionsalsodescribeandanalysethemainrecommendationsfor thesecomponents of CS trainingfound in the literature, andthe subsequentcomparisonbetweentheDelphifindingsandthelitera- turerecommendations.Allthesefindingsarethensummarizedand elaborated in Section 4.7 to present the final CS training frame- work.Finally,themainconclusions,plannedfutureworkandpos- sibledirectionforlaterresearcharepresentedinSection5.

2. Relatedwork

Whilea numberofCS trainingframework hasbeensuggested in the literature, to the best of our knowledge there has yet to beaframeworkthathasbeendevelopedtakingintoconsideration theindividual-specificfactorsdiscussedinSection 1.Nonetheless, aselectionofCS trainingframeworksofinteresthavebeenfound intheliterature,whichhavealsobeenconsultedforthedevelop- mentoftheCStrainingframeworkproposedinthisworkandare presentedinthissection.

In our previous work in Chowdhury and Gkioulos (2021a), we conductedan extensive literature review ofCS training offer- ings,whichincludedCStrainingframeworks,platforms,test-beds, amongothertypesofofferings, withafocus ontrainingofferings forCriticalInfrastructure(CI)protection.Atotalof68articleswere includedin thereview. Methodsof trainingdelivery,target audi- ences,analysisofevaluationcriteria,togetherwithgeneraldiscus- sionregardingadvantagesanddisadvantagesofgroupsofsolutions were presented in the work. Based on the findings of the liter- ature review,delivery methods that offered hands-on experience were often preferred over traditional methods. Simulation-based andgame-basedtraininginparticularwereshowntobeapopular CStrainingtool,bothforCI-sectorspecifictrainingandgeneralCS concepttraining.Inthework,itwasconcludedthatagreementon whichsolutionshouldbeconsideredoptimalhasyettobereached andthatfurtherresearch isneededtoestablishhowtooptimally integratedesiredattributesfoundacrossdifferentproposals.

In Patriciu and Furtuna (2009), the authors propose a step- baseddesignandgeneralguidelinestobe followedforthedevel-

Fig. 1. Design steps for developing CS exercises, proposed by Patriciu and Fur- tuna (2009) .

opment ofCSexercises. The modelproposed bytheauthors con- sistsof7steps,asshowninFig.1.

Asidefromdesignconsiderations,theauthorsalsosuggestpos- sible metrics for evaluating the effectiveness ofexercise, by sug- gestingexemplaryPerformanceIndicators(PIs).Themodeloffersa generalinitialapproach fordevelopingCS exercises,ina method- ologicalandstructuredmanner.Nonetheless,thisdesignlackscon- siderationsforthehumanfactorintraining.

Beuranetal.(2017)developedCyTrONE,whichisdescribedas an integrated CS training framework, designed andimplemented to address shortcomings of CS training which requires manual setup and configuration of training environment, by automat- ing the training content generation and environment setup task (Beuranetal.,2018).CyTrONEusesinputfromatrainingcoordina- tortogeneratethetrainingcontentforaparticulartrainingsession anduploads it toan e-learningsystem. CyTrONE alsocreatesthe cyber range training environment corresponding to that training content.CyTrONEhasbeendevelopedbycombiningthefollowing components: (1) a User Interface (UI), a training database, (2) a trainingdescriptiongeneratorthatallowstotaketheorganizerin- put to selectthe appropriate sources from the trainingdatabase, (3) a content description processing module which converts the trainingcontent description that isgenerated bythe trainingde- scription generation module to a format that is suitable for e- learning systemsandfinally (4) a cyber-range instantiationmod- ule,namedCyRIS(CyberRangeInstantiationSystem)(Phametal., 2016).Thislastcomponentwasdevelopedtoautomaticallycreate a cyberrange based on its specification. This includes: (i) train- ingenvironmentsetupfunctions;and(ii)securitycontentgenera- tion functions. CyTrONE’s highlevelofpersonalization, automatic generationoftrainingscenarioandcyberrangeinstantiationmake it an overall greattool to integrate inanyCS trainingframework that requires highscalability and is to be used by a large num- berofparticipants.Sincedevelopmentsofthetrainingcontentfor CyTrONE are still at their initial stages,further work andexperi- mentationisnecessarytoverifywhetherthetoolcanbeofusein CStraininginenterprisesettings.

Brilingait˙eetal.(2020)presentaframeworktoaidinthedevel- opment andassessmentof cybersecuritycompetencesduringhy- brid CS exercises, which involve both CS skilled and non-skilled workers. The framework involves 4 phases: pre-exercise assess-

ment,pre-exercisetraining, liveexercise andpost-exerciseassess- ment,asshowninFig.2.

TheframeworkproposedbytheauthorssupplementstypicalCS exerciselifecyclestoenablecompetenceassessmentatanindivid- ual level to reach learningobjectives. While the authors dogive consideration to motivation asa criticalaspect of the success of a CS exercise in achieving its initial goal, other factors, such as meta-cognitionandlearningstyles,arenotconsidered.Also,when it comes to assessment, the authors analyze possible evaluation methodsonlyatahighabstractionlevel,withoutindicatinginde- tailtheadvantagesofcertainmethodsoverothers.

Zhangetal.(2021)developedatheoreticalframeworkforcon- ductingacost-benefitanalysisofCSawarenesstrainingprograms.

TheauthorsdifferentiatethreetypesofCSawarenesstrainingpro- grams (constant, complementary andcompensatory) in terms of their costs and four types (ineffective, consistent, increasing and diminishing) with respect to their benefits. The authors also in- vestigatethe impactof CS awarenesstrainingprograms withdif- ferentcosts andbenefitson acompany’s optimaldegreeofsecu- rity,andfoundthatifacompanyistoimplementsuchaprogram witha projectedconsistent costanda constant benefit,theopti- maldegreeofsecuritywillremainthesame,whileaprogramwith acompensatory cost willhelp acompany move toa higherlevel ofsecurity since thecompany can take advantageof such apro- gramtoincuralowercost atahighersecuritylevel,whereasthe oppositeistrueforacomplementaryprogram.Theauthor’sanaly- sisprovidesaninterestingtool tobetterunderstandwhetheraCS trainingprogramisvaluableforitsoverallcost,whichisoftenone ofthekeycriteriausedbycompaniestoselectwhichtrainingpro- gramtoimplement.Thatbeingsaid,theanalysisprovideddoesnot considertheimpactofCSawarenesstrainingprograms onacom- pany’s total cybersecurity cost because companies with different sizesmayvarysignificantlyintheiranticipationandfailurecosts.

Rajamäkietal.(2018)proposeaholisticcyberresilienceandse- curityframeworkfordevelopinganddeliveringamultilateraledu- cationalandtrainingschemebasedonaproactiveapproachtocy- bersecurity.Theframeworkproposedbytheauthorsisbuiltonthe principlethat“educationandtrainingmustbeinteractive,guided, meaningfulanddirectlyrelevanttotheuser’ operationalenviron- ment” (Rajamäkiet al., 2018). The framework addresses capacity mapping,cyber resiliencelevel measuring, utilizingavailable and mappingmissingresources,adaptivelearningtechnologiesanddy- namiccontentdelivery.

In Aldawood and Skinner (2018) and later in Aldawood and Skinner (2019a) the authors discussCS trainingsolutions foras- sessingandraisingawarenessofsocialengineeringthreats.Inthe study, a variety of methods for training are identified, includ- ing serious games, gamification, virtual labs, tournaments, simu- lations, andthe use ofother modern applications. Similarly, cur- rent awareness programs that educate against social engineering threatsincludingvideostreaming,compliance,theme-basedtrain- ings,awarenesscampaigns,andconferencesarealsoincluded.Se- rious games and simulations are noted in the work as some of the mosteffective and latest solutions against social engineering threats, which confirms that their applicability and effectiveness extendoutsideofregularCStraining.Bothtechniquesusereallife experiencesofsocialengineeringthreatscenariosinasingleloca- tion or fora whole department in which the participants get to knowdifferent situationsthey may faceasa threat andthe best methodstotacklethem.

3. Methodology

TodevelopatheoreticallyfoundedmodelforCStrainingframe- works,we startedby conductinga literature analysisof theories oflearningandtrainingapplicabletoCSanddigitalenvironments.

Fig. 2. Phases for CS training based on the CS framework proposed by Brilingait ˙e et al. (2020) .

Thepurposeofthisanalysiswastoensurethatthelaterdeveloped modelfortrainingconsidershumanattributesthat affecttheout- comeoftrainingprograms.Theseattributesincludecognitiveabili- ties,learningstylesandvariousformsofbiases,amongothers.This information was utilized together with the data previously col- lectedinChowdhuryandGkioulos(2021b),ChowdhuryandGkiou- los (2021a) andChowdhury et al.(2021) focused on CS training offeringsintheliteratureandintheindustrytodeveloptheques- tionnairesutilizedduringtheDelphimethod.Themethodwasuti- lizedbothforthedevelopmentandvalidationoftheproposed CS trainingframework.TheDelphiconsistedofsendingoutelectron- ically adescription ofthecentral problemandprovidingthecol- lectedbackgroundknowledgetoapanelofexpertsandstakehold- ersinCStrainingintheindustry.ThedecisionofusingtheDelphi methodasavalidationtechniqueforourmodelcamefromitsabil- itytocollectequallyweightedfeedbackfrommultipleparticipants andallowforopendebate,withoutrequiringpracticalexperimen- tation. Thepanelwasselectedbasedonastakeholderanalysisfor CS trainingprograms,withthegoaloffindingasufficientamount ofindividualsforeachofthefollowingcategories:

• CS trainers and educators (both individuals and entities that provideCStrainingservices);

• Traineesintheindustry(thiscouldbebothlimitedtoCS roles orextendedtoallpersonnel);

• Researchersworkingonrelatedtopics;

• Personnelinchargeofestablishing,maintainingandsupporting CStraining(managerialpersonnel,humanresources,etc).

Invitationsforparticipatingintheprocessweresentto18indi- viduals basedontheircompetences,knowledge,previousandcur- rentoccupation.Thefinalselectioncameto10individualsafterac- ceptance,whichisabovethesuggestedminimumrequirementof8 participantsforDelphi(HallowellandGambatese,2010).Thepanel wascomposedofsenior professorsinthefield ofinformationse- curity aswell asresearchersinthesamearea,experiencedCS in- structors with industry work experience and other CI personnel thatcoveredorganizationalCSrolesandwereinchargeofmanag- ing theirrespectivecompanies’CS trainingprograms.Theremain- deroftheDelphimethodconsistedoftheactivitieslistedbelow:

1. Establishaproblemstatement-Theproblemstatementrepre- sentsthegeneralquestionthatwillbecentraltothetopicsdis- cussedduringtheprocess.In ourcase,theproblemstatement is the following: How to develop a Cybersecurity (CS) Training Frameworkthattakesintoconsiderationthehumanfactor?

2. Appointingfacilitator -The mainauthorofthis workwasap- pointedasthefacilitator ofthe Delphimethod. Thefacilitator hadtheresponsibilityofrecruitingthepanelofexperts,devel- opingandsending outallquestionnairesforeachroundofthe Delphimethod.

3. FirstroundofDelphi -Thefirst roundoftheDelphi wascon- ducteddigitallyandconsistedofparticipantsanswering anin- teractivequestionnaire. The questionnairehasbeen developed byusingMentimeterandiscomposedof3maintopics:(i)CS trainingdevelopmentmethodology,(ii)CStrainingcomponents andattributes,and(iii)thehumanfactorinCStraining.Tofa- cilitatedialogue,eachsectionbegunwithmultipleoptionques- tionsandrankingquestions.Attheendofeachsection,weal- lowedparticipants to open discussion andsuggestions to the selectedtopics,throughdigitalopenpanels.

4. Following round of Delphi - After completion of the first roundofDelphi,agreementregardingCStrainingdevelopment methodology and training attributes was reached by partici- pants,while suggested approachesto consider forthe human factorintrainingwere moreheterogeneous.Forthisreason, a secondroundofDelphiwasconductedtodiscussproposedap- proachesandcometoamajoritydecision.

5. Conclusion of the Delphi method - Once agreement on each topicproposedandraisedbythepanelwasreached,theDelphi wasdetermined to beconcluded.The resultswere distributed toallparticipantsandusedtodevelopthefinalmodel.

Initialinput forthetopicsdiscussed duringthetworounds of Delphi came from our previous work in Chowdhury and Gkiou- los(2021b)andChowdhury andGkioulos(2021a), additionallyto further literature analysis, particularlywhen it comes tolearning theoriesthataccount forthehumanfactor.Moredetailonthisis foundinSection4.6.Afterobtaining andtextualizing thefinalre- sults of the Delphi into a final report, we further compared the finalsuggestionsofthepanelwiththerecommendationscollected fromtheliterature.Theresultofthiscomparisonwasthenutilized todeveloptheproposedCStrainingframework.

4. Results

AsbrieflymentionedinSection3,duringtheDelphithefollow- ing aspects ofdesigning anddeveloping aCS trainingframework werediscussed:

Eachoftheaspects showninTable1wasseparately discussed duringthetwo RoundsofDelphi, andlater comparedto therec- ommendationsfromtheliterature.Bothfindings fromthe discus- sionduringtheDelphiandthecomparativeanalysiswiththeliter- aturecan befound inthefollowingsections.Thesefindings have laterbeenusedtodevelopthemodelproposedinSection4.7. 4.1. CSTrainingdevelopmentmethodology

The initial selection for the development methodology of the proposedCS trainingframeworkcametousingtheADDIE(Analy- sis,Design,Development,Implementation,andEvaluation)model,

Table 1

Key aspects of CS training discussed during the two Delphi rounds . Aspect Description

CS Training development methodology

Preferred methodology for designing each module of the training and the overarching final product CS Training:

desirable attributes

Attributes often recommended in the literature or by the panel of experts to be incorporated during development of the CS training

CS Training delivery methods

Preferred training methods (game-based training, simulation-based training, for example), based on reported preferences of CS training participants and instructors

Training content

Content of training based on key skills and competencies that are required by CS personnel, based on literature reviewing

Training assessment &

evaluation

Methods for evaluating both formatively and summatively both individual training components and the overall training program, additionally to metrics and other methods for evaluating participants’ progress in knowledge and skills acquisition

CS training:

The human factor

Individual-specific factors that may affect the final outcome of the training. Examples of these include preferences in learning styles, engagement and motivation, among others

one of the more renowned models for instructional technology (Molenda,2003).

Whileoriginal iterationsofthemodelpropose astatic,cyclical process forthecompletion ofall phases, startingfromtheanaly- sis phase to theevaluation phase, duringthe panel discussion it wasconcludedthat amoremodern approachbasedonrapidpro- totyping wouldbe preferable.Suchanapproach usescontinualor formative feedbackduringeachphasetoassesseachoutputatits timeofdevelopment,allowingforadditionaldynamicityandinter- activeness(NixonandLee,2001).Twomainadvantageswerenoted whenusingthisapproach:(i)itallowsforeasieridentificationand directrevisionofanysinglecomponentthat mayrequirechanges, based on feedback andevaluation;(ii) it alsodecreases the time neededforoverall evaluationandrevision,duetotargetingissues independently. Aconclusiveevaluationofthefinal productisstill recommended, as the final result of the developed training may differfromtheresultofeachsinglecomponent.

Definitionofalltaskstobecompletedduringeachphaseofthe modelwasthen established.Asthe modeldevelopedismeantto be applicabletodifferentformsofCS training,thetaskswerede- fined to be inclusive and adaptable forany required application.

The initialapproach wasbasedon thesuggestedprinciplesofin- structional design by Gagne et al.(2005) and the revised activi- tiesfortheADDIEmodelsuggestedinAllen(2006).Theseconcepts were thenadapted,withaparticularfocusoncurrentresearchon e-learningapplications(Alajmi,2009).

InAllen(2006)itisalsonotedthatevaluationactivitiesmayin- volvedifferentformsofassessment,whichareoftendistinguished as:(i)formativeevaluation,comprisingprocessandproducteval- uationstoprovidefeedback.Theseareconductedduringtheanal- ysis, design phases, andthe development phase; (ii) summative evaluation, consisting of operational test cases and tryouts con- ductedaftertheconclusion ofthedevelopmentphase;(iii) opera- tionalevaluation,consistingofperiodicinternalandexternaleval- uation ofthetrainingandits componentduringthe implementa- tionphase.

Furtherdiscussionbroughttothefinalagreementontheactiv- itieslistedbelowforeachphase:

• Analysis Phase: Analysetrainingneeds. Establish goals,possi- blepre-requirements,targetaudienceandaudiencepreferences (on trainingdelivery,content, etc.),andresourcerequirements

Fig. 3. Revised ADDIE model, based on rapid prototyping.

(budgetoverhead,instructors,hardwareandsoftwareresources, facilities).

• DesignPhase:Designtrainingsolutionalignedwithgoalsand requirements, based on inputs from analysis phase. This in- cludes selection of trainingdelivery method andpossible de- cisiononwhethernewmaterialmayneed tobe developedto satisfytherequirementsandgoalsoftraining.

• DevelopmentPhase: Developan action plan, neededtraining resourcesanda pilottest.Validatethecomponentsdeveloped.

Validation may be done by internal reviewing, test cases, or both.

• ImplementationPhase:Implementtrainingsolutionbyprepar- inga trainingenvironment andtrainingactivities. Engagepar- ticipantsintraining.

• Evaluation Phase: Evaluate the quality of the training re- sources,userengagementandsatisfaction,andoveralltraining results.Evaluationistobeconductedthroughoutthelifecycle ofthetrainingdevelopmentandalsoregularly afterits instal- lationanduse.

Fig.3showstheagreedfinalversionofthemodel.

4.2. CSTraining:Desirableattributes&components

AsmentionedinSection 4.1,developmentofthevariouscom- ponents ofCS trainingshould bedependent onthe requirements and goals of training, with trainees preferences and overall re- source overheadbeing additionalconsiderations. Thatbeing said, manyrecommendationsarefoundintheliteraturewhenitcomes tokey componentsanddesirableattributestoconsiderwhen de- veloping a CS trainingprogram or tool. For thisreason, the sec- ondpartoftheDelphiprocessfocusedonCStrainingcomponents andkeyattributes,aswellasevaluationcriteriatobeusedforde- terminingtheeffectiveness oftraining.In ChowdhuryandGkiou- los(2021a),ananalysisandsummaryofattributestoconsiderdur- ing the development of training frameworks is presented. These include considerations for proposed training delivery and mate- rial,contentoftraining, aswellasotherfactors.Similarly,Heand Zhang(2019)givesrecommendationsregardingbestpracticesand key attributes to include when developing CS training for enter- prise. These considerations, together with additional suggestions,

Table 2

Key attributes and considerations for the development of CS training activities, according to Chowdhury and Gkioulos (2021a) .

Attributes Description

Suitability Training content should be appropriate to the target audience and specific company in terms of content, skills developed and level of training ( Adams, 2018 ).

Real-life Experience

Training should include hands-on activities developed to emulate or simulate real-life scenarios. Such activities should also focus on developing communication and team skills of participants.

Scalability &

Adaptability

Training should be developed so that modification, upgrade and extension of content should be possible, based on the skills and level of knowledge of the target audience, as well as new information on technologies and vulnerabilities.

Accessibility Training activities should be accessible to all staff that may benefit from such activities. Developing remotely accessible training further benefits this goal.

Frequency of Training and Periodical Updates

Training should be conducted and updated periodically. Progress sessions should be planned to ensure that KSAs of personnel are up-to-par to current standards and recommendations ( He and Zhang, 2019 ).

Cost Efficiency Training activities should take into consideration resource constraints of a company (budget, time and training personnel constraints.)

Consideration for the human factor

CS training should consider participants engagement and motivation ( Gross, 2018; Kostadinov, 2018 ), adapting to different learning styles ( Kostadinov, 2018; Nadkarni, 2012 ) and stimulating metacognition ( Pokorny, 2017 )

havebeensummarizedinthelistofkey attributesforCStraining showninTable2.

Discussionoftheseattributeswiththepanelofexpertsfocused onestablishingtwomainconclusions:(i)validity,overallvalueand prioritization of the mentioned aspects in the development of a CSframework;(ii)additionalkeyattributestobeconsideredwhen developing CS training. While overall agreement on all attributes being of relevance when developing a CS training program was reached by participants, certain attributes were weighted higher thanothers,anddiscrepanciesbetweendifferentstakeholders’pri- oritizationwasnoted.Easeofimplementationandusewasmen- tionedasanadditionaldesirableattribute,andwasrankedhighest amongallattributeswhenitcomestoprioritization.Inparticular, stakeholdersfromtheindustryfocusedontheimportanceoftrain- ing offerings that can easily be implemented, either as comple- mentary toestablished solutions orasindependent ones. Easeof implementationandusewouldbenefittothesatisfactionofother desirable attributes, making scalability and cost efficiency more easilyachievable.

When itcomestosuitabilityoftraining, discussionfocusedon differences in development between generalized CS training and role-specificCS training. GeneralizedCS trainingactivities usually refer totrainingofferingsthatare meant forall personnelofone firm or multiplefirms (Leeet al., 2016). Thesetypes ofactivities are often part of CS awareness campaigns with the objective of providingbasicknowledgeaboutCStopics,informingpersonnelof companiessecuritypoliciesandriskprofiles,andoverallincrement inalertness(Badaetal.,2019;Tirumalaetal.,2019).

The experts from the industry suggestedthat trainingof per- sonnel should consider companies risk profiles andlink them to the trainingneeds ofspecific rolesin thecompany. Twobenefits ofdoingthistypeofanalysisare“improvedtailoringoftrainingto satisfy initial requirementsaswell aspersuading the firm tofur- ther investintraining, by indicatingthe risksthat could incurin itsabsence”.

4.3. CSTrainingdeliverymethods

InreviewingCStrainingofferingsproposed intheliteraturein Chowdhury andGkioulos(2021a),we established5 majorgroups oftrainingdeliverymethods. Table3listexamples oftrainingfor eachofthegroup,togetherwithadvantagesanddisadvantagesfor eachgroupoftraining

Intheliterature,game-basedandsimulation-basedmethodsof trainingare oftenrecommendedforCS training, orthey are sug- gestedasaformofcomplementarytrainingtotraditionaldelivery methods(Abawajy,2014b).

The main advantages that these two methods provide in- clude allowing participants to conduct interactive, hands-on ac- tivities, develop teamskillsincluding communicationandorgani- zational skill,asmentioned in Table 3.Additionally, theseactivi- tieshave been demonstrated to be more engaging andstimulat- ing than traditional training methods. As mentioned in He and Zhang (2019) and Bada et al. (2019), user engagement and mo- tivation are two of the mostsignificant factors inthe success of CSeducationandtraining.Tedious ornon-engagingtrainingsolu- tions oftenfail to change employeessecurity behaviour andatti- tudes (He and Zhang,2019). This preference wasalso confirmed bythepanelofexperts,withanequalmajorityofparticipantsse- lectingthesetwomethodsoverothersuggestions.

When itcomes toselecting betweengame-basedtrainingand simulation-basedtraining,afewdistinctionsinpropertiesandpos- sibleapplications should be made. Game-basededucation hasas itsmain strengththat ofbeinghighlyengaging.Gamificationand gamemechanics havebeenprovedtostimulatecollaborationand competition, self-efficacy andself-assessment, while maintaining.

Thereducedcostsofgamifiedtraininghavealsocontributedtothe increaseinpopularityofgame-basedtraining. Ontheotherhand, simulation-basedtrainingsuffersfromhighinitialoverhead.Never- theless,simulation-basedtrainingistheonlytrainingmethodthat allowsparticipantstoconductexercisesthatareequivalenttopos- siblereal-lifescenarios.

4.4. Trainingcontent

Whilethespecificcontentoftrainingwasnotdiscussedduring theDelphi,inChowdhuryandGkioulos(2021b)we conductedan extensiveliteratureanalysisofcompetenciesandskillstodevelop forCStraining.

Basedonourpreviousfindings,whenitcomestocompetencies andskillsrequiredbyCSpersonnel,4maincategoriescanbeiden- tified: technical skills, non-technical (soft skills), implementation skills, andmanagerial skills. Examplesof the main competencies requiredforeachofthesecategorieshavebeenlistedinTable4.

Additionally,in Chowdhury et al.(2021), we interviewed sev- eralCSworkersinNorwegian CriticalInfrastructurecompanies to investigateonthetrainingofferingscurrentlyavailableattheirre- spectivecompanies and thecontent ofthese offerings. According tothestudy,thefollowingtopicswerethemostcommonfocusof currenttrainingofferings:

• NetworkArchitecture;

• InformationHandling(informationdisclosure,informationshar- ingandreporting);

• Cyberthreats& relevantpotentialattacksandsystemvulnera- bilities;

• Proceduresandpreparednessplansforcyberincidents;

• SecurityManagementSystem(Riskassessment&management, mitigationstrategies,controlstrategies,documentation);

• Human factor aspects (Communication, trust management, teamworkskills,decisionmaking);

• Surveillance;

Table 3

Classification of CS training methods according to Chowdhury and Gkioulos (2021a) , with examples found in the literature and associ- ated advantages and disadvantages.

Delivery

Method Examples Advantages Disadvantages

Conventional Methods

On-site training;

Classroom training and exercises;

Presentations

& Conferences;

On-site training sessions;

Usability; Familiarity of format; Multiple messages can be conveyed at once; Ease of communication between instructor and participants; Real-time resolution of issues;

No guarantee of personnel active participation; Can be perceived as tedious ( Leach, 2003 ); Does not always provide hands-on experience; Provides a static solution for a fluid problem

( Valentine, 2006 ); High cost and resource overhead; Time-consuming;

Online and Software-based

Online courses;

Cloud-based training;

Web-accessible training material and software;

E-mail tests;

Remote and multi-modal accessibility ( Abawajy, 2014a ); Industry-wide standard use; Cost-effective; Hands-on exercises;

(Possible) team skills development;

Users may undermine the value/pay less attention; Not always very scalable and adaptable; High cost and resource overhead, if personalized solution is needed; Does not provide instructor assistance;

Game-based Serious Games for CS Awareness and Training

Team skills development, Engaging to users; Hands-on exercises, Demonstrated effectiveness ( Antonioli et al., 2017;

Beuran et al., 2018 ); Adaptability;

(Possible) Remote Usability; (Possible) High scalability;

Older audiences may not be familiar with mechanics; Time-consuming; May not reflect real-life processes. High initial development cost and resource overhead;

Video-based Educational

videos Accessibility, Usability; Cost-efficient;

Time efficient; Limited content. Lack of interactivity with other trainees or instructors. Lack of hands-on experience. No guarantee of personnel active participation; Requires constant integration and updates for scalability;

Simulation and virtualization- based

Testbeds, Simulation platforms, Simulated Laboratory exercises

Team skill development; Hands-on experience; Replication of real-life incidents; Adaptability; (Possible) Remote Usability; (Possible) High scalability;

Hard to coordinate ( Kumar et al., 2015 );

Requires pre-existing knowledge;

Time-consuming; High initial

development cost and resource overhead;

• Crisiscontingency&management;

• Incidentresponse&management;

• Intrusiondetection;

• Managerialskillstraining;

While this information provides a general overview of what typeofcontentshouldbetrainedduringCStrainingsessions,exact subjects should be determined based on the goals ofboth train- ing participants and the institution offering the training, which should bedeterminedduringtheinitial analysisphaseoftheAD- DIEmodel.

4.5. Trainingassessment&evaluation

In ourprevious workinChowdhury andGkioulos (2021a),we foundalackofagreementinpreferredevaluationmetricsandper- formance indicators tobe used when evaluating CS trainingout- put.Additionally,researchintheareaislimitedandnostudieson thebestusesofdifferentevaluationcriteriawasfound.

Thislimitationinboth literatureandinorganizationswasalso noted bythe NationalInstitute ofScienceandTechnology (NIST).

To addressthe issue,researchersat NISTpresented thefollowing list of suggestions of measures that can be usedat an organiza- tional level to indicate whether any implemented CS training is successful:

• Top-Down Leadership buy-in: leadership (executives, man- agers, supervisors)support is notedas one ofthe key factors to measure effectiveness ofCS training(Adams, 2019). Seeing leadersparticipatingorchampioningtrainingofferingsnotonly

indicatespositiveevaluationofthetraining,butalsomotivates employeestoparticipateandbemoreengaged.

• Workforce Training Measures: Specific measures should be used to determine in a continuous cycle the non-technical, technicaland managerialcapabilities ofpersonnel after train- ing(Adams,2019).

• Risks, Vulnerabilities, POA&M Measures, & Cybersecurity Compliance:risk and vulnerability captured from assessment can determine what security control implementation users need to be trained on. CS compliance can show that a suc- cessful training program has been implemented that enables theorganizationtounderstandandmeetsecurityrequirements (Adams,2019).

Aside fromthesegeneralrecommendations,severalevaluation criteriahavebeenutilizedintheliterature toassesspost-training results.Chowdhury andGkioulos (2021a) provides a summary of metricsandassessmentmethodscommonlyreportedinthelitera- tureforCStraining,whicharereportedinTable5.

When discussingevaluationcriteriawiththe panel, focuswas givenonevaluationoftheeffectivenessoflistedcriteria,discussion onadditionalcriteriaandfinallyon feedbackcollection.Feedback collectionandcomparisonofpreandposttrainingevaluationwere selectedbythepanelasthemosteffectivemethodsofevaluation.

Thetwomethodsprovidesignificantlydifferentoutputs,asthefor- meris aqualitative approachbased ontrainingparticipantinput, while the latter is a quantitative approach that analyzes specific performanceindicators(PI).

Table 4

Mapping of skills and competencies for Critical Infrastructure Protection (CIP), according to Chowdhury and Gkioulos (2021b) . Skills & Competencies Mapping Table

Technical Skills Soft Skills Implementation Skills Management Skills 1. Understanding of digital

security concepts;

2. Understanding of evolving threats;

3. Understanding of attack intelligence;

4. Penetration testing skills;

5. Cryptology knowledge;

6. SW & HW security skills;

7. Network security skills;

8. Computer forensics skills;

9. Programming skills;

10. Data analytics skills;

11. Information security skills;

12. Wireless security skills;

13. Ability in using IDS tools;

1. Information sharing and communications;

2. Public speaking and presentation skills;

3. Situational Awareness;

4. Cognitive and behaviour analysis;

5. Ability to work independently;

6. Trust management;

7 Teamwork;

8. Motivation;

9. Time management;

10. Networking;

11. Confidence;

12. Work habits;

1. Threat and vulnerability assessment &

management;

2. Event and Incident Response;

3. Continuity of Operations;

1. Risk management;

2. Identity and access management;

3. Asset, change and configuration management;

4. System administration;

5. Workforce management;

6. Cyber-security program management;

7. Supply chain and external dependencies management;

8. Evaluation of policies effectiveness;

9. Project planning;

4.5.1. Performanceindicatorsfortrainingevaluation

Performance indicators or key performance indicators (KPIs) are defined as measurable values that demonstrate the effective achievement of certain key objectives. These indicatorsare often utilizedinbusinessesandfirmstoevaluatetheperformanceofin- dividuals,processesandoftheorganizationasawhole.

In the context of CS training, performance indicators are de- fined in the NIST documentation ascomputable performance as- sessment, as derived from a combination of metrics or in other words, compute value based on post-analysis which may uti- lize one or many primitive values to perform the computation (Tang,2017).

ManyKPIscanbefoundintheliteraturerelatingtoCSperfor- mance analysisof specific areas,such asprocess control systems (Tang, 2017), robotic (Zimmerman, 2017), big data (Petrenko and Makoveichuk, 2017) among others. Less research has been con- ducted to establish preferred KPIs for CS training, due to the variance ofobjectives for differenttraining andexercisesas well as lack of agreement in evaluation methods for training. In Chowdhury and Gkioulos (2021a), a list of exemplary KPIs and metrics of evaluation is given, based on an analysis of evalua- tion methods adopted fordifferenttypes ofCS trainingfor Criti- calInfrastructurepersonnel.Additionally,Boerman(2020)provides a classification of KPIsbased on thefive NIST perspectivesof CS (Identify, Protect,Detect, Respond,Recover). In the work, theau- thorsrecommend selectingKPIsbasedon stakeholders’input and preferences.FurthergradingonwhichKPIsmayyieldthemostrel- evantdatatoevaluationthetrainingobjectiveshouldalsobeused toprioritize certainindicatorsandavoidoverencumberingevalu- ation.

In the NIST documentation for developing a CS scorecard (Wagner, 2016), it is recommended to start by selecting one key performance indicator (KPI),basedona specificdesiredoutcome.

In addition to this first KPI, it is suggested to add complemen- tary indicatorsthat mayaidinmeasuring thevariousfactorsthat can influence the outcome of the training. Samuel (2019) sug- geststhat KPIsforCStrainingshould measureone ofthefollow- ing attributes:Accuracy,Timeliness,CompletenessandAuthoriza- tion.A numberofexemplaryKPIsfordifferenttypesofmeasure- ments suchasidentity& accessmanagement,configurationman- agement, security awareness,security incidents, compliance, data

leakprevention,vulnerabilityandpatchingaregiveninthelitera- ture(Samuel,2019)andshouldbeconsideredduringthedevelop- mentofspecificCStrainingexercises.

The aforementioned criteria for KPI selection were also con- firmed by the panel of expertsduring the Delphi process, albeit specificdefinitionandselectionofKPIswasavoidedduetodiscus- sionbeingfocused ongeneralparametersfortrainingandnot on thedevelopmentofaspecificCS trainingoffering.Oneoftheex- pertsdidhoweversuggestutilizationofrole-specificKPIs,meaning KPIsrelatedtoeach roleofaCSresponse team,inbothteamex- ercisesandindividualtraining.

4.5.2. Feedbackcollection

Feedback is often described as an essential tool and perfor- manceindicator for post-training evaluation(Andriotis, 2018;Fa- rooqetal., 2011).Benefits ofcollecting feedbackasanevaluation tool are many, including constant trainingprogram improvement basedonlearners’input,increaseparticipants’motivationandper- formance(DeFranzo,2018).

Feedback is differentiated in formative and summative feed- back. Formative feedback is collected during the training and is used to enhance or modify training components in real-time (UniTo, 2018), while summative feedback provides an evaluation ofhow much a student andthe class haslearned. When tiedto specificlearningobjectives,itcanbeusedascoursefeedback,pro- vidingtheinstructorwithfeedbackabouttheeffectivenessofthe coursedesign.Examplesofsummativefeedbacktechniquesinclude exams,finalprojects,andresearchreports(Miller,2018).

Whengatheringfeedback,UniTo(2018)suggeststhat datacol- lectionshouldremain optional,anonymous, andnot linked toin- dividualevaluation. The mostcommon methodsfor post-training feedbackcollectionincludequestionnairesandsurveys,bothbyus- ing onlinetools andpaper-based datacollection.Interviews with participantsarealsoanotherpossiblefeedbackcollectionmethod, albeig lessused.When collecting feedback, itis criticalto estab- lishwhattypeofinformationneedtobecollected,aswellashow thisinformationcanbeusedtoimprovethecurrenttrainingoffer- ing.AccordingtoAndriotis(2018),thefollowing5elementsshould alwaysbeincludedduringpost-trainingfeedbackcollection:

• Effectivenessoftraining:Effectiveness isacriticalelement to measuretheperformanceofatrainingprogramasitestablishes

learnersperception ofwhetherthe coursehelped themattain theirlearningobjectivesandhowrelevantitwasforthem.

• Comprehension: Comprehension refers tothe effectivenessof the coursedelivery andassuch is focused aboutthewaythe course content was delivered. This element also includes the concisenessandclarityofcontent.

• Attractiveness: Attractiveness of a training program refers mostly to how the material and tools used during training looked and felt to the learners. It is especially relevant for software-basedtraining,suchasgame-based,simulation-based oronlinetraining.

• Engagement: One of the mostcritical aspects in the success ofatrainingprogramdependsonuserengagement.Asoverall training engagementis a multifaceted issue,evaluation of in- dividualtrainingcomponentsshouldbe collectedfrompartici- pantstohighlightanyweakpoints.

• Suggestions: Suggestions for improvement from training par- ticipantsshouldalsobecollected.Andriotis(2018)noticesthat suggestionsare oftenskippedduringfeedbacksurveysandfor thisreasonrecommendsaskingparticipantstoincludeamini- mumrequirednumberofsuggestions.

When itcomes tocomponentsof trainingto evaluatethrough feedback,Sviridenko(2018)recommendsanalyzingcontent,course length,exercises,instructorandtools,platformsandanyothertype of training media & material. There are also certain limitations, however,totheeffectivenessoffeedbackasanevaluationtoolfor training. Firstly, gathering andanalyzing feedback can be a long, complex process, especiallyin thecaseof large numberoftrain- ingparticipantsandifinformationiscollectedinanon-automated manner. Awaytocircumventthisissueisto developorincorpo- rateautomated feedbackcollectiontoolstothetrainingprograms that canallowtogenerateasummative logorreportofthefeed- backobtained.

Input from the panel was collected during the first round of Delphi regarding what information and measures should be col- lectedfromtrainingparticipantfeedback.Suggestionsfromtheex- pertsincludedperceived knowledgeandskills,motivationandin- terest towards the type and content of training, level of under- standing,selfandteamassessmentandrelevancytotheirroleand work. To circumvent the previously mentioned limitationsto the subjective, qualitative assessment provided by feedback, one par- ticipantsuggestedplaybackpossibilityasawaytocompensatethe limitationandallowforaquantitativeassessmentofimprovement.

4.6. Cybersecuritytraining:Thehumanfactor

The final section of the Delphi focused on how individual- specific factors may influence the outcome of CS training. Fac- torssuchascognition&meta-cognition,engagement&motivation, adaptability,humanerrorandlearningstyleshaveallbeencitedby thepanelasbeingofinfluenceintheeffectivenessoftrainingand shouldbeconsideredinthedevelopmentoftraining.

ToconciliatethetechnicalrequirementsofspecializedCStrain- ing and the factors mentioned by the panel as affecting train- ingoutcome,researchonlearningtheory,instructionaldesignand learningtaxonomieswasconductedprior toinitiating theDelphi.

Afterpreliminaryresearch,anapproachbasedonmergingtheAD- DIE model to two learning taxonomies tailored to training sup- ported by the use of digital instrumentation was proposed. The selectedtaxonomiesincludedBloom’sdigitaltaxonomyandSolo’s taxonomy.

Bloom’sdigital taxonomyis a modernized variationof there- vised Bloom’s taxonomy (Churches, 2010). In Fig. 4, the initially proposed taxonomies discussed with the panel of experts are shown.

Severalcriticisms were raised duringthe discussion regarding appropriateness of the indicated taxonomies and their possible alignmenttoCStrainingdevelopmentmethodologies.

More in detail, the panelists found that aligning the ADDIE modeltothesetaxonomiesmaynotbeeasilyfeasibleandmayre- quireextensive further research andwork in adapting the afore- mentioned learning taxonomies to the requirements of CS train- ing. Panelistsalso criticizedthe taxonomieshierarchical structure oflearning, citingthatwhileitmaybeappropriateforknowledge acquisition,theymaynotbeadequateforCStraining.

Forthisreason,duringthesecondroundofquestionnaires,ad- ditionalinputwascollectedfromthepanelonmethodsandlearn- ingtheoriesthat maybemoresuitable toCStraining. Oneofthe mainconclusionsofthediscussionbetweenthepanelistswasthat differentmodesofinformationcommunication (utilizationofim- ages,text,videos, verbalcommunication, etc.)aswellasdifferent training delivery methods may be required to satisfy and tailor trainingtospecifictargetgroupsorindividuals.Forthisreason,an additionalsuggestionofutilizingmodern findingsofpersonalized learningtheory(PLT)wastakenintoconsideration.

PLTisgenerallydefinedasaneducationapproachthat aimsto customizelearningbasedonstudents’needs,interestsandabilities (WalkingtonandBernacki, 2020).ResarchonPLTisstill relatively novel,andmanyheterogeneousapproachesandmodelshavebeen proposed in the literature. Certain key elements of PLT that are commontomostoftheproposedmodelshavebeenestablishedin theliterature.InDiana(2019),thefollowing5elementsarehigh- lighted:

• Student Agency:Student Agency (SA)or other timesreferred toasownershiporcontrolindicatestudentsactivelytakingre- sponsibility and becoming active participants over their own learning,bybecomingmoreawareoftheirstrengthsandweak- nesses,advance mastered skills andreinforce skills they lack.

Teachersonly facilitatethe content acquisition whilestudents internalize it and own it too. To support this, it is necessary toprovidestudentswithpersonallearningspacesandactivities such as the ones provided by Learning Management Systems (LMS)andothersimilartools,onlineforumsandcommunities.

Additionallytogivingthemmoreautonomyandself-regulation, thisalsoallowtogiveandreceivefeedbackfrompeers,leading togreaterachievementlevels,greaterclassparticipation,better preparation, self-awareness and decreases in behavioral prob- lemsDiana(2019).

• FlexibleLearningEnvironments:Instructionisoftenstilldeliv- eredintraditionallearningenvironments, typicallyclassroom- based,whichareknowntohinderthelearningprocess.Inflex- ible learning environments, students have more control over howthey learn. Thisis achieved by modifying the traditional learningenvironment to one that enablesadditional coopera- tionbetweenstudents,aswell asmoreinteractivity.Designing sucha spacewithplaces forsolitary work,collaborative work andfordebatesandminilessonsgivesstudentsconfidenceand it leadsto improved academic results, better peer interaction andlessboredstudents(Diana,2019).

• Individual Mastery: With independent modules designed as part of a larger learning goal, students can focus on master- ingskills,atpreferredschedules,locationandpaceoflearning.

Forthis,teachershavetoofferindividualizedsupportandguid- anceandstudentsneedself-motivation,grit,perseveranceand agency.

• Personal Learning Paths: Personalized learning involves ad- justinginstruction to students learningpace withthe goal of trackinglong-termlearning. Thisiscurrentlyoftenassistedby LMS which help educators createclasses,assess students and add individualized learning paths. An LMS provides the nec-

Fig. 4. Revised versions of Bloom’s Digital Taxonomies and Solo Biggs’ Taxonomy for CS training.

essarytools forteachers to meetstudents needs by personal- izing goals within paths and creating a flexible virtual learn- ing environment. LMSs also give students agency over their learning andallows them to become proactive by having ac- cesstoself-assessmenttools,suchasquizzesandsurveyswith instant feedback, butalso a space to express themselves and keep track oftheir progress (Diana,2019). Personalized paths adapttomultiplelearningstylesandfocusonhowstudentsex- perience learningofferingcustomizablemodulesto respondto individuallearningneeds.

• Learner Profiles: Having individual learner profiles give pow- erful insights on progress, to teachers andstudents alike. By analyzing students progress teachers can create personalized content, assign individual goals and give customizable feed- back whilestudentsare abletobuild ontheir strengths,over- come their weaknesses andfollowtheir own goalsand inter- ests. While the process of creating learner profiles may have been more challenging in previous eras, the introduction of ever more sophisticated LMS has allowed both teachers and learnerstobettertracktheirprogressanddeveloppersonalized contentbasedontheirprofiles.

Models for PLT vary greatly, due to the heterogeneity of stu- dents’preferencesandneeds.Outofthevariousproposedmodels intheliterature,thefollowing4modelshavebeenhighlightedas themostcommoninacademicsettings(Morin,2020),withpossi- bleapplicabilityalsoforlearninginprofessionalenvironment:

• Learner Profiles-based models: This model keeps an up- to-date record that provides a deep understanding of each learner’sindividualstrengths,needs, motivations,progressand goals. The records are associated to each learner’s profiles, whichare periodicallyandoftenupdated, toaid bothinstruc- torandlearnertokeep trackoftheindividual’sprogressorto understandifthereisanyneedforchangesinlearningmethod- ologiesorotherrequirements.

• Personalized learningpaths-based models:Thismodel helps eachlearnercustomizealearningpaththatrespondsoradapts based on progress, motivations, and goals. An example of this would be a learner’s schedule based on weekly updates abouttrainingprogressandinterests.Eachlearner’sscheduleis uniqueandmightincludeseverallearningmethods.Aperson-

alizedlearningpathallowsalearnertoworkondifferentskills atdifferentpaces.

• Competency-based progression: This model continually as- sesseslearnerstomonitortheirprogresstowardspecificgoals.

Thissystemmakesitcleartolearnerswhatthey needtomas- ter.Thesecompetencies includespecific skills,knowledge and mindsets. Students are given options of how and when to demonstratetheirmastery.Forexample,astudentmightwork witha teachertoweavecertain mathskillsintoan internship ataretailstore.Thestudentmightworkonseveralcompeten- ciesatthesametime.Whentheymasterone,theymoveonto thenext.Eachstudentgetsthenecessarysupportorservicesto helpmastertheskills.Theemphasisisn’t ontakingatestand gettingapassingorfailinggrade.Instead,it’saboutcontinuous learningandhavingmanychancestoshowknowledge.

• Flexible learning environment-based models: This model simplyadaptstheenvironmentlearnerslearnin,basedonhow theylearnbest.

PLT hasbeen notedin the literature as beingparticularly ad- vantageouswhencomparedtotraditionallearningtheoriesdueto severalofitsproperties.AccordingtoastudybyPaneetal.(2015), whereusageofPLTinseveralacademicinstitutionwasmonitored andanalyzed, it isreported that compared to peers, students in schoolsusing PLTpractices are making greater progress over the course of two school years,and that those students who started out behindare catchingupto performatorabove nationalaver- ages.Thestudyfindsthatteachersatmostschoolswereusingdata to understandstudent progressandmake instructional decisions, all schoolsoffered time for individual academicsupport, andthe use of technology for personalization was widespread. However, some strategies,such ascompetency-basedprogression, wereless common andmore challengingto implement.Positive effects on studentmathematicsandreadingperformance wereshownasre- sultofadoptionofPLT,withevenlowest-performingstudentsmak- ing substantial gains relative to their peers. Adoptionof person- alizedlearningpracticesvariesconsiderably.Personalizedlearning practices that are direct extensions of currentpractice are more common, but implementation of some of the more challenging personalizedlearningstrategiesislesscommon.Threeelementsof PLThavebeencitedto givethe largestachievementeffectswhen implemented in tandem: Learner Grouping, Learning Space Sup-

Table 5

Metrics categories identified from the literature.

Metrics and

KPIs cat. Type Classification Measurement Units Data Source CS incident

records

Quantitative Effectiveness Number of data breaches or other incidents that occurred before and after training.

Internal Reports on attacks and incidents.

User Performance

Quantitative Effectiveness Outcome of CS exercises and tests; Comparison of pre-training and post-training test results;

Evaluation of threats detection, prevention and report rates, from tests and real-life occurrences

Data analytics from exercises; reports from evaluators; analytics about threat detection and reporting times.

User Feedback Qualitative Effectiveness &

Comprehen- siveness

User evaluation of training program’s content, delivery methods, accessibility, usability;

Improvement suggestions

Surveys; Questionnaires; Interviews.

Compliance to User Needs and Roles

Quantitative.

Comprehensiveness

Results of maturity models scoring; Internal evaluation (User feedback

& user performance evaluation methods);

Standard certification evaluation; Company or National standard/guidelines/ best practices compliance;

Compliance to Companies’

Requirements

Quantitative

Comprehensiveness

Results of maturity models scoring; User Performance evaluation methods;

Standard certification evaluation;

Maturity Models; Company or National standard/guidelines/ best practices compliance;

portsModel,andLearnersDiscussData(Paneetal.,2015).Accord- ing to a survey conducted on the students and teachers of the academic institution that participated in the studyconducted by Pane et al.(2015), teachers’ greater useofpractices that support competency-basedlearningandgreateruseoftechnologyforper- sonalizationintheschoolsinthisstudywithimplementationdata wasnoted.

To successfully integrate any of the PLT models identified in CS training,considerationonoverallrequirementsfortraining,re- sources available aswell asinput from learnersshould be taken intoconsiderationandbeutilizedtodevelopeachtrainingcompo- nentaccordingly.

4.7. CSTrainingframeworkmodel

After completing thesecond round ofquestionnaireofDelphi, we summarized the results of the discussion and classified the collectedfeedbackintocategoriesassociatedwithvariouscompo- nentsofCStrainingandtrainingdevelopment.Table6summarizes therecommendationsanddecisionstakenbythepanel.

Aconceptualmap ofthetrainingframeworkdevelopedispre- sentedinFig.5

As it can be seen from Fig. 5,when developing a CS training framework, 4 main aspects orcomponents arehighlighted asre- quiringprioritization:(1)trainingdevelopmentmodelandlearning modelselection,(2)trainingcontent,(3)trainingdeliverymethods andfinally(4)assessmentandevaluation.

For each of these components, it is recommended to imple- ment the recommendations found in the previous Sections 4.1– 4.5, respectively. Additionally,it can be noted that it is also rec- ommended to involve all training participants during the deci- sionanddevelopmentprocessoneachoftheseaspects,aswellas any ofthetrainingcomponents. Continuousfeedback isalso rec- ommended be collectedduring the developmental phases, to al- low developingpersonalizedprofilesorlearningpathsforpartici- pants,followingtoPLTrecommendations.Whileitisexpectedthat following such process will require an initial highresource over- head aswell asa longerset-up period,thisshould also facilitate

andshortenfuture updateandmodification requirements,by be- ingpreemptivelyvalidatedbytrainingparticipants.

Tofurtherdetailonthestepstobe takenduringdevelopment, wethenanalyzedpossibleintegrationsoftheADDIE modeltothe considerationsraisedinSection4.6.Forthis,wealignedtheobjec- tivesdefinedforeach ofthephaseoftheADDIEmodel,shownin Section4.1,totheconsiderationsofPLT.

• Analysis Phase: During the analysis phase, when establishing trainingneedsandgoals,itisrecommendedtoinvolvethese- lected target audiencein the process, by both analyzingtheir preferencesintrainingdelivery,butalsoonwhetherthegoals oftrainingalign withtheir currentgoals.Aside fromproblem andgoaldefinition,allother activities toconductduringanal- ysisphase includethe previously mentioned desiredoutcome establishment,pre-requirementdefinition,selectionofpossible learningenvironmentandestablishingoveralldurationoftrain- ing.All these decisions should take inconsideration any pos- sibleresourceconstraintthat maybe presentatthe organiza- tionthat isplanningon incorporatingthe training, bothinfi- nancialandmaterialtermsaswellashumanresourcesneeded.

Inthisphase aswell asinother phases,revisionshould occur basedonprogressivefeedbackgivenbybothtrainingdesigners andparticipants oneach establisheddecision, untila majority agreementisreachedonallattributes.

• Design Phase: When designing both the overall trainingpro- gram and each of its single modules, it is critical to take intoconsideration theindividual-specificfactors mentioned in Section 4.6. This means that learning material should be de- velopedbased on the learningstyle of participants andtheir preferences(audio,visualorothertypeofmaterial).Asidefrom the material, it is important that an overall structure of how the trainingwill be conducted andwhat content will be uti- lized during each module is decided by this point, together with more detailed lesson planning. It is important that the design phase is systematic and specific (Instructional, 2021), withsystematicmeaning logical,orderlymethodofidentifica- tion, development and evaluation of a set of planned strate-