This thesis followed Peffers et al.'s DSRM [349] to approach the development of the contextualized learning application for software security as a series of five iterations, with each iteration indicating a specific design cycle (see Figure 3.5). Hevner et al.
([203], p. 89) term this iteration the ‘generate/test’ cycle. The evaluation of our artifacts, as for most DSR that deals with human–artifact interaction, took the form of an experiment. In a DSR project, the research process frequently iterates between development and evaluation phases rather than flowing in waterfall fashion from one phase into the next [255].
After the identification of the research problem and motivation, given in sections 1.2 and 1.3 respectively, a five-iteration design activity was carried out, in which each design cycle (DC) contained the following steps: objectives for a solution, design and development, demonstration, and evaluation. Evaluations were done for each cycle, rather than only once at the end of the design process. Each design-cycle not only derives designed artifacts but also results in knowledge contribution through communication, which involves professional and scholarly publications and presentations [349].
DC 1: A Socio-technical framework for security learning in the context of OSS development
Drawing on Figure 3.5, the first design cycle concerns establishing a socio-technical framework of security learning in the context of OSS development. This design activity started with analyzing the existing body of knowledge on OSS security practices using the method of Systematic Literature Review (SLR). SLR study is a defined and methodical way to summarize the empirical evidence concerning treatment or technology, to identify missing areas in current research or to provide background in order to justify new research. It provides a much stronger basis for making claims about the research questions [230, 326]. Based on the identified and relevant articles, the result of the SLR study gave an insight into the gaps in the literature on socio-technical perspectives and knowledge management practices of OSS security. This step of the DSR process was addressed with research question 1.1 and outlined in RP I.
After SLR, two empirical studies were conducted to investigate the real-world problems and to identify prospectus, limitation, and uncertainty embedded in the security practices and learning of security knowledge in OSS development environments. The empirical study is a way to gain knowledge by the collection and analysis of primary data based on direct observation and/or measurement methods in the ‘problem domain’ [518]. The former refers to qualitative, and the latter refers to quantitative research methods [29]. Qualitative research methods are used to explore why or how a phenomenon occurs, to develop a theory, or describe the nature of an individual’s experience, while quantitative methods address questions about
54
Figure 3.5:Iterations of DSR design cycles.
55
causality, generalizability, or magnitude of effect [146]. To answer RQ 1.2, a quantitative research approach was adopted, in which a questionnaire was prepared to gather information from OSS participants on the social and cultural aspects related to secure software development in OSS communities. The findings from the study were summarized and reported in RP II.Further, to answer RQ 1.3, in the second study, a Mixed-Method Research (MMR) design was selected in order to broadly explore and understand the socio-technical aspects of security learning in the context of OSS development, as well as the interaction effect among the observed factors. MMR frequently referred to as the
‘third methodological orientation’ [436], draws on the strengths of both qualitative and quantitative research. While there is no universal definition of mixed methods research, Creswell and Plano Clark [95] outline its core characteristics: in a single research study, both qualitative and quantitative strands of data are collected and analyzed separately, and integrated – either concurrently or sequentially – to address the research question. Onwuegbuzie and Combs [337] concur, writing, “mixed analyses involve the use of at least one qualitative analysis and at least one quantitative analysis – meaning that both analysis types are needed to conduct a mixed analysis” (p. 414). Instead of approaching a research question using the binary lens of quantitative or qualitative research, the mixed methods research approach has the ability to advance the scholarly conversation by drawing on the strengths of both methodologies.
In MMR, qualitative data is first collected and analyzed, and themes are used to drive the development of a quantitative instrument to further explore the research problem [95, 337, 436]. As a result of this design, two stages of analyses were conducted: an exploratory stage and a confirmatory stage. The reason for employing an exploratory study in the first stage was that important constructs relate to socio-technical aspects of OSS development and their influence on security learning were unknown, and relevant quantitative instruments were not available. In the first stage, data were collected adopting a qualitative-ethnographic research method in the three selected OSS projects. Ethnography focuses all the details of what members of culture do in their daily actions since culture is enacted through these details [18, 407]. Specifically, this study employs a socio-technical systems approach to systematically and holistically take into account the social context as well as technological aspects of the studying subjects. In fact, a socio-technical perspective can provide a stronger framework than any other approach because of its integrative and holistic nature [279]. With the identification of socio-technical challenges, the study then examined the main factors that were once disproportionately considered in the learning of security knowledge in OSS development. This study resulted in a conceptual socio-technical framework, which describes the interrelationship among social aspects (cultural and structural), and security knowledge sharing and learning behavior. This conceptual framework accompanying seven hypotheses was validated through an empirical examination, including the questionnaire design, data collection, and statistic correlation and linear regression analysis from 324 valid questionnaires. The
56
findings from the exploratory stage were reported in research paper III, while the results of the confirmatory stage were reported in RP IV.
DC 2: A context-based learning approach for software security
The primary objective of the second design cycle was to propose a novel method of artifacts for structuring and presenting software security knowledge, answering research question 2.1. To this end, a context-based learning approach was first proposed, adopted from concepts of CBL and literature from psychology and education. The artifact was further evaluated to prove the effectiveness in improving learners’ learning outcomes on studying software security, answering research question 2,2. A two-round experiment was conducted with 42 Bachelor students to evaluate the effectiveness of the proposed learning approach versus conventional learning materials. The method of experiments allows researchers to achieve high internal validity by carefully controlling the conditions under which an experiment is carried out [223]. Two types of the instrument were designed and built in the data collection scheme, including (1) pre-tests and post-tests for measuring knowledge gain, and (2) survey questionnaires for measuring learning satisfaction. After the design and evaluating this iteration, this work was communicated to the research community with RP V.
DC 3: A context-based ontology for managing contextualizing security knowledge Taking the proposed learning approach into further design consideration, DC 3 focused on the artifact of the ontological knowledge model, addressing RQ 3. The ontology is a key component to model the security knowledge and to support the development of the learning system, so having a distinct design cycle to validating this component was necessary. The objectives of DC 3 were three-fold, (1) to design and construct an ontological knowledge base to manage contextualized knowledge, (2) to validate the feasibility of ontology, and (3) to visualize the knowledge representation as a pre-study for DC 4. In accordance with the strategies in the proposed learning approach, the design of the ontology was composed of three modeling activities: application context modeling, domain knowledge modeling, and contextualized knowledge modeling. The ontology was constructed and demonstrated in Protégé editor and validated through a three-phase evaluation process: domain expert evaluation, competency question evaluation, and application-based evaluation. The design, development, and evaluation of the ontology in this design cycle were summarized in RP VI.
DC 4: An ontology-based contextualized learning system for software security The objective of the fourth design cycle was to develop a contextualized learning system for software security. This artifact was designed as a proof-of-concept to security educators regarding an ontology-driven web application for context-based learning, integrating the designed artifacts from DC 2 and DC 3. The former suggests