The field of data fusion has been around for a long time. Data fusion has been used in a various number of areas, including situational awareness in military context, bioinformatics, robotics, medical diagnosis, remote sensing, and man-ufacturing [38, 39]. One of the earliest definitions of data fusion is given by White [40] as:
"a process dealing with the association, correlation, and combina-tion of data and informacombina-tion from single and multiple sources to achieve refined position and identity estimates, and complete and timely as-sessments of situations and threats, and their significance. The process is characterized by continuous refinements of its estimates and assess-ments, and the evaluation of the need for additional sources, or modifi-cation of the process itself, to achieve improved results."
Other definitions [41, 42, 43] focus on the use of multiple sensor sources to create an optimal estimate. Later work [44, 45] defines it as the process of com-bining data from multiple sensors to provide a better understanding of the sce-nario. That is, performing more specific inference which could not have been performed using single sensors. In [38], the authors present an overview of the various definitions given in literature. They identify common criteria and pro-pose a new definition based on the identified criteria. The propro-posed definition is:
"Information fusion is the study of efficient methods for automat-ically or semi-automatautomat-ically transforming information from different sources and different points in time into a representation that provides effective support for human or automated decision making"[38].
By performing this process of refinement, the collected data can be transformed into information, and further into knowledge. More specifically, knowledge can be defined as the interpretation of the information contained in the data [28].
In literature, several models for data or sensor fusion have been proposed.
The early work in the 1980’s related to data and sensor fusion and situational awareness was in military context [46]. Many of the models reflect this by being extensively oriented towards military domain, both process wise and terminol-ogy wise. The following section will present the design of the earlier proposed models together with their capabilities and flaws. Several of the models have common elements, and comparisons will be made.
3.2.1 The Intelligence Cycle
The Intelligence Cycle has its root in military operation. As with much termi-nology in data fusion, several terms may describe similar elements. The J-P 2.0 Joint Intelligence by the US Department of Defense [19] describes this as the Intelligence Process, and includes 5 phases for the process of creating intelli-gence from operational environments. The process is presented in Figure 10 -Intelligence process.
Figure 10: Intelligence process [19]
Planning and Direction – Includes activities related to the development of plans and the consecutive execution of such. Including, but not limited to: the identification and prioritization of intelligence requirements; the development of concepts of intelligence operations and architectures required to support the commander’s mission; tasking subordinate intelligence elements for the col-lection of information or the production of finished intelligence; submitting requests for additional capabilities to higher headquarters; and submitting requests for collection, exploitation, or all-source production support to exter-nal, supporting intelligence entities[19].
Collection Includes activities related to the acquisition of data as defined in the Planning and Direction phase.
Processing and Exploitation Includes activities related to the conversion of col-lected data into formats readily for entities such as commanders, decision makers, intelligence analysts and other consumers.
Analysis and Production Includes activities related to the production of intel-ligence from the collected information and from refined intelintel-ligence from other parties.
Dissemination and Integration Includes activities associated with the delivery to and use by a consumer. Means of delivery are determined according to needs.
3.2.2 JDL Fusion Model
The JDL Fusion Model was originally proposed by the US Joint Directors of Lab-oratories Data Fusion Sub-Group in 1985 [40, 47], and has thereafter been up-dated several times [48, 49]. The JDL Fusion Model is well presented in [50], by one of the authors working on the revisited version. The model describes the se-quential flow from (i) data, measurements and observations, to (ii) information, data placed in context, indexed, and organised, to (iii) knowledge, information understood and explained [50]. The proposed model is presented in Figure 11
-Figure 11: Process of data fusion as proposed by Waltz [50]
Process of data fusion as proposed by Waltz. The model uses five different levels of data refinement.
Level 0 Data refinement Calibration and filtering of raw data, such as bias cor-rection.
Level 1 Object refinement The measures are aligned to a common frame of reference. Correlation is performed based upon an association process in-dicating which observations from different sensors have common elements.
Level 2 Situation refinement Situational awareness is created based upon the aggregated sets of objects. Elements such as behaviour, common points of origin, common protocols, common targets, and other high-level attributes are used.
Level 3 Threat (meaning) refinement Future possible outcomes are determined using situational knowledge to model and analyse feasible future behaviour.
Level 4 Resource management (process refinement) The whole process is re-fined in this management level. It refines based on current situational awareness and additional data when required.
This model governs the process of data fusion well. However, the proposed model has a general approach towards data fusion, and more detailed specifica-tions is needed when applied to real life scenarios. The model provides a good basis for this project, which will focus more on a detailed modelling of early data fusion and reduction. The model does also not describe how to define the balance between data reduction and loss of valuable data well.
3.2.3 The Boyd Control Loop
The Boyd control loop [51, 52], commonly known as the OODA loop, contains four phases. Observe, Orient, Decide, and Act as shown inFigure 12 - Boyd Con-trol Loop. This process is represents the decision-support for situational aware-ness commonly used in the military. As situational awareaware-ness is one of the goals in data fusion, the Boyd control loop has been used in sensor and data fusion.
The four phases can be mapped to the JDL model. The authors in [46] com-pare the two models as follows,
Observe Comparable to level 0 of the JDL
Orient Comparable to the functions of level 1, 2, and 3 Decide Comparable to level 4
Act No directly comparable function as the JDL model does not close the loop.
Figure 12: Boyd Control Loop [46]
Figure 13: The Waterfall Fusion Model [53]
3.2.4 The Waterfall Model
A waterfall based model proposed by [53] heavily focuses on the lower level processing functions. The stages of this model as presented in Figure 13 - The Waterfall Fusion Modelcorresponds to level 0, 1, 2, and 3 in the JDL model.
Due to its similarities with the JDL model it has many of the same flaws [54].
The waterfall model is more detailed in analysing the fusion process, however, it lacks any feedback data flow. As seen inChapter 2 - Security Operation and Threat Intelligence (p. 5), security operation is a continuous process and a feedback loop is crucial.
3.2.5 The Dasarathy Model
Dasarathy [55] identifies five possible categories or levels of fusion. The cate-gorisation is dependent on the input and output of the fusion, and the author presents how previous categories can be mapped to this categorisation. The five
categories are presented inTable 5 - The Dasarathy Model.
Input Output Notation Analogues Data Data DAI-DAO Data-level fusion
Data Features DAI-FEO Feature select and feature extraction Features Features FEI-FEO Feature-level fusion
Features Decisions FEI-DEO Pattern recognition and pattern processing Decisions Decisions DEI-DEO Decision-level fusion
Table 5: The Dasarathy Model [55]
3.2.6 The Omnibus Model
Bedworth and Obrien [46] states that the existing fusion models are oriented towards military domain, thus the need for a model fitting the extensive data fusion community was necessary. They propose the Omnibus Model, which are based on the advantages of the previous models. It has the cyclic nature from the Intelligence Cycle and the Boyd Control Loop, the detailed definitions of the Boyd Control Loop which all can be mapped to one of the levels in the JDL model and Dasarthy Model.
Figure 14: The Omnibus Model [46]