The Experiment in Large

The experiment in large was a success according to its premises. Most goals were met, and the experiment sessions were executed as planned without any technical or procedural problems.

However, we acknowledge that we may have embraced too much into this experiment. It was too many things going on at the same time. The execution timeframe was also too short. We realize that studying complex phenomena, such as problems within the area of information sharing and collaboration, probably would require multiple experiments that collect a common set of data so that the contributions of each individual factor can be isolated (19). Then, over time – moving to other experiments – the factors can be brought together in increasingly larger combinations so that their independent and interactive effects are highlighted.

The selection of the participants (operational personnel) was not ideal. They differed very much in background and skills. E.g. some of them had not relevant background and experience in intelligence work. This had of course implications for how well the teams performed together during the experimentation runs.

8 CONCLUSIONS

Explorative experiments are designed to generate new ideas or ways of doing things (“thinking outside the box”). As an exploratory experiment the experiment presented in this report has provided few clear answers, yet they are in accordance with our expectations. Our main conclusion is that the results support our view as promising:

• New technological solutions can increase the ability to establish a COP in situations where dynamic configuration of forces is necessary. This can increase shared situation awareness.

• The processes of picture compilation should be tailored to get the most operational value out of the new technological possibilities.

Generally, the participants had some overconfidence in their individual SA. Due to the complex scenario, the participants were in average only aware of 12 % of the elements in the situation (SA level 1). Despite the low awareness of elements, the participants were able to understand the situation correctly (SA level 2) and select right projections (SA level 3) to a larger extent.

Due to small number of responses the analysis of team SA and shared SA were limited to analysis of team SA level 1. There were differences on level 1 SA, however. It also seemed that there were variations in the level of teamworking between the groups. There was no significant correlation between the level of teamworking and the three SA levels. Yet, when comparing the team SA level 1 and the teamwork, a pattern emerged: A tendency towards a positive relation between teamwork and good SA. We could not decide upon the significance of this result. However, we regard this as an interesting observation in our further effort to study collaboration in teams and team SA.

what the participants were used to. This had implications on how well the teamwork was performed and to which degree they were able to utilize the new technology to efficiently support the team performance.

The results show that the participants appreciated the technology demonstrator, both the GeoViewer and the NetViewer. These components made it somewhat easier, efficient and effective to perform their tasks and problem solving activities. The experience of support from chat was more neutral. However, chat might represent a small improvement of problem solving efficiency. There were also interesting connections between the assessment of chat and SA. Overall SA in SART correlated positively on all support variables, i.e. good SA is related to positive evaluations of the demonstrator and chat.

The results of the experiment were in accordance with many of our expectations. Much has been learned about the possibilities and problems of measuring situation awareness. We also have gained more insight into the complex interplay between the involving organisational, procedural, human- and technological elements that constitute technology-supported collaboration in military operations. Several positively interesting observations and questions for further studies have been identified.

Altogether we consider the experiment a success, especially when considering the fact that this experiment was the first of its kind in the NBD-program at FFI (sosio-technical type of experiment that besides technology also embraced the cognitive and social domain) and one of the firsts and very few experiments in the area of the Decision-support Component (regarding the structural model in the Norwegian NBD) in the Norwegian CD&E program.

The results support our assumption that the human and organisational aspects - together with technology - must be included in the NBD transformation efforts in order to explore and learn more about this complex interplay.

References

(1) Hansen Bjørn Jervell, Mevassvik Ole Martin, Bråthen Karsten, Rose Kjell (2004): A Picture Compilation Concept for Network Based Defence, FFI/REPORT 2004/00983, Norwegian Defence Research Establishment (FFI) (In Norwegian).

(2) Hansen B J, Mevassvik O M, Bråthen K (2003): A Demonstrator for Command and Control Information Systems Technology Experimentation Proceedings of the 8^th

International Command and Control Research and Technology Symposium, Washington, USA, 2003.

(3) Rasmussen R, Gagnes T, Gustavsen R, Hafnor H, Hansen B, Haakseth R, Mevassvik O M, Olafsen R, Rose K (2004): Exploratory Experiment “Ad hoc Organization of Picture Compilation” Conducted during Blue Game 2004: Evaluation Report, FFI/RAPPORT-2004/01940 (Main Report), Norwegian Defence Research Establishment (FFI).

(4) Hafnor H, Olafsen R (2004): Exploratory experiment of ad hoc organization of picture compilation in NBD: Evaluation of operational value – Blue Game 2004, FFI/NOTAT-2004/01885, Norwegian Defence Research Establishment (FFI) (In Norwegian).

(5) Reitan B, Enemo G, Bjørnstad A L, Hafnor H (2005): Experiment Report: “Negotiation Based Resource Allocation” – Battle Griffin 2005, FFI/RAPPORT-2005/01590,

Norwegian Defence Research Establishment (FFI) (In Norwegian).

(6) Endsley, M R (1995): Towards a theory of situation awareness in dynamic systems, Human Factors 37, 1, 23-64.

(7) Hauland G (2002): Measuring Team Situation Awareness in Training of En Route Air Traffic Control: Process Oriented Measures for Experimental Studies, Ph.D. Thesis, University of Aarhus, Department of Psychology.

(8) Dickinson T L, McIntyre R M (1997): A Conceptual Framework for Teamwork Measurement. In: Brannick M T, Salas E, Prince C (Eds): Team Performance

Assessment and Measurement. Theory, Methods and Applications. Lawrence Erlbaum.

(9) Paley M J, Serfaty D, Baker K, Miller P, Baily A, Ganber G, Wan L (2002): Adaptive Performance in Warfighting and Peacekeeping Best Practices Report: Description of the DDD-SASO, Technical Report A003, Army Research Lab, Contract Ng1339-01-0049, Aptima, Inc, Woburn, MA.

(10) Endsley, M R (1988): Design and evaluation for situation awareness enhancement.

Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 97-101). Santa Monica, CA: Human Factors Society.

(11) Taylor, R M (1990): Situational awareness rating technique (SART): The development of a tool for aircrew systems design, Situational awareness in aerospace operations (AGARD-CP-478) (pp. 3/1-3/17). Neuvilly Sur Seine, France: NATO – AGARD.

support of Situation Awareness in NBD - Method and results, FFI/RAPPORT-2005/01614, Norwegian Defence Research Establishment (FFI) (In Norwegian).

(13) Rasmussen R, Rose K, Hansen B J, Langmyr A, Bjørnstad R P, Fjeld S I (2005): Picture Compilation Demonstrator - Used for experimentation during Battle Griffin 2005, FFI/NOTAT-2005/01474, Norwegian Defence Research Establishment (FFI) (In Norwegian).

(14) Hafnor H, Hansen B J, Rose K (2005): (U) Scenario specification for the Battle Griffin 2005 experiment, FFI/NOTAT-2005/01542 (Restricted), Norwegian Defence Research Establishment (FFI) (In Norwegian).

(15) Hafnor H, Normark R (2005): Ad hoc Organisation of Distributed Picture Compilation and Support for Situation Awareness in Network Based Defence – An Exploratory Experiment, Proceedings of the 10^th International Command and Control Research and Technology Symposium, McLean, VA, 2005.

(16) Fjeld S I (2005): NetViewer - Technical insight into a Web Services oriented GUI client, FFI/NOTAT-2005/01616, Norwegian Defence Research Establishment (FFI) (In Norwegian).

(17) Hedenstad O E (2004): Experiment Specification, Vedlegg til Battle Griffin EXercise Planning Instructions (EXPI).

(18) Mevassvik O M (1996): SensorSim – A Simulation Tool for Ocean Surveillance, FFI/NOTAT-96/05092, Norwegian Defence Research Establishment (FFI) (In Norwegian).

(19) Alberts D S, Hayes R E (2005): Code of Best Practice: Campaigns of experimentation, Pathways to Innovation and Transformation, CCRP Publication Series. 2005.

APPENDIX