2.5 Spectral Image Quality
2.5.3 Round-Robin Tests
Assessing the true performance of an analytical technique or of scientific equip-ment on real experiequip-mental data is important in order to determine both their utility and to understand how the technique or equipment can best be used. As-sessments of this sort are usually carried out in a single laboratory, where the technique or equipment is tested against reference standards or against other validated equipment. However, scientific equipment is not always used under the ideal controlled conditions that exist in testing laboratories. Exactly how the equipment or technique is used and the environment in which it is used can significantly influence the results obtained.
A round-robin test attempts to take these influences into account and, instead of carrying out tests within a single laboratory, a number of locations and testing facilities are used to assess performance. The variability found in the obtained results can provide additional insight into the use of a technique and help iden-tify procedures or protocols that require improvement.
Round-robin tests are, therefore, a useful means of comparing equipment or methodologies and have been successfully carried out in a wide range of fields.
Within fields related to spectral imaging, round-robin tests have included topics such as the measurement of the BRDF (Bidirectional Reflectance Distribution Function) of diffuse reflectors [103], the radiometric calibration of a satellite multispectral sensor[104]and tests of field spectrometers in laboratory settings [105].
Several papers within this thesis contain or refer to results from a large-scale round-robin test that was carried out to evaluate a range of multispectral and hyperspectral imaging systems. The rest of this section, therefore, provides some supplementary background information on this round-robin test in order to put this work into context.
The experimental data analyzed in Paper 2and referred to in Paper 4 con-sisted of a subset of the data acquired from a round-robin test carried out for the COSCH project [106]. The round-robin test involved nineteen different institutions, including research laboratories, universities, equipment manufac-turers and museums and an overview of the tests is provided in Supporting
Paper S1, “A Study of Spectral Imaging Acquisition and Processing for Cultural Heritage”[8].
The round-robin test was a coordinated effort to gain a better understanding of the instrumentation, the processes of data acquisition and the effects of the de-vices and methodology on the reliability of the spectral imaging data. The goal was to evaluate the effective limits in accuracy of the spectral imaging equip-ment in use within the participating institutions and examine the challenges and issues that arose from bringing these different data sets together. By mea-suring and understanding the variability seen within them, the goal was to help improve protocols for the acquisition, handling, processing and sharing of spec-tral data sets.
The data acquisition for the round-robin test was carried out using either mul-tispectral or hyperspectral equipment from different manufacturers and with different experimental acquisition setups, procedures and methodologies. The aim of the comparison was not to compare the hardware specifications or raw performance of the imaging devices themselves, but to measure the resulting effective performances of the systems globally under their standard operating conditions and after the application of the calibration and processing workflows that are usually applied by each institution to their system. The variability due to the different setups, operating procedures and the way data was processed was, therefore, an important factor to take account of and include in the study.
In this way, an insight into the practical limits in accuracy of spectral imaging systems within routine operating environments could be gleaned.
The overall aim was to use the insight obtained from these tests to help stan-dardize methodologies and best practices for imaging cultural heritage objects through spectral imaging techniques and the insight gained contributed to the acquisition and calibration workflows defined inPaper 4.
A number of test targets were used, each with very different characteristics and each of which aimed to test different aspects of the spectral imaging equipment under evaluation. These test targets consisted of four objects, which can be seen in Figure 2.10 and included: a Macbeth ColorChecker, a wavelength standard, a custom-made pigment panel and a printed polychrome lithography 19thcentury Russian icon.
The ColorChecker used was a standard Xrite color chart consisting of twenty-four colored patches in a twenty-four-by-six grid, which provided a basis for standard-ized colorimetric comparisons between systems.
2.5. Spectral Image Quality 29
Figure 2.10: Targets used in the spectral imaging round-robin test: 24 patch Macbeth ColorChecker (top left), wavelength standard featuring well-defined and narrow features over the UV, visible and near infra-red spectral ranges (top right), custom-made pigment panel with 7 Renaissance-era pigments (bottom left) and a mass-produced 19thcentury Russian icon made through polychrome lithography (bottom right). (Image source:[8])
The wavelength standard was a Zenith Polymer wavelength standard consist-ing of a chemically inert diffuse lambertian reflectance standard composed of PTFE (Polytetrafluoroethylene) doped with the oxides of the rare earth ele-ments Holmium, Erbium and Dysprosium. This combination gives the standard a stable spectrum of characteristic, well-defined and narrow features over the ultra-violet, visible and near infra-red spectral ranges, which is suited for use in accurate spectral calibration.
The pigment panel was a test target created especially for the round-robin test by one of the participating institutions. It was made up of seven historical
pig-ments at different concentrations in an egg tempera binder. The paint prepa-ration and application on the panel aimed to authentically reproduce the me-dieval Tuscan painting technique described in Cennino Cennini’s 15thcenturyIl Libro dell’Arte[107]. The panel consisted of a wooden support with a gypsum ground, a canvas layer, and a second gypsum ground layer. Before applica-tion of the paint layer, five types of drawing materials (watercolor, charcoal, graphite, a lead and tin-based metalpoint and a lead-based metalpoint) were used to create lines and line patterns that were then covered with paints ap-plied with two different thicknesses. Additional details on the panel and on the source and composition of the pigments used for the panel can be found in [108].
The final object was a 19th century, mass-produced Russian icon, printed by polychrome lithography, using eight different inks, onto a tinned steel plate and nailed onto a wooden panel. It has a glossy surface over the colored areas and a high specular reflectance from the golden metallic surface. The icon was used to investigate the spatial imaging characteristics of the spectral imaging systems as well as their behavior with highly reflective surface.
Supporting Paper S1 provides an overview and preliminary results from the round-robin tests with further results found in several different papers. Paper 2provides an in-depth analysis of the hyperspectral data from two of the test targets. Details and results from the Macbeth ColorChecker can be found in both[109]and[110]. Results from the 19thcentury Russian icon can be found in Supporting PaperS2,“Assessment of Multispectral and Hyperspectral Imaging Systems for Digitisation of a Russian Icon”[7].