Single image super-resolution (SISR)
CHAPTER 4. SINGLE IMAGE SUPER-RESOLUTION (SISR) 50
Figure 4.2: Scatter plots comparing the objective IQA methods PSNR and Ma et al. (Y-axes), with the Mean Opinion Score (MOS) (X-axes). Every point is a super-resoluted image, colored by SISR algorithm. Reprinted from Ma et al.,2017. Copyright 2021, with permission from Elsevier
on these scatter plots alone, there seem to be strong evidence in favor of the Ma et al. metric over the PSNR metric.
In the overview of IQA metrics (Figure 4.1) we also notice the terms No-Reference (NR) and Full-Reference (FR). A FR IQA method compares the distorted image, i.e., a SISR estimate, with the ground truth image, the reference image. A FR method is thus reliant on the existence of a reference image, something that is seldom available in real-world applications of IQA. A NR IQA method, on the other hand, is only a function of the distorted image itself. FR and NR has its strengths and weaknesses. In SISR research it is common to report metrics from both categories, as long as a reference image is available. (Athar & Wang,2019)
4.2.1. Peak Signal-to-Noise Ratio (PSNR)
The Peak Signal-to-Noise Ratio (PSNR), a metric derived from the well-known MSE we intro-duced in Section3.6, has historically been the default objective IQA method. A decibel scale is used to increase interpretability compared to a raw MSE metric. The historical standing, combined with its ease of use and good interpretability, is probably the reason why it is still used extensively in SISR today. PSNR is historically defined for one-channel (grayscale) images only.
For color images, a color space transformation, i.e., an RGB-to-HSV, is usually performed so that PSNR may be computed only on the value/brightness (V) channel. (Athar & Wang,2019)
PSNR = 10 log10 L2
MSE = 20 log10 L
MSE (4.1)
whereL is the dynamic range of pixel intensities. In many image applicationsL= 28−1 =
CHAPTER 4. SINGLE IMAGE SUPER-RESOLUTION (SISR) 51 255, the dynamic range of an 8-bit image. In the case of 11-bit WorldView-2 and GeoEye-1 satellite images,L= 211−1 = 2047. Lmay also be a decimal number, for instance1.0, a typical value when dealing with preprocessed images in neural networks. The use of L makes PSNR, to a certain extent, invariant to the dynamic range of the image.2 (Horé & Ziou, 2010; Maxar, 2019a,2019c)
Despite its status as a default IQA metric, it has been repeatedly shown that it correlates poorly with subjective IQA methods. It relies on assumptions that do not hold when measuring visual perceptual quality. For instance, PSNR is independent of any spatial relationships in the image; it is only a pixel-by-pixel evaluation. Shifting the whole image by one pixel in any direction would lead to a disastrous PSNR score, yet probably no noticeable difference when evaluated by a human observer. We illustrated this with a checkerboard example in Section1.2.
For a comprehensive, and as a matter of fact funny, evaluation of PSNR, see Z. Wang and Bovik, 2009.
A note on MSE and MAE of 2D, 3D and 4D tensors
We introduced MSE (3.4) and MAE (3.5) as functions operating on vectors. In SISR problems we are usually interested in 2D (H×W) or 3D (H×W×C) images stored in tensors. Additionally, during training the loss is computed over a mini-batch of images, extending the number of dimensions to 4 (B×H×W×C). Luckily, the extra dimensions does not add much complexity to the computation of MAE and MSE. Given pairs of SISR predictions XSR and ground truth HR imagesXHR we compute the residualsE
E=XHR−XSR, whereE, XHR, XSR∈RB×Hhr×Whr×C
MAE and MSE are then computed element-wise. This is equivalent to vectorizing, or flat-tening,E intoe.
e= vec(E), e∈RBHhrWhrC
By doing this we may then use the vector notation in (3.4) and (3.5), with a modified denominator:
2Rounding errors occur, especially for integer data types.
CHAPTER 4. SINGLE IMAGE SUPER-RESOLUTION (SISR) 52 4.2.2. Natural Image Quality Evaluator (NIQE)
The Natural Image Quality Evaluator (NIQE) is a frequently used so called opinion-unaware no-reference IQA algorithm introduced by Mittal et al., 2013. The metric is on a continuous scale where most images receive a score between 0 and 10, and a lower score is better.
A reference model is created by fitting a multivariate Gaussian model to a training set of natural and pristine images. Let us consider a distorted image. At evaluation time the same model is fitted to the distorted image. The NIQE score is then a function of the distance between the parameters of the two model fits. A shorter distance means that the distorted image is more similar to the training set of natural and pristine images. By this measure one can say that the distorted image is more natural and pristine, i.e., it has a higher perceptual quality. (Mittal et al.,2013)
NIQE is a no-reference metric since the algorithm does not depend on having access to a reference image. It is opinion-unaware because it has not been trained on human-evaluated distorted images. NIQE is only aware of how natural and pristine images look and use this knowledge to estimate image quality. (Athar & Wang,2019)
4.2.3. Ma et al.
Figure 4.3: The main steps of the Ma et al. performance measure. Reprinted from Ma et al., 2017Copyright 2021, with permission from Elsevier
Ma et al., 2017 proposed an image quality metric specifically to evaluate SISR algorithms.
They did not explicitly name their algorithm, so it is commonly referred to as the Ma et al.
metric or just Ma metric for short. The metric is on a continuous [0,10] scale where a higher score correlates with higher perceptual quality. Similarly to NIQE it is also a no-reference measure. However contrary to the opinion-unaware NIQE, Ma is opinion-aware. A regression model has been fitted to a dataset of human-evaluated super-resoluted images. This also has the added effect of making it optimized to pick-up on typical SR artefacts.
CHAPTER 4. SINGLE IMAGE SUPER-RESOLUTION (SISR) 53