Adapting the filtering window

The smoothing matrix, H, affects the shape of the window function. The simplest choice of the smoothing matrix is H=hI, whereh is a global scale parameter and I is the identity matrix. This corresponds to an isotropic filter support. A large scale parameter setting will reduce the variance, but can lead to overly smoothed images (bias). Ideally, it is desirable to have large window supports in regions where the smooth polynomial model, used for the reconstruction, is a good fit to the underlying signal, while keeping the window size small close to the edges or important image features. The optimal choice of his dependent on the sensor characteristics (noise) and the irradiance in the scene. In paper D, we treated the scale parameterhas a user defined quantity and set it to a global value used for the reconstruction of all HDR pixels. We refer to this method as Local Polynomial Approximation (LPA). However, to obtain a better trade-off between image sharpness and noise reduction, in paper F we propose to locally adapt the smoothing parameterhto image features and the noise level of observed samples. For this purpose we describe an iterative algorithms for selecting the locally best smoothing parameter,h, for each HDR pixel estimate,z_j, individually.

The intuition for the scale selection algorithm is that the scale parameters should be increased as long as the variance in the signal reconstructions can be ex-plained by the underlying signal noise. Figure5.9illustrates how a signal value, the black point, is being estimated using a kernel with a gradually increasing smoothing parameter,h. When the smoothing parameterhis increased from h₀ theh₁, i.e. a higher degree of smoothing, the variance in the estimated value can be explained by the signal variance. When the smoothing parameter is increased from h₁ toh₂, the kernel reaches the step in the signal and the estimation at the black point can no longer be explained by the signal variance.

Smoothing parameterh₁thus produces a better estimate.

The adaptation of the smoothing parameter,h, is carried out iteratively. We start with a smallhand try to increment it in small steps. In each iteration we estimate the signal value and its variance. Anupdate ruleis then applied, which determines ifhshould be increased further or not. This is repeated until the update rule terminates the recursion or the maximumhvalue,h_max, is reached.

In paper G we presented two such update rules, described in more detail below.

EVS - Update ruleThe first update rule is built on the intuition that if the weighted mean reconstruction error is larger than the weighted mean standard deviation, the polynomial model does not provide a good fit to the underlying image data. The smoothing parameter, h_i, is iteratively increased with an incrementh_inc. In each iteration,i, the EVS update rule computes the weighted reconstruction errore_ias

e_l= ∑

k

W(k,k)∣f˜(X_k) −fˆ_k∣, (5.10) This estimate is compared to the standard deviation of the HDR pixel estimate that can be computed using the covariance matrixM_C for the fitted polynomial coefficients, ˜C, given by

M_C= (Φ^TWΦ)⁻¹Φ^TWΣW^TΦ(Φ^TW^TΦ)⁻¹ (5.11) whereΣ=diag[σ²_f1,σ²_f2, ...,σ²_fk]is the variance of the observations. During the iterations, the smoothing parameter,h_i, is updated toh_i+1=h_i+h_incas long as the weighted reconstruction error,i, is smaller than the standard deviation ˜σl, i.e. _i<Γσ˜zˆ_j,hi, whereΓis a user specified parameter controlling the trade-off between levels of denoising applied by the kernel.

ICI - Update ruleThe second update rule proposed in Paper F is based on the Intersection of Confidence Intervals (ICI) rule for adaptive scale selection, first developed for non-parametric statistics applications [71] and later also applied to other imaging problems such as denoising [107]. Using ICI, the smoothing parameter,h_min≤h_i≤h_max, is iteratively increased. For each iteration,i, the ICI rule determines a confidence interval,D_i= [L_i,U_i]around the estimated signal as:

L_i=zˆ_j,hi(x) −Γ˜σ_zˆj,

hi (5.12)

U_i=zˆ_j,hi(x) +Γ˜σzˆ_j,hi (5.13) where ˆz_j,hi(x)is the estimated radiant power given the scaling parameterh_iand σ˜zˆ_j,hi is the weighted standard deviation of this estimate computed using Eq.

5.2 ● Contributions 87 5.11.Γis a scaling parameter controlling how wide the intersection interval is.

During adaptation,h_iis increased as long as there is an overlap between the confidence intervals, i.e. h_iis updated to h_i+1=h_i+h_inc if there is an overlap betweenD_i andD_i+1. In practice, we utilizeΓas user parameter enabling a intuitive trade-off between image sharpness and denoising. A detailed overview of the ICI rule and its theoretical properties can be found in [18].

Anisotropic window supports

A problem with isotropic window supports is that they limit the adaption to the underlying image features. If, for example, an output pixel is located near a sharp edge, the neighboring samples cannot be represented accurately with a finite polynomial expansion, and thus a small isotropic window support needs to be used. However this limits the number of observed samples that are used to fit the polynomial and hence increases the noise in the estimate.

It is therefore desirable to adapt the window support so that it can include several observations on the same side of the edge, but not the ones on the other side of the edge. Intuitively the shape of the window support should be: circular and relatively large in flat areas to reduce noise, elongated along edges, and small in textured areas to preserve detail. In paper F we propose to use a method inspired by previous work in image denoising [195] to adapt anisotropic window functions for this purpose.

To adapt the window functions we use a two step approach: First, we use a regular reconstruction with isotropic window support andM≥1 to compute an initial estimation of the gradients,[^∂f(X_∂x1^j),^∂f(X_∂x ^j)

2 ]∀j, in the image. In a second step we then adapt the smoothing matrix,H_j, for each HDR output pixel, to reflect the dominant directions of the estimated gradients in a neighborhood aroundX_j. The result of this process is elliptical window functions that elongate along edge structures in the image, as well as adapt their size based on the signal smoothness in the neighborhood.

To robustly estimate the window supports in areas with noisy gradient esti-mates we use a parametric approach, that describes the gradient covariance in the neighborhood using three parameters,σ_jdescribing an elongation along the principal directions,θ_idescribing a rotation angle andγ_idescribing an overall scaling. These parameters are fitted to the local gradient structure using regu-larization to ensure that the window is not adapted to drastically, eg. shrinking to a single point. The details of the anisotropic window estimation is given in paper E, following the approach proposed by Takeda et al. [195], but with the difference that we adapt the window functions for each HDR output pixel instead of the kernel functions for each observed measurement.

Color reconstruction

For isotropic window function we generally consider different scale parameters for the green and red/blue color channels. This as the standard bayer pattern uses more green samples per unit area. Generally, different color channels also saturate at different levels, which can require different minimum values ofhto make sure that enough samples are used to fit the local polynomial model, ie to ensure that(Φ^TWΦ)is invertible in equation (5.9).

In paper F, we proposed to adapt anisotropic windows to the gradient infor-mation in the green channel first, and then use these adapted windows to also estimate the red and blue channels. This effectively forces the interpolation to be performed along the same signal features across all color channels. This often leads to less perceptually disturbing color artifacts and fringes in high frequency regions. We refer to this reconstruction method asColor Adaptive Local Polynomial Approximation (CALPA).