The camera response is estimated from a set of input images based on the expectation maximization approach [Robertson et al. 2003]. The input images capture exactly the same scene, with correspondence at the pixel level, but the exposure parameters are different for each image. The exposure parameters have to be known and the camera response is observed as a change in the output pixel values with respect to a known change in irradiance. For the sake of clarity, in this section we assume that the only parameter is the exposure time, but in general case it is necessary to know how many times more or less energy has been captured during each exposure. Since the exposure time is proportional to the amount of light captured in an image sensor, it serves well as the required factor. In the mathematical formulas below, we obey the notation given in TableA.1and consider only images with one channel.
There are two unknowns in the estimation process. The primary unknown, the camera response function I, models the relation between the camera output values and the irradiance at the camera sensor, or luminance in the scene. The camera output values for a scene are provided as input images, but the irradiance x coming from the scene
A.2. MATHEMATICAL FRAMEWORK FOR RESPONSE ESTIMATION 121 is the second unknown. The estimation process starts with an initial guess on the camera response function, which for instance can be a linear response, and consists of two steps that are iterated. First, the irradiance from the scene is computed from the input images based on the currently estimated camera response. Second, the camera response is refined to minimize the error in mapping pixel values from all input images to the computed irradiance. The process is terminated when the iteration step does not improve the camera response any more. We explain the details of the process below.
Estimation of Irradiance
Assuming that the camera response function I is correct, the pixel values in the input images are mapped to the relative irradiance by using the inverse function I−1. Such relative irradiance is proportional to the true irradiance from the scene by a factor in-fluenced by the exposure parameters (e.g. exposure time), and the mapping is called linearization of camera output. The relative irradiance is further normalized by the ex-posure time tito estimate the amount of energy captured per unit of time in the input images i at pixel position j:
xi j=I−1(yi j)
ti . (A.1)
Each of the xi images contains a part of the full range of irradiance values coming from the scene. This range is determined by the exposure settings and is limited by the dynamic range of the camera sensor. The complete irradiance at the sensor is estimated from the weighted average of this partial captures:
xj=∑iwi j·xi j
∑iwi j . (A.2)
The weights wi jare determined for camera output values by the certainty model dis-cussed later in this section. Importantly, the weights for the maximum and minimum camera output values are equal to 0, because the captured irradiance is bound to be incorrect in the pixels for which the sensor has been saturated or captured no energy.
Refinement of Camera Response
Assuming that the irradiance at the sensor xjis correct, one can recapture the camera output values y′i jin each of the input images i by using the camera response:
y′i j=I(ti·xj). (A.3)
In the ideal case when the camera response I is perfectly estimated, the y′i jis equal to yi j. During the estimation process, however, the camera response function needs to be optimized for each camera output value m by averaging the recaptured irradiance xjfor all pixels in the input images yi jthat are equal to m:
Em={(i,j): yi j=m}, (A.4) I−1(m) = 1
Card(Em)
∑
i,j∈Em
ti·xj. (A.5)
Certainty model
The presence of noise in the capture process is conveniently neglected in the capture model in equations (A.1,A.3). A complete capture model would require characteri-zation of possible sources of noise and incorporation of appropriate noise terms to the equation. This would require further measurements and analysis of particular capture technology in the camera, thus is not practical. Instead, the noise term can be accounted for by an intuitive measure of confidence in the accuracy of captured irradiance. In typ-ical 8-bit cameras, for instance, one would expect high noise in the low camera output values, quantization errors in the high values, and good accuracy in the middle range.
An appropriate certainty model can be defined by the following Gaussian function:
w(m) =exp
−4·(m−127.5)2 127.52
. (A.6)
The certainty model can be further extended with knowledge about the capture process.
Normally, longer exposure times, which allow to capture more energy, tend to exhibit less random noise than short ones. Therefore an improved certainty model for input images yi jcan be formulated as follows:
wi j=w(yi j)·ti2. (A.7) Such weighting function minimizes the influence of noise on the estimation of irradi-ance in equation (A.2). This happens apart from noise reducing properties of the image averaging process itself.
Minimization of Objective Function
After the initial assumption on the camera response I, which is usually linear, the re-sponse is refined by interactively computing equations (A.2) and (A.5). At the end of every iteration, the quality of estimated camera response is measured with the follow-ing objective function:
O=
∑
i,j
w(yi j)·(I−1(yi j)−ti·xj)2. (A.8)
The objective function measures the error in the estimated irradiance for input images yi j when compared to the simulated capture of the true irradiance xj. The certainty model requires that the camera output values in the range of high confidence give more accurate irradiance estimates. The estimation process is terminated as soon as the objective function O falls below predetermined threshold.
The estimation process requires an additional constraint, because two dependent un-knowns are calculated simultaneously. Precisely, the values of xjdepend on the map-ping of I and the equations are satisfied by infinitely many solutions to I which differ by a scale factor. Convergence to one solution is enforced, in each iteration, through normalization of the inverse camera response I−1by the irradiance causing the medium camera output value I−1(mmed).