Random Sparse Coded Aperture for Lensless Imaging

Fulltekst

(1)Pacific Graphics (2014) J. Keyser, Y. J. Kim, and P. Wonka (Editors). Short Papers. Random Sparse Coded Aperture for Lensless Imaging Z. Wang and I. Lee School of Information Technology & Mathematical Sciences The University of South Australia, Mawson Lakes, South Australia, 5095. Abstract This paper develops a computational lensless imaging system based on a random sparse coded aperture. The camera consists of a thin mask with a coded pattern and a standard sensor array. The proposed coded aperture contains multiple square pinholes, and forms a superposition of multiple pinhole images. In order to reduce the artefact due to diffraction or interference and simultaneously to facilitate the fabrication, the pinholes are designed bigger than some other proposed ones, and sparsely spread on the mask. Only the diffraction pattern for one pinhole imaging model needs be taken into account to improve the angular resolution. An arising issue is that the resulting optical transfer function (OTF) involves many zero-value spectrums, which adversely affects the reconstruction quality with conventional image decoding techniques. We introduce a reselection scheme, which selects partial Fourier samples to reduce the impact of zero entries in OTF. Then, the total variation minimization with quadratic constraints algorithm is applied to attain a good quality reconstruction. Categories and Subject Descriptors (according to ACM CCS): I.4.1 [IMAGE PROCESSING AND COMPUTER VISION]: Digitization and Image Capture—Sampling. 1. Introduction Despite the popularity of lens based optical imaging, coded aperture (CA) based lensless imaging has recently received lots of attentions [GIG11]. The basic idea of coded aperture imaging (CAI) is the use of multiple pinhole apertures to acquire a superposition of pinhole images. The recorded picture is then decoded using digital image processing techniques. Similar to pinhole optics, CAI has several advantages over lens cameras, such as wide field of view, virtually infinite depth of focus, and free from linear distortion. In contrast to single pinhole cameras, the use of multiple pinholes reduces the exposure time and increases the signalto-noise ratio (SNR). A significant drawback for pinhole imaging is its low resolution in comparison with lens based cameras. CA was put forward originally for imaging short wavelength (e.g. x-rays and gamma rays) [Dic68] where diffraction and interference are negligible, so the pinholes can be made small enough to obtain the desired resolution. However, the optical effects have to be taken into account when CA is applied to visible imaging. The resolution for one pinhole aperture is limited by its geometrical or wave optics. A big pinhole produces a large uniform geometrical shadow of the hole, so the resolution depends on the pinhole size. A tiny pinhole produces c The Eurographics Association 2014.. DOI: 10.2312/pgs.20141251. a Fraunhofer or Fresnel diffraction pattern which causes a blurry image. In addition, densely distributed pinholes produce severe interference noise. So, seeking a proper CA is vital for specific applications. In the past decade, lots of efforts have been devoted to lensless imaging. In 1999, Mielenz gave a theoretical bound of diffraction limit for lensless imaging [Mie99]. In 2003 Potuluri et al. [PGAB03] developed a reference-structurebased lensless imaging system to track the motion of an object, and then in 2006 Zomet and Nayar [ZN06] substituted a controllable aperture for the lens to construct a lensless camera. The apertures in those cameras did not comprise of pinholes, so their imaging principles are different. Probably the first CA based lensless camera was developed for remote visible or infrared imaging by Slinger and Ridley et al. [SEG∗ 07, RdVP∗ 09] in 2007. Their CA contains lots of tiny pinholes, randomly distributing on the mask. They made their CA using specific materials and manufacturing techniques. Another implementation of lensless camera was conducted by Chi and George in 2011 [CG11]. They employed a spatial light modulator (SLM) screen to construct a pattern of uniformly redundant array (URA) [FC78] for their CA. URA is able to produce an ideal delta function by convoluting its conjugate function so that the reconstruc-.

(2) Z. Wang & I. Lee / Random Sparse Coded Aperture for Lensless Imaging. tion is perfectly solved. Both the above two kinds of CAs consist of dense small pinholes. So, the optical effects occur. Slinger and Ridley et al. even leveraged the diffraction to obtain a fine resolution image, but they had to attain the accurate diffraction pattern by extra physical test. Chi and George reduced some of the diffraction noise via using an iterative phase retrieval method at decoding, and the image quality was trade off. Recently, a time varied sparse pinhole array was proposed by Schwarz et al. [SSZ14] for acquiring high-resolution images. There are only several larger pinholes sparsely distributed on the CA so that each pinhole can image independently with little interference. The diffraction pattern for single pinhole imaging system can be analyzed theoretically. However, the optical transfer function (OTF) for such a CA involves lots of zero-value spectrums so that the target image cannot be ideally reconstructed with common linear methods. So, Schwarz et al. acquired the target scene several times using different CA patterns. The sum of the resulting OTFs preserves sufficient frequencies to extract a perfect reconstruction. In order to change the CA pattern conveniently, they adopted a DLP matrix.. Coded Aperture. Object. A1. Table-I Parameters for our prototype system Aperture size 7.68×7.68mm Image distance (f ) 40mm Object distance (u) ≥ 5.2m Square pinhole size (d) 0.24mm Pixel pitch (w) 30µm Yellow-green Wavelength (λ) 550nm Sensor array (pixels) 720 × 480. P1. s. O. p. A2 u. P2 f. l. io. Tr. ia. l. Figure 1: Sketch of the optical imaging system.. ia. de sk. sk PD. F. St. ud. io. PD F. Tr 2. Principles of our Imaging System. d. de. Motivated by the above novel cameras, in this paper we propose a random sparse CA for lensless imaging. A few square pinholes are sparsely scattered on the mask. The number of pinholes is adjustable for specific applications. But, the separations between adjacent pinholes are as big as possible to reduce the interference noise. If the pinholes are dispersed in a random sense, the recorded image is encrypted. Similarly, the OTF for the sparse CA involves many zero entries so its inverse produces many spikes, leading to small noise introducing a big effect. Conventional linear inversion methods cannot deal with such a problem. Hence, we employ a reselection scheme. The recorded samples are transformed into Fourier domain. Partial Fourier samples are then selected by following the principle of compressed sensing (CS) [CRT06a,CRT06b]. Meanwhile, those samples corresponding to the zero (or near zero) entries in OTF are discarded. Although the selected samples are incomplete, the sparse recovery algorithms can be manipulated to recover a good approximation. The hypothetical application for the proposed camera is outdoor surveillance. The sunlight is strong so a few pinholes can intake sufficient photons at short exposure time. In addition, a random distribution of pinholes may provide an encryption at sampling.. Sensor Array. St ud. 50. Figure 2: (a) random sparse CA; (b) the PSF for the basic pattern; (c) diffraction pattern. The optical imaging system is depicted in Figure 1. Besides, the parameters of the imaging system are listed in Table-I for the convenience of following explanation. The optical setup is composed of a CA and a sensor array. There are many pinholes on the CA, but each pinhole can perform independently as a single pinhole imaging model. A good virtue for pinhole optics is its linearity. For the simplicity of illustrating such a feature, we assume that there are only two identical pinholes in the imaging system. Let O denote the object, H1 and H2 denote the point spread funcN tions (PSF) of two pinholes A1 and A2 , and P1 = O H1 , N N P2 = O H2 denote the two corresponding images where denotes the convolution, then the recorded picture P = P1 + P2 = O. O. O. H1 + O. H2 = O. O. (H1 + H2 ). (1). Furthermore, P1 and P2 are the same except the position. The displacement of P1 and P2 in one dimension is calculated as p = s + f s/u.. (2). When the object distance is unknown, the displacement will be difficult to figure out. In order to make the second term negligible, we set u large enough so that f s/u < w where w is one pixel width (the smallest unit of the sensor array). Then, c The Eurographics Association 2014..

(3) 51. Z. Wang & I. Lee / Random Sparse Coded Aperture for Lensless Imaging. In an analogous manner, we can prove that objects off the optical axis follow the same conclusion as well. Then, the resulting imaging system becomes a linear shift invariant (LSI) system. A significant benefit brought by the LSI principle is that the imaging model is determined once the CA is designed. Let {xi , yi }m i=1 denote the positions of m pinholes on the CA, then the final recorded picture can be expressed as m. P=. ∑O. O. H(xi , yi ).. (4). i=1. And the entire PSF for the CA is m. A=. ∑ H(xi , yi ).. (5). i=1. It is a sum of many pinhole PSFs with different phase. Although the object distance is constrained by the LSI principle, it can still be adjusted by changing the image distance. In order to ensure each pinhole image to fall at the boundary of a pixel, the pinhole is square and its size is an integer multiple of the pixel size.. 3. Decoding. Once the above requirements are satisfied, the pinholes can be arranged freely. A randomly distributed CA offers an additional feature of encryption at sampling. The recorded picture by a CA camera is a mixture of many pinhole images. It cannot present a clear content before reconstruction. The reconstruction depends on the PSF, and the PSF is determined by the CA pattern. Therefore, a random CA can encrypt the target image at the sampling stage, which is impossible for lens based cameras. The PSF is vital for CA reconstruction. Thanks to the linearity of pinhole optics, the PSF for the proposed CA can be calculated based on geometrical optics. Our prototype CA is c The Eurographics Association 2014.. St F PD sk de. where λ is the target wavelength [You71]. Thus, the side length of our square pinhole selects 0.24mm after considering both (6) and the alignment of pinholes and pixels. On the other hand, in order to reduce the interference noise, each row or column contains a few separate pinholes, including no pinhole. Such an arrangement also facilitates the fabrication.. ud. io. For imaging at visible wavelength without lens, major challenges come from optical diffraction and interference. A single pinhole camera usually adopts a compromising pinhole size so that the diffraction mostly falls within the scope of geometrical imaging. Many works have shown that a pinhole camera can capture a good quality image as well. The optimum diameter for a circle pinhole camera is p d = 1.56 λ f (6). If one pinhole is regarded as a basic unit, the imaging system suffers from a coarse angular resolution of d/ f , which is around 6mrad for our prototype camera. So, we have to investigate the PSF within the geometrical shadow of one pinhole. It is improper to regard the geometrical shadow as a uniform square since the diffraction cannot be disregarded. Meanwhile, the diffraction pattern for a single square aperture can be analyzed theoretically [Mie98]. We utilize both theoretical analysis and experimental test, and then synthesize a diffraction pattern as Figure 2(c). Each pixel on the sensor array has a width of 30µm, so each pinhole square is divided into 8×8 grids. The above PSF is then extended 8 times in each dimension by replacing the ‘1’s with the diffraction pattern and stuffing the rest with ‘0’. And the angular resolution is improved to 0.75mrad.. l. (3). ia. P2 (x, y) = P1 (x + s, y + s).. shown in Figure 2(a). As done in [FC78], a cyclic version is employed to make each point source contribute a complete cycle on the recorded picture. The middle part outlined by the red square presents a basic 16×16 pattern, and the whole CA is a cyclic 32×32 pattern. At first, the PSF for single pinhole is assumed to be a Kronecker delta function. Each d×d white square means a pinhole, which corresponds to ‘1’s in the PSF. The black part means light proof, and similarly each d×d black square corresponds to ‘0’s in the PSF. The phase displacement of two ‘1’s equals the separation of their pinholes. Then, the PSF for the basic pattern is shown in Figure 2(b). The whole PSF is too big to show, but it is easy to figure out in a similar way.. Tr. the image translation of a point source passing through any two pinholes is equal to the separation of the two pinholes in each dimension. In other words,. (a). (b). Figure 3: (a)plot for. 1 ; |F (A)|. (b)Reselection (26 lines). The CA imaging can be formulated in convolution as O. (7). F(P) = F(O). ∗ F(A) + N. (8). P=O. A + N,. or in Fourier transform as. where A is the PSF for the CA, O the target image, P the recorded picture, N the noise, N the noise in Fourier form, F the Fourier transform, F(A) the OTF, and (.∗) the elementwise multiplication. Conventional decoding algorithms are mostly based on deconvolution or linear inversion. For URA,.

(4) 52. Z. Wang & I. Lee / Random Sparse Coded Aperture for Lensless Imaging. a conjugate function G can be constructed to satisfy that N N A G is an ideal delta function but N G is closed to zero, which has been applied in [CG11]. When F(A) does not contain zero entries, general inversion of F(A) can be used to reverse (8) to retrieve O, e.g., in [SSZ14]. In [SEG∗ 07], a regularization parameter is added to F(A) so that it becomes invertible, then the Tikhonov regularization method is adopted. For our sparse CA, it is difficult to find an ideal conjugate function as URA. Besides, the F(A) contains lots of zeroes (see red points in Figure 3(a)), so that the noise corresponding to the zero entries produces a bigger impact on the reconstruction than others. Both linear inversion and Tikhonov regularization do not differentiate them. The recent CS theory proposed a new non-linear way to solve a perfect reconstruction from incomplete noisy Fourier samples, which has been successfully applied to medical imaging [CRT06a]. In brief, CS tackles the insufficiency of samples by taking advantage of the sparsity characteristic of image signals in specific domains. Likewise, some of the Fourier samples are corrupted in our imaging system because they involve a multiplicative of zero. So, a reselection scheme is introduced to exclude those corrupted samples at decoding. The rest samples are incomplete. However, if the selection follows the CS principle, the sparse recovery algorithms can be leveraged for reconstruction. It is well known that the low frequency parts of an image in Fourier domain are dominating and important. The zerofrequency component is shifted to the center of spectrum in the course of reconstruction. In a similar way to [CRT06a], a few lines passing through the central point are selected (e.g., Figure 3(b)). The Fourier samples on these lines are collected, but among them those corresponding to the zerovalue (or close to zero) spectrums in the OTF are discarded. Suppose L denote the selecting operator, then 1, |F (A)| ≥ δ; L(F(A)) = (9) 0, otherwise.. x, TV (x) := ∑. q. (xi+1, j − xi, j )2 + (xi, j+1 − xi, j )2 .. ij. Another benefit of the proposed method is that the demand on memory could decrease since the selected Fourier samples are reduced, which is useful for large-scale image processing. Besides, the proposed method can be applied to image deblurring for lens based cameras with a coded aperture [LFDF07] as well. 4. Experimental Results. Figure 4: The arrangement of CA and sensor array. δ is the threshold of selection. ‘1’ means selected, and ‘0’ means discarded. If the selected index set is denoted by L as well, we can derive (10) from (8), L[F(P)] = L[F(O)]. ∗ L[F(A)] + L[N ].. (10). Tr. ia. l. Figure 5: Our prototype system: a)the entrance pupil; b) the CA mask; c) the prototype camera; d) the test site io. (11). where ε is the noise tolerance and TV (x) denotes the sum of the magnitudes of the discrete gradient of the digital image. F PD. There are different geometric arrangements of mask and sensors [CSDC∗ 87]. For a particular arrangement, there are usually two types of field of view (FOV): fully coded field de sk. de sk. PD. F. s.t. kF −1 {L[F(x)]. ∗ L[F(A)]} − F −1 {L[F(P)]}k2 ≤ ε. St ud. io. Tr. TV (x). St ud. min. ia. l. (10) reveals that the noise which can introduce big effect are excluded at decoding. Although (10) is ill-posed, the total variation (TV) minimization with quadratic constraints in [CRT06a] can be used for its reconstruction by solving. c The Eurographics Association 2014..

(5) Z. Wang & I. Lee / Random Sparse Coded Aperture for Lensless Imaging. 53. Figure 6: The raw recorded pictures on the Canon sensor array (720 × 480 pixels). of view (FCFV) and partially coded field of view (PCFV). The FCFV defines a scope in which all the light sources are completely modulated by the mask; the PCFV defines a scope where the light sources are partially coded by the mask. Light sources from the PCFV are difficult to analyze due to the incomplete coding, so they should avoid reaching the sensors. We select an arrangement as Figure 4. The mask comprises a 2r-by-2s cyclic version of the basic pattern where r and s mean the width and height of the target image, and the sensor array just below it contains r-by-s pixels. Such an arrangement employs a small sensor array but provides a wide FOV. When the incident lights only come from the FCFV, an r-by-s pixel image can be reconstructed from the recorded samples.. Tikhonov regularization and our reselection plus TV minimization method are recognizable and shown in Figure 7 column (c,d).. The mask is fabricated by a laser cutter with the highest resolution of 1200dpi, and then attached to the Canon EOS 600D camera base. The lens set is removed. A calibration is carried out for aligning the mask and the sensor array. An entrance pupil, modified from a Holga pinhole camera (see Figure 5(a, b)), is equipped to ensure that the target scene is just the FCFV. The prototype camera and test site are shown in Figure 5(c) and (d) separately. The parameters are listed in Table-I. In our prototype camera, the CA dimension is 7.68×7.68mm which is limited by the entrance pupil. Thus, the recorded pictures have a resolution of 128×128 pixels. The test results are shown in Figure 6 and 7. Six target images (Figure 7 column (a)) are displayed on the LCD screen, and then captured by our lensless camera in a dark environment to reduce the noise from the PCFV. Because the Canon sensor array is bigger than the required size, cyclic versions of samples are recorded on it (see Figure 6). The central ones corresponding to the basic pattern (Figure 7 column (b)) are sufficient and necessary for reconstruction. Different recovery methods are performed, but only the reconstructions by. 5. Conclusions. c The Eurographics Association 2014.. In the first row of Figure 7, the target scene is an ‘R’ letter. Several ‘R’s are clearly observed on the recorded picture, which verifies 1) that the interference noise is insignificant for our sparse CA and 2) that the recoded picture is a linear sum of multiple pinhole images as (4). The recorded pictures in other rows are nearly unreadable owing to overlapping, providing an initial encryption. But, the reconstructions are perceptible. In addition, the reconstructions by our method are clearer than those by Tikhonov regularization. So, the superiority of the proposed scheme is testified.. This paper developed a prototype lensless camera which records the target scene in a coded manner. The proposed sparse CA is able to reduce the interference noise, simplify the analysis of the imaging model, and facilitate the fabrication, but suffers from lots of zero-value spectrums. To mitigate the negative effect from the zero-value spectrums, we introduce a reselection scheme, and then leverage the well-developed sparse recovery algorithms to retrieve a better reconstruction than the classic Tikhonov regularization method. The proposed lensless camera is promising for preliminary outdoor surveillance because of its low cost and additional encryption at sampling. However, this paper is a proof-of-concept study on computational lensless imaging system. The result obtained so far is inferior to that of lens based cameras. It is certainly desirable to extend the work of this paper to more practical applications. Recent researches have shown that the image quality could be improved by using multiple masks [DF14]..

(6) 54. Z. Wang & I. Lee / Random Sparse Coded Aperture for Lensless Imaging. PD. F. St. ud. io. Tr. ia. l. [DF14] D E W EERT M. J., FARM B. P.: Lensless coded aperture imaging with separable doubly toeplitz masks. Proc. SPIE 9109 (2014), Q1–12. URL: http://dx.doi.org/10.1117/ 12.2050760, doi:10.1117/12.2050760. 5. de sk. de sk. PD. F. St. ud. io. Tr. ia. l. gamma-ray astronomy. Space Science Reviews 45, 3-4 (1987), 349–403. 4. (a). (b). l Tr ia l. de sk. PD. F. St. ud. io. [FC78] F ENIMORE E. E., C ANNON T. M.: Coded aperture imaging with uniformly redundant arrays. Appl. Opt. 17, 3 (1978), 337–347. URL: http://ao.osa.org/abstract.cfm? URI=ao-17-3-337, doi:10.1364/AO.17.000337. 1, 3. de sk. PD. F. St. ud. io. Tr ia. [FO12] F ORTUNATO H. E., O LIVEIRA M. M.: Coding depth through mask structure. Computer Graphics Forum 31, 2 (2012), 459–468. URL: http://dx. doi.org/10.1111/j.1467-8659.2012.03025.x, doi:10.1111/j.1467-8659.2012.03025.x. 6. de. sk. PD. F. St. ud. io. Tr. ia. l. [GIG11] G OTTESMAN S. R., I SSER A., G IGIOLI J R . G. W.: Adaptive coded aperture imaging: progress and potential future applications. Proc. SPIE 8165, 13 (2011), 1–9. URL: http: //dx.doi.org/10.1117/12.894913, doi:10.1117/ 12.894913. 1. de. sk. PD. F. St. ud. io. Tr. ia. l. [LFDF07] L EVIN A., F ERGUS R., D URAND F., F REEMAN W. T.: Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics (TOG) 26, 3 (2007), 70. 4. sk. PD. F. St. ud io. Tr. ia. l. [Mie98] M IELENZ K. D.: Algorithms for fresnel diffraction at rectangular and circular apertures. Journal of Research-National Institute of Standards and Technology 103 (1998), 497–510. 3. de. de. sk. PD. F. St. ud io. Tr. ia. l. de. sk. PD. F. St. ud. io. Tr. ia. l. de. sk. PD. F. St. ud. io. Tr. ia. l. de sk. PD. F. St. ud. io. Tr ia. l. de sk. PD. F. St. ud. io. Tr ia. l. [Dic68] D ICKE R.: Scatter-hole cameras for x-rays and gamma rays. The Astrophysical Journal 153 (1968), L101. 1. (c). (d). Figure 7: Column (a) ground truth; (b) recorded pictures by CA lensless camera; (c) decoding by Tikhonov regularization; (d) decoding by reselection + TV minimization.. Besides, the OTF for the current mask contains lots of zerovalue spectrums, which corrupts many samples. Adopting different pinhole sizes [FO12] may be able to reduce them and enhance the reconstruction quality. Those techniques encourage us to conduct further research along this direction. References [CG11] C HI W., G EORGE N.: Optical imaging with phase-coded aperture. Opt. Express 19, 5 (2011), 4294–4300. 1, 4 [CRT06a] C ANDES E., ROMBERG J., TAO T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 2 (2006), 489–509. 2, 4 [CRT06b] C ANDES E. J., ROMBERG J. K., TAO T.: Stable signal recovery from incomplete and inaccurate measurements. Communications on pure and applied mathematics 59, 8 (2006), 1207–1223. 2 [CSDC∗ 87] TALUCCI. [Mie99] M IELENZ K. D.: On the diffraction limit for lensless imaging. Journal of Research-National Institute of Standards and Technology 104, 5 (1999), 479–486. 1 [PGAB03] P OTULURI P., G OPINATHAN U., A DLEMAN J., B RADY D.: Lensless sensor system using a reference structure. Opt. Express 11, 8 (2003), 965–974. 1 [RdVP∗ 09] R IDLEY K. D., DE V ILLIERS G. D., PAYNE D. A., W ILSON R. A., S LINGER C. W.: Visible band lens-free imaging using coded aperture techniques. Proc. SPIE 7468, 9 (2009), 1– 10. URL: http://dx.doi.org/10.1117/12.828580, doi:10.1117/12.828580. 1 [SEG∗ 07] S LINGER C., E ISMANN M., G ORDON N., L EWIS K., M C D ONALD G., M C N IE M., PAYNE D., R IDLEY K., S TRENS M., D E V ILLIERS G., W ILSON R.: An investigation of the potential for the use of a high resolution adaptive coded aperture system in the mid-wave infrared. Proc. SPIE 6714, 8 (2007), 1– 12. URL: http://dx.doi.org/10.1117/12.736071, doi:10.1117/12.736071. 1, 4 [SSZ14] S CHWARZ A., S HEMER A., Z ALEVSKY Z.: High resolution and energetically efficient lensless imaging system based upon time varied pinholes array. Proc. SPIE 8979 (2014), G1– 13. URL: http://dx.doi.org/10.1117/12.2037186, doi:10.1117/12.2037186. 2, 4 [You71] YOUNG M.: Pinhole optics. Appl. Opt. 10, 12 (1971), 2763–2767. URL: http://ao.osa.org/ abstract.cfm?URI=ao-10-12-2763, doi:10.1364/ AO.10.002763. 3 [ZN06] Z OMET A., NAYAR S.: Lensless imaging with a controllable aperture. In CVPR’06 (2006), vol. 1, pp. 339–346. doi:10.1109/CVPR.2006.175. 1. C AROLI E., S TEPHEN J., D I C OCCO G., NA L., S PIZZICHINO A.: Coded aperture imaging in x-and c The Eurographics Association 2014..

(7)

No results found