An Adaptive Deghosting Method in Neural Network-Based ... - MDPI

4 downloads 0 Views 5MB Size Report
Jan 13, 2018 - to compensate for the deghosting failure of Hardie et al. ...... Torres, S.N.; San Martin, C.; Sbarbaro, D.; Pezoa, J.E. A neural network for ...
sensors Article

An Adaptive Deghosting Method in Neural Network-Based Infrared Detectors Nonuniformity Correction Yiyang Li, Weiqi Jin *, Jin Zhu, Xu Zhang and Shuo Li School of Optoelectronics, Beijing Institute of Technology, Key Laboratory of Photo-electronic Imaging Technology and System, Ministry of Education of China, Beijing 100081, China; [email protected] (Y.L.); [email protected] (J.Z.); [email protected] (X.Z.); [email protected] (S.L.) * Correspondence: [email protected]; Tel.: +86-010-68914697 Received: 22 October 2017; Accepted: 10 January 2018; Published: 13 January 2018

Abstract: The problems of the neural network-based nonuniformity correction algorithm for infrared focal plane arrays mainly concern slow convergence speed and ghosting artifacts. In general, the more stringent the inhibition of ghosting, the slower the convergence speed. The factors that affect these two problems are the estimated desired image and the learning rate. In this paper, we propose a learning rate rule that combines adaptive threshold edge detection and a temporal gate. Through the noise estimation algorithm, the adaptive spatial threshold is related to the residual nonuniformity noise in the corrected image. The proposed learning rate is used to effectively and stably suppress ghosting artifacts without slowing down the convergence speed. The performance of the proposed technique was thoroughly studied with infrared image sequences with both simulated nonuniformity and real nonuniformity. The results show that the deghosting performance of the proposed method is superior to that of other neural network-based nonuniformity correction algorithms and that the convergence speed is equivalent to the tested deghosting methods. Keywords: fixed pattern noise; nonuniformity correction; noise estimation; neural network

1. Introduction Imaging systems based on infrared focal plane arrays (IRFPAs) have been widely used in military and civilian applications. However, as they are limited by the manufacturing quality and the level of the processing technology, the responsiveness of the individual photodetectors in the focal plane array will vary from detector to detector. This is known as the nonuniformity of an IRFPA and is also called fixed pattern noise (FPN). Nonuniformity severely corrupts infrared images and must be corrected in practical applications. To this end, many nonuniformity correction (NUC) techniques have been proposed to compensate for FPN. These NUC techniques are categorized into two classes, namely, calibration-based and scene-based techniques. Calibration-based nonuniformity correction (CBNUC) algorithms are generally used to reduce FPN, with the simplest and most used one being the two-point calibration [1]. However, as FPN drifts temporally, CBNUC algorithms must be repeated by inserting a uniform radiation source into the view, thus affecting the imaging in use [2]. To address this disadvantage, scene-based NUC (SBNUC) algorithms were proposed. SBNUC algorithms use scenes to fix the FPN time fluctuation without a uniform radiation source. These approaches are divided into neural network (NN) algorithms [3], temporal high-pass algorithms [4,5], constant–statistical algorithms [6], and registration-based algorithms [7,8]. Among them, the NN-based SBNUC algorithms have received extensive attention due to the few requirements of computing resources and storage spaces, and this type of correction is very likely to replace CBNUC in the infrared imager. Sensors 2018, 18, 211; doi:10.3390/s18010211

www.mdpi.com/journal/sensors

Sensors 2018, 18, 211

2 of 15

Sensors 2018, 18, 211

2 of 15

The NN-based SBNUC algorithm was first proposed by Scribner in [3]. This algorithm estimates The NN-based SBNUC algorithm was first proposed by Scribner in [3]. This algorithm estimates the needed gain and offset parameters on a frame-by-frame basis by means of the least mean square the needed gain and offset parameters on a frame-by-frame basis by means of the least mean square (LMS) method. However, this method does not handle the problem of ghosting artifacts, which is (LMS) method. However, this method does not handle the problem of ghosting artifacts, which is a a collateral effect caused byby thethe NUC Asshown shown in Figure 1, first the row firstisrow is frame collateral effect caused NUCon onthe the scene. scene. As in Figure 1, the frame 130 and130 and thethe scene has remained stationary for 130 frames. By using Scribner et al.’s algorithm to reduce scene has remained stationary for 130 frames. By using Scribner et al.’s algorithm to reduce the the FPN, thethe textures become blurred. ghosting artifacts when hand FPN, texturesofofthe the building building become blurred. TheThe ghosting artifacts appearappear when the handthe blocks the field in in frame 132.132. Such artifacts have have become an obstacle to the application of the blocks the fieldof ofview view(FOV) (FOV) frame Such artifacts become an obstacle to the application algorithm. of the algorithm.

Frame 130

Frame 132

1. An example infrared sequence processed Scribner’s method. method. The is is thethe Figure Figure 1. An example infrared sequence processed byby Scribner’s Thefirst firstcolumn column the second column is processed the processed image. originaloriginal image,image, the second column is the image.

In order to reduce ghosting artifacts, Vera et al. improved Scribner et al.’s retina-like neural

In network order toapproach reduce method ghosting artifacts, Vera learning et al. improved Scribner et al.’sout retina-like neural using an adaptive rate [9]. Rossi et al. pointed that ghosting network approach method using learning rate [9]. Rossiofetclassical al. pointed out that ghosting artifacts are generated mainlyan inadaptive strong edge areas. Hence, instead low-pass filters, they artifactsintroduced are generated mainly in strong edge areas. Hence, instead of classical low-pass filters, they a bilateral filter which preserves edges [10]. Zhang et al. introduced an edge detection mechanism in thefilter calibration [11].edges The binarized edge detection results arean used to detection control introduced a bilateral whichprocess preserves [10]. Zhang et al. introduced edge the learning As withprocess a bilateral filter, thebinarized threshold edge used to distinguish edgesare andused noisetoincontrol this mechanism in the rate. calibration [11]. The detection results algorithm difficult select. These consistently highlight residual nonuniformity the learning rate.isAs with atobilateral filter,two the methods threshold used to distinguish edges and noise in this noise, which istoconsidered to be false edges.consistently Torres et al.highlight improvedresidual the learning rate function, algorithm is difficult select. These two methods nonuniformity noise, following temporal variations in the operation point of each detector [12]. However, the frame motion which is considered to be false edges. Torres et al. improved the learning rate function, following detection is based on phase correlation. Increasing the step size in the local motion blurs stationary temporal variations in the operation point of each detector [12]. However, the frame motion detection areas. Hardie et al. proposed a deghosting method based on a temporal gate [13]. By gating the is basedupdate on phase correlation. Increasing the stepavoids size in the local motion stationary Hardie of the NUC parameters, this method the accumulation of blurs incident radiationareas. estimation et al. proposed a deghosting method based on a temporal gate [13]. By gating the update of the NUC errors during long motion pauses. Up to now, Hardie et al.’s deghosting strategy has been widely parameters, this Nonetheless, method avoids the accumulation of such incident radiation estimation errors during long used [14]. for some common scenes, as when the imager is observing a scene with Hardie fails. Fanstrategy et al. proposed a combined temporal spatial motion jitter, pauses. Up et to al.’s now,deghosting Hardie etalgorithm al.’s deghosting has been widely used [14]. and Nonetheless, deghosting compensate deghosting failure of Hardiewith et al.’s algorithm for some commonmethod scenes,tosuch as whenfor thethe imager is observing a scene jitter, Hardie [15]. et al.’s However, this method’s spatial limits with a hard threshold are similar to those of the previous deghosting algorithm fails. Fan et al. proposed a combined temporal and spatial deghosting method method. In where the nonuniformity noise is the constant spatial threshold it to compensate forcases the deghosting failure of Hardie etserious, al.’s algorithm [15]. However, thismakes method’s difficult to take into account noise removal and the ghosting inhibition. spatial limits with a hard threshold are similar to those of the previous method. In cases where the This paper proposes a combined adaptive spatial and temporal deghosting (AST) method. We nonuniformity noise is serious, the constant spatial threshold makes it difficult to take into account use adaptive threshold edge detection so that the rigidity of the spatial limitation can be adjusted noise removal and inhibition. noise. In other words, when the noise is serious, a large according to the the ghosting residual nonuniformity This paper proposes a combined adaptive ability. spatialInand deghosting (AST)will method. threshold is used to enhance the de-noising thetemporal opposite case, the threshold becomeWe use adaptive detection so that the rigidity of the spatial limitation can be adjusted smallerthreshold to suppressedge the ghosting artifacts. The rest of the paper is organized noise. as follows. Related work iswhen introduced in Section 2. Section 3 according to the residual nonuniformity In other words, the noise is serious, a large is dedicated to a detailed description of the proposed algorithm, and also highlights the threshold is used to enhance the de-noising ability. In the opposite case, the threshold will become smaller to suppress the ghosting artifacts. The rest of the paper is organized as follows. Related work is introduced in Section 2. Section 3 is dedicated to a detailed description of the proposed algorithm, and also highlights the improvements

Sensors 2018, 18, 211

3 of 15

made. In Section 4, the NUC experimental results are discussed. In Section 5, the efficiency of the algorithm is analyzed. The conclusion is presented in Section 6. 2. Neural Network-Based Nonuniformity Correction 2.1. Nonuniformity Observation Model We begin by assuming that each infrared detector is characterized by a linear model, without considering the additive temporal noise. For the (i,j)th detector in the IRFPA, the measured readout signal yi,j at a given time n can therefore be expressed as: yi,j (n) = Gi,j (n) xi,j (n) + Oi,j (n),

(1)

where Gi,j (n) and Oi,j (n) are the gain and the offset coefficients of the (i,j)th detector, and xi,j (n) is the real incident infrared photon flux collected by the respective detector. The main purpose of the NUC method is to estimate the value of xi,j (n) accurately. Using Equation (1), the value of xi,j (n) can be obtained by the following equation: xi,j (n) = wi,j (n)yi,j (n) + bi,j (n),

(2)

where the new parameters wi,j (n) and bi,j (n) are related to the gain and offset coefficients, respectively, of each detector as follows: wi,j (n) =

1 , Gi,j (n)

bi,j (n) =

Oi,j (n) , Gi,j (n)

(3)

2.2. Deghosting Methods The key of the SBNUC algorithm is to obtain multiple sets of incident values and their corresponding response readout signals to calculate the gain and offset correction parameters. In the NN-based SBNUC algorithm, the estimated incident radiation value is calculated as:  Xˆ i,j (n) = κ xi,j (n) ,

(4)

where the spatial low-pass filter κ constitutes the hidden layer. Commonly used filters are the four-neighborhood filter, the Gaussian filter, the bilateral filter, the guide filter, and the non-local average filter [16]. These filters are used to force the estimated desired image closer to the real incident image. However, any estimation of the incident radiation cannot be accurate, and, even worse, this estimation is, in most cases, wrong. Therefore, the NN algorithms do not directly calculate the gain and offset matrix according to the estimated value Xˆ and the readout signal y. However, according to the LMS algorithm, the gain and offset correction parameters are constantly iterated by gradient descent through a large amount of radiation estimation values Xˆ and the gray values y. This also improves the dynamic range of correction. Therefore, the final error function is given by: 2 Ei,j (n) = wi,j (n)yi,j (n) + bi,j (n) − Xˆ i,j (n) , (5) Using the LMS algorithm, the correction parameters wi,j and bi,j are recursively and smoothly updated with a portion of each respective error gradient, as follows:  wi,j (n + 1) = wi,j (n) − ε i,j (n) xi,j (n) − Xˆ i,j (n) yi,j (n),

(6)

 bi,j (n + 1) = bi,j (n) − ε i,j (n) xi,j (n) − Xˆ i,j (n) ,

(7)

where ε is the step size known as learning rate, which controls the convergence speed of the algorithm. Higher values of ε can provide a faster convergence, but the results are unreliable. Conversely, smaller

Sensors 2018, 18, 211

4 of 15

values of ε can produce more reliable results, but with a lower rate of convergence. A suitable value for learning rate ε can better mitigate ghosting artifacts. Vera et al. use local variance to control ε. A large local variance means that the incident radiation estimation error is large, so a small ε is used. A large ε is used as a contrast. This method reduces ghosting artifacts to a certain extent and actually increases convergence speed. However, because the learning rate is never actually set to zero with the adaptive LMS algorithm, this method does not eliminate burn-in ghosting for long motion pauses. The ghosting artifacts occur when motion resumes. Hardie et al. use a temporal threshold to gate the step size. With a stationary scene, the pixel gray value does not change significantly from one frame to the next. Therefore, a threshold can be set to stop NUC parameters from being updated. The gating method can be represented by:

ε i,j (n) =

 

k alr 1+σy (n) i,j

Zi,j (n + 1) =

,

(8)

else

 0 (

Xˆ i,j (n) − Zi,j (n) > T

Xˆ i,j (n) Zi,j (n)

Xˆ i,j (n) − Zi,j (n) > T , else

(9)

where σy is the local spatial standard deviation of the input image y. The parameter kalr is the maximum step size. T is the temporal threshold, and Z is used to detect the temporal motion. The scene information has not been taken into account in Hardie et al.’s method. When the edges appear and disappear on the same pixel, or when the image scene shakes, ghosting artifacts can still be generated. Fan et al. proposed a combined temporal and spatial deghosting method to solve the problem.  k alr  Xˆ i,j (n) − Zi,j (n) > T AND Si,j (n) = 1 1+σy (n) i,j , (10) ε i,j (n) =  0 else ( Xˆ i,j (n) Xˆ i,j (n) − Zi,j (n) > T AND Si,j (n) = 1 , (11) Zi,j (n + 1) = else Zi,j (n) ( 1 ∀( p, q) ∈ Ω, Xˆ p,q (n) − Xˆ i,j (n) ≤ Ts Si,j (n) = , (12) 0 else where Si,j (n) is the spatial local correlation. Si,j (n) = 1 means that the pixel (i,j) may be in the uniform area. Ω is the local window. One can see here that the correction parameters can only be updated if a pixel satisfies both the temporal motion detection and the spatial correlation detection. This method can, for some cases, effectively compensate for the deghosting failure of Hardie et al.’s algorithm. However, the spatial threshold of the algorithm is constant. When the noise is serious, the calculated local spatial correlation is large, and it is necessary to use a large spatial threshold to ensure the removal of nonuniformity noise. With the gradual lowering of nonuniformity noise through the iteration correction, the local spatial correlation of the image is mainly affected by the scene. The previously determined large spatial threshold cannot continue to limit the correction parameters that are updating at the edges, so ghosting artifacts can also be generated. Therefore, a more reasonable threshold should be adaptively adjusted according to residual nonuniformity noise. 3. Proposed Deghosting Method 3.1. Combined Temporal Gate and Edge Detection Generally, a pixel located in the flat area has a low estimated error of incident radiation, so it should not be limited for correction. In contrast, the NUC parameter updating in the area with scene textures should be stopped, as the error of the incident radiation estimation would be large [11,12,15].

Sensors 2018, 18, 211

5 of 15

Sensors 2018, 18, 211

5 of 15

Sensors 2018, 18, 211 of 15 Generally, a pixel located in the flat area has a low estimated error of incident radiation, so5 it

should not bealimited for correction. In area contrast, NUC parameter updating in the area withsoscene Generally, pixel located in the flat has athe low estimated error of incident radiation, it textures should be stopped, as the In error of the the incident estimation large [11,12,15]. should not be limited for correction. contrast, NUCradiation parameter updatingwould in the be area with scene As the scene textures are often directional, we Sobel as shown in 2, to calculate As the scenebe textures are directional, wethe used theoperator, Sobel operator, as shown in[11,12,15]. Figure 2, to textures should stopped, asoften the error of theused incident radiation estimation would beFigure large the image gradient in horizontal and vertical directions in order to locate the scene information imageare gradient in horizontal directions to inlocate the2, scene As calculate the scenethe textures often directional, we and usedvertical the Sobel operator,inasorder shown Figure toareas information areas in the image. in thecalculate image. the image gradient in horizontal and vertical directions in order to locate the scene information areas in the image. 2 1 1 0  0 1 2 1   0   0  10 20 1    1 2 1

1 2   1

 1 0 1  2 0 2   0 1   10 02 1  0 1



90° 0°2. Sobel templates 90° Figure two directions. Figure 2. Sobel templates in in two directions. Figure 2. Sobel templates in two directions.

Here, the object of the edge detection is the low-noise corrected image. Subsequently, we define Here, the object of the edge detection is the low-noise corrected image. Subsequently, we define the gradient imageofasthe follows: Here, the object edge detection is the low-noise corrected image. Subsequently, we define the gradient image as follows: the gradient image as follows: (13)(13) 0  edgexi,jedge = x|i , ej 0| e+ |ee9090 |,, edgexi , j  e0  e90 , (13) In the obtained gradient image, pixels with larger grayvalues values exhibit exhibit aanotably error In the obtained gradient image, the the pixels with larger gray notablyhigher higher error in estimating the incident radiation in the hidden layer, and vice versa. The purpose of edge detection In thethe obtained gradient image, the pixels with larger a notablyof higher in estimating incident radiation in the hidden layer, andgray vicevalues versa.exhibit The purpose edgeerror detection in our algorithm is therefore toinextract the large that are the scene estimating the incident the hidden layer,gradients and vice versa. Themainly purposecaused of edgeby detection in ourinalgorithm is therefore radiation to extract the large gradients that are mainly caused by the scene textures. textures. The spatial threshold S is used to binarize the edge x to obtain the large gradient in our algorithm is therefore to extract the large gradients that are mainly caused by the pixel sceneset The spatial threshold S is used to binarize the edgex to obtain the large gradient pixel set denoted as Φ. denoted as spatial Φ. Compared with Fan et al.’s spatial correlation thelarge Sobelgradient operator, based textures. The threshold S is used to binarize the edgex tooperator, obtain the pixel set on Compared with Fan et al.’s spatial correlation operator, the Sobel operator, based on the first order the first gradient,with is lessFan sensitive the shot noise, as operator, shown in the Figure 3b,c. In Figure 3, theontwo denoted asorder Φ. Compared et al.’sto spatial correlation Sobel operator, based gradient, is less sensitive to the shot noise, as shown in Figure 3b,c. In Figure 3, the two operators used the same (3shot × 3)noise, and threshold (20). Here, a more intelligent edge theoperators first orderused gradient, is lesswindow sensitivesize to the as shown in Figure 3b,c. In Figure 3, the two the same window size (3 × 3) and threshold (20). Here, a more intelligent edge detection operator detection operator such window as Cannysize was(3 not× used because Sobel(20). preserves most intelligent primitive gradient operators used the same 3) and threshold Here, the a more edge the image and has a relatively smaller computational complexity. such detection as information Cannyoperator wasofnot used Sobel preserves the most primitive information such as because Canny was not used because Sobel preserves thegradient most primitive gradientof the information the imagesmaller and hascomputational a relatively smaller computational complexity. image and has aof relatively complexity.

(a) (b) (c) (a) (b) (c) Figure 3. Comparison of image edge extraction: (a) original image; (b) spatial correlation by 3 × 3 window; and (c) Sobel operator. Figure 3. Comparison image edgeextraction: extraction: (a) (b)(b) spatial correlation by 3by × 33 × 3 Figure 3. Comparison of of image edge (a) original originalimage; image; spatial correlation window; (c) Sobel operator. window; and and (c) Sobel operator. In summary, the combined temporal gate and edge detection controlled learning rate can be expressed as: the combined temporal gate and edge detection controlled learning rate can be In summary, In summary, expressed as: the combined temporal gate and edge detection controlled learning rate can be  kalr Xˆ i , j  n   Zi , j  n   T AND xi , j  n   expressed as: 1k    ,  i , j  n  y n (14) alr    i, j  TTAND xi , j x n (  ZZi,ji , j(nn) >  k alr Xˆ i,jXˆ(i ,nj )n− AND /Φ i,j n ) ∈ 1   1 + σ ,  n    y ( n ) y n (14) 0 else  i,j  i, j   ε i,j (n) =i , j , (14)  00 else else 

Xˆ i , j  n   Zi , j  n   T (  Xˆ i , j  n  Zi , j  n  1ˆ ˆ (15) ˆ ˆ XX Z n(nn) XX − ZZii,j, j (nn) >T T , , i,ji(, jn)nelse Zi,j (Zni ,+ 1) =  i , ji,j (15) i, j   (15) , j  n  1    Z (n) else else  Zi , ji,j n  Consequently, the correction parameters can be updated only if a pixel responds to the temporal motion detection and is located in the non-edge region. The key to this method lies in how to obtain an intelligent threshold S that is associated with the nonuniformity noise gradient.

3.2. Adaptive Spatial Threshold Because of the influence of FPN, it is difficult to accurately diagnose whether the image gradient is caused by scene or noise. When the spatial threshold is too large, it is not possible to limit the

Sensors 2018, 18, 211

6 of 15

correction parameters that are updating in the edge area, and this may in turn cause ghosting artifacts. On the other hand, a threshold that is too small causes the presence of noise that cannot be removed. For this reason, we divided the correction process into the denoising stage and the deghosting stage. (1) During the denoising stage, the image gradient is greatly affected by the nonuniformity noise, and the main task of the algorithm is to remove such noise. At this point, the update of the correction parameters should be as unlimited as possible. A large adaptive spatial threshold should therefore be used; (2) After hundreds of iterations, when the noise has been gradually removed, the algorithm should focus on suppressing ghosting. Here, a smaller, adaptive spatial threshold should be used to conserve the detail gradients. A flag STA is used to represent the two states. STA = 1 denotes the denoising stage, otherwise it is the deghosting stage. Regarding the frame-by-frame iteration correction algorithm, the more noise the correction matrix contains, the smaller the residual nonuniformity in the corrected image. Without loss of generality, it is assumed that the gain nonuniformity is either a known constant given by the infrared imager manufacturer, or that it can be initially calculated by using a priori information on the particular infrared imager under study [2,17,18]. Thus, we only need to correct the offset nonuniformity that results from the temperature drift. In the denoising stage, the noise contained in the offset correction matrix is continuously increased. At the same time, the FPN in the corrected image gradually decreases. After multiple frame iterations, the noise in the correction matrix remains essentially unchanged, and the dynamic correction is temporarily completed and then enters the deghosting stage. Thus, the temporal variation of the first order gradient of the correction matrix can be used as the basis for the convergence of the correction. The Sobel operator is used to mask the correction matrix, and edgeb is obtained. The mean value of edgeb is denoted as edgeb . The larger edgeb is, the more noise is contained in the correction matrix, which means that less noise is included in the corrected image. The temporal variation of edgeb is represented as:     T ( n ) = max edge σi,j , edge , . . . , edge − min edge , edge , . . . , edge b(n) b ( n −1) b ( n − m +1) b(n) b ( n −1) b ( n − m +1) ,

(16)

T ( n ) is large, the gradient of the correction matrix where m is the temporal parameter. Usually, when σi,j changes greatly, which indicates that the correction is not convergent and that the residual noise of the corrected image is serious. On the contrary, when the correction process tends to be stable, the corrected image becomes noiseless. Thus, it is possible to distinguish the stages of denoising and deghosting.  1 σ T ( n ) > T 1 i,j T σi,j , (17) (n) = STA(n) = 0 σ T (n) ≤ T1 i,j

where T1 is the threshold. In the application, we selected T1 = 1 and m = 50 to achieve satisfactory results. According to a previous analysis, the design of the edge detection threshold is determined by the severity of nonuniformity noise. In the two correction stages that have been distinguished, the threshold has different strategies. Inspired by the successful application of noise estimation algorithms in the field of computer vision, we used some of the noise evaluation ideas found in the literature [19–22]. For images with offset noise, the degradation model can be represented as: edge gi,j = edge f i,j + edgeOi,j ,

(18)

where f is the noiseless image, and g is the corrupted image. Without loss of generality, we assume that images always contain some flat areas, such as roads, the sky, and so on. In these areas, there is often some noise distribution but almost no scene texture. Thus, Equation (18) can be rewritten as: edge gi,j = edgeOi,j ,

(19)

Sensors 2018, 18, 211

7 of 15

where f is the noiseless image, and g is the corrupted image. Without loss of generality, we assume that images always contain some flat areas, such as roads, the sky, and so on. In these areas, there is often some noise Sensors 2018, 18, 211 distribution but almost no scene texture. Thus, Equation (18) can be rewritten7as: of 15

edgegi , j  edgeOi , j ,

(19) That is, the gradient of nonuniformity noise is the same as that of the corrected image in the flat That is, the gradient of nonuniformity noise is the same as that of the corrected image in the flat area. The flat area in the image is located by calculating the local variance V of each W × W window; area. The flat area in the image is located by calculating the local variance V of each W × W window; the region that has the least local variance is the flat region. The formula is as follows: the region that has the least local variance is the flat region. The formula is as follows: ( )!

 

Np,)Nq  locmin min VV g ( N p,(Nq =)loc i, j W W i,j

i, j

 ,

W ×W gi,j

,

(20) (20)

The W × W area with (Np, Nq) at its the center is the flattest. The maximum gradient of this area The W × W area with (Np, Nq) at its the center is the flattest. The maximum gradient of this area can be used as the gradient level of noise, just as follows: can be used as the gradient level of noise, just as follows:





nug  maxn edgegi , j , i, j  o, (21) nu g = max edge gi,j , i, j ∈ Θ , (21)  where nug represents the noise gradient of the image g and represents the set of pixels in the where nug represents the noise gradient of the image g and Θ represents the set of pixels in the flattest flattest area. In this way, we have estimated the noise gradient using an image. However, due to the area. In this way, we have estimated the noise gradient using an image. However, due to the global global distribution of nonuniformity noise, the noise gradient calculated via the flattest region is distribution of nonuniformity noise, the noise gradient calculated via the flattest region is always too always too small. Therefore, in the denoising stage, in order to fully remove the FPN, an expansion small. Therefore, in the denoising stage, in order to fully remove the FPN, an expansion coefficient coefficient e is used to adjust the estimated noise gradient, and the result serves as the adaptive edge edetection is used tothreshold adjust thefor estimated noise and thestage, result the serves as the adaptive edge detection this stage. In gradient, the deghosting noise gradient estimated via the threshold for this stage. In the deghosting stage, the noise gradient estimated via the corrected image is corrected image is used directly as the threshold. As a result, the adaptive edge detection threshold used as the threshold. As a result, the adaptive edge detection threshold S can be represented as: S candirectly be represented as: (   n  x (n) e  enu· xnu SS(nn)=  nu x(n) nu   xn 

n  1 1 STASTA (n)= ,, STASTA (n) n= 0 0

(22) (22)

We ability, but We can can see see that that the the larger larger the the value value of of e, e, the the more more aggressive aggressive the the noise noise removal removal ability, but the the weaker the deghosting ability. We recommended that 2 ≤ e ≤ 10. weaker the deghosting ability. We recommended that 2 ≤ e ≤ 10. 3.3. 3.3. Algorithm Algorithm Summary Summary An An overview overview of of our our algorithm algorithm is is presented presented in in Figure Figure 4. 4. The The input input image image is is the the readout readout data data yyi,ji,j,, which is processed by the offset NUC parameter to get the corrected image x . The neighborhood which is processed by the offset NUC parameter to get the corrected image xi,ji,j. The neighborhood ˆ in order to calculate the error filter is applied to the corrected image xi,j , generating the target X filter is applied to the corrected image xi,j, generating the target Xˆ i,j in order to calculate the error i,j function Ei,j . Edge detection extracts the large gradient of xi,j , and is combined with temporal motion function Ei,j. Edge detection extracts the large gradient of xi,j, and is combined with temporal motion detection and local operation to control the step factor of the parameter updates. Following this, detection and local operation to control the step factor of the parameter updates. Following this, the the steepest descent method is used to obtain the NUC parameters for the next frame. Gradient steepest descent method is used to obtain the NUC parameters for the next frame. Gradient extraction extraction calculates the gradients of the offset NUC matrix to obtain edgeb , and then STA is obtained ̅̅̅̅̅̅b, and then STA is obtained through a calculates the gradients of the offset NUC matrix to obtain edge through a temporal calculation. The result of locating the flat area modulated by the STA is used as the temporal calculation. The result of locating the flat area modulated by the STA is used as the final final edge detection threshold. Comparing with the block schemes in [9,15], the adaptive threshold edge detection threshold. Comparing with the block schemes in [9,15], the adaptive threshold calculation composed of the above three modules is our algorithm’s main contribution. calculation composed of the above three modules is our algorithm’s main contribution. Gradient Extraction

b1,1

+ b1,2 Input Image

Corrected Image

+ bm,n

Temporal Calculation

Locating flat area

+ Local Operation

Neighborhood Filter Edge Detection

Error Parameters Update

+

-

Temporal Motion Detection

Figure 4. Scheme of proposed adaptive spatial and temporal deghosting–least mean square nonuniformity correction (AST–LMS NUC) algorithm.

Sensors 2018, 18, 211

8 of 15

4. Experimental Results The main goal of this section is to evaluate the efficacy of the proposed adaptive algorithm. The algorithm was tested using an image sequence with simulated nonuniformity and two real infrared image sequences. 4.1. Nonuniformity Correction with Simulated Data A clean 14 bit infrared video sequence was acquired by a 336 × 256 focal-plane array (FPA) uncooled imager (FLIR TAU2-336). The corrupted video sequences were obtained using a global synthetic offset with a zero-mean Gaussian distribution with a standard deviation of 30 and a vertical stripe offset with a standard deviation of 15. The entire acquisition was performed outdoors. Scenes typically known to easily cause ghosting artifacts were imaged to verify the ability of the adaptive threshold to remove noise and suppress ghosting. For the first 600 frames, the imager moved in a random and consistent manner. Then the imager jittered toward the buildings. The imager suddenly moved away from the FOV at frame 955. In total, there were 1000 frames. The metric used to measure the NUC performance was the peak signal-to-noise ratio (PSNR), which is widely used in the image processing literature to quantify the differences between two images, and it is defined as: s 2 1 (23) Ii,j − Iˆi,j RMSE = ∑ MN i,j  PSNR = 20 · log10

2c − 1 RMSE

 (24)

where c is the number of bits of image data (here c = 14). Ii,j is the clean raw image and Iˆi,j is the corrected image. In the experiment, we used Hardie et al.’s Gated–least mean square (Gated–LMS) algorithm [13], Fan et al.’s combined spatial and temporal deghosting–least mean square (ST–LMS) algorithm [15] and the algorithm proposed in this paper to calibrate the offset parameters of the image sequence. All of the LMS methods used the same kal = 0.8, T = 8 and a four neighborhood filter. The spatial correlation window used in ST–LMS was 7 × 7, and the spatial threshold Ts was set to 50 in order to ensure the removal of the noise. Our algorithm used W = 7 to calculate the threshold, and e was set to 8. Figure 5 shows some typical images in the sequence. The top row in Figure 5 is the 964th frame when the infrared imager was jittering against the outdoor scene. The 959th frame, after the FOV suddenly changed is shown in the bottom row in Figure 5. One can see that the FPN was almost removed after the processing of the three algorithms, but the Gated–LMS and ST–LMS have obvious ghosting artifacts, as indicated in the red boxes in Figure 5b,c. There were almost no ghosting artifacts in the image processed by our algorithm, as shown in Figure 5d. The visual effect of our algorithm is more satisfactory than the two compared algorithms, and this can be more comprehensively seen in Video S1. The PSNR curve is shown in Figure 6. As can be seen, in the first 480 frames, the PSNR curves of the three algorithms coincide with each other and rise rapidly due to the random motion of the imager. This shows that the adaptive threshold in our algorithm did not hinder the removal of nonuniformity noise and that the convergence speed was as fast as that of Gated–LMS’s. Subsequently, due to the unintentional jitter accompanying the movement of the imager, the edges appeared and disappeared at the same position. The PSNR curves of the Gated–LMS and ST–LMS show obvious downward spikes caused by ghosting artifacts in the image. The imager was jittered deliberately from frame 600 onwards. At this point, the PSNR curves of the Gated–LMS and ST–LMS remain in a downward trend, and there are multiple downward spikes. At this stage, although the spatial threshold of the ST–LMS failed to suppress most ghosting artifacts, it also stopped the iterative correction of some strong edges, so that the PSNR curve is slightly higher than that of the Gated–LMS. During the jitter

Sensors Sensors 2018, 2018, 18, 18, 211 211

99 of of 15 15

Sensors 2018, 18, 211

9 of 15

spatial threshold of the ST–LMS failed to suppress most ghosting artifacts, it also stopped the iterative correction of some strong edges, so that the PSNR curve is slightly higher than that of the Gated– process, the PSNR of our algorithm stays atofthe levelstays reached, which is obviously higher than LMS. During the jitter process, the PSNR ourhighest algorithm at the highest level reached, which that of the other two algorithms. is obviously higher than that of the other two algorithms.

(a)

(b)

(c)

(d)

Typical scenes in the the correction correction process: (a) (a) uncorrected uncorrected image; image; (b) (b) Gated–least Gated–least mean mean square square Figure 5. Typical (Gated-LMS); (c) and temporal deghosting–least meanmean square(ST-LMS); (d) our(d) image. (Gated-LMS); (c)combined combinedspatial spatial and temporal deghosting–least square(ST-LMS); our The topThe rowtop is the when the imager jitteredjittered towardtoward the outdoor scene. scene. The bottom row is image. row954th is theframe, 954th frame, when the imager the outdoor The bottom the 959th imager suddenly move. move. row is theframe 959th when framethe when the imager suddenly 70 70 68 68

Gated-LMS Gated-LMS ST-LMS ST-LMS Ours Ours

66 66

PSNR(dB)

64 64 62 62 60 60 58 58 56 56 54 54 52 52 00

100 100

200 200

300 300

400 400

500 500 Frames Frames

600 600

700 700

800 800

900 900

1000 1000

Figure 6. 6. PSNR PSNR versus versus frame frame number number for for the the tested tested algorithms algorithms using using simulated simulated nonuniformity nonuniformity data. data. Figure

In addition, as shown in Figure 7, we we analyzed analyzed the the frame-by-frame frame-by-frame variation of the correction ̅̅̅̅̅̅bb and matrix threshold S. The RMSE was used as an index of residual matrix gradient gradient edge the adaptive and the adaptive b nonuniformity RMSE values indicate lessless residual nonuniformity noise, and nonuniformity noise. noise.Specifically, Specifically,lower lower RMSE values indicate residual nonuniformity noise, vice versa. In Figure 7, in7,order to facilitate the observation of the RMSE curve, we we hadhad expanded its and vice versa. In Figure in order to facilitate the observation of the RMSE curve, expanded value by 10 The figure shows that in theinfirst due todue serious noise innoise the image, its value by times. 10 times. The figure shows that the200 firstframes, 200 frames, to serious in the there were always some correction parameters that were updating, regardless of whether the imager image, there were always some correction parameters that were updating, regardless of whether the was moving or stationary. Until frame the425, correction matrix matrix was close obtaining the noise imager was moving or stationary. Until425, frame the correction wasto close to obtaining the distribution of the of original image. Its gradient no longer increased significantly. At this point, the noise distribution the original image. Its gradient no longer increased significantly. At this point, corresponding RMSE was was reduced to 4.8,toand residual nonuniformity noise of the corrected image the corresponding RMSE reduced 4.8,the and the residual nonuniformity noise of the corrected was weak, indicated by the arrow Figurein7.Figure After frame 425,frame both the edge bb curve and the RMSE image wasas weak, as indicated by theinarrow 7. After 425,̅̅̅̅̅̅ both the edge curve and b curve remain stable. In other words, when the ̅̅̅̅̅̅ edge bb changed the residual noise was strong. the RMSE curve remain stable. In other words, when the edgeviolently, changed violently, the residual noise b ̅̅̅̅̅̅ On contrary, when edgebb reached stability, the residual noise was weak. Thewas curve shows that the wasthe strong. On the contrary, when edge reached stability, the residual noise weak. The curve b

Sensors 2018, 18, 211

10 of 15

Sensors 2018, 18, 211

10 of 15

shows that the adaptive spatial threshold always shows some fluctuations influenced by the scene adaptive spatial threshold S always showsSsome fluctuations influenced by the scene information. In information. In general, however, it exhibits a downward state. The S curve decreases sharply after general, however, it exhibits a downward state. The S curve decreases sharply after 425 frames, which 425 frames, which is consistent withAfter the RMSE After frame becauseof of̅̅̅̅̅̅̅ the bregulation of is consistent with the RMSE curve. framecurve. 425, because of the425, regulation edge , the S curve edge , the S curve fluctuates near 20, ensuring that the ghosting artifacts were suppressed while some b fluctuates near 20, ensuring that the ghosting artifacts were suppressed while some weak noise was weak noiseThe wasanalysis removed. The analysis results the index edgeb used in thecan algorithm can ̅̅̅̅̅̅ removed. results show that the show indexthat edge b used in the algorithm effectively effectively theofintensity residual nonuniformity in theimage. corrected image. Under characterizecharacterize the intensity residual of nonuniformity noise in thenoise corrected Under its control, its control, the adaptive spatial threshold is able to restrict the ghosting artifacts more intelligently the adaptive spatial threshold is able to restrict the ghosting artifacts more intelligently without without the convergence of the algorithm. affectingaffecting the convergence speed ofspeed the algorithm. 1500

S edgeb RMSE 1000

500

0

0

100

200

300

400

500 Frames

600

700

800

900

1000

Figure 7.7.The The variation ofimportant the important variables in our during algorithm during the simulated Figure variation of the variables in our algorithm the simulated nonuniformity nonuniformity data calibration. data calibration.

4.2. Nonuniformity Correction with with Real Real Data Data 4.2. Nonuniformity Correction Because it it is is difficult difficult to to use use PSNR PSNR as as the the NUC NUC evaluation evaluation index index for for real real data, data, and and because because the the Because FPN is expressed as spatial high frequency information, the amount of FPN present in a real image FPN is expressed as spatial high frequency information, the amount of FPN present in a real image is is usually measured using roughness index. usually measured using thethe roughness index.

h1  I  h2  I

kh1 ∗ I k11 + kh2 ∗ 1I k, 1 ρ =  , k IIk11

(25) (25)

where hh11and andh2hare 2 are a horizontal vertical difference respectively; is theunder image under where a horizontal andand vertical difference filter, filter, respectively; I is the Iimage analysis; 1 1 analysis; is theofl f.norm of f. Specifically, the value lower for theρ,value for ρ,the theFPN, lowerand thevice FPN, and and k f k1 and is the‖f‖ l 1norm Specifically, the lower the the lower versa. vice versa. The first real infrared data was collected for the indoor scene at 4 p.m., using a 320 × 240 InSb firstinfrared real infrared data was collected the indoor scene at 43p.m., using a 320 240 InSb FPA The cooled imager from Cedip. Thefor imager operates in the −5 µm range and× works at FPA cooled imagerThe from Cedip. of The in 14-bits the 3−5image μm range and at 50 frames perinfrared second (FPS). F-number theimager optics operates is 0.75. The data of theworks detector 50 frames per with second (FPS). Thenonuniformity F-number of the optics is 0.75. The 14-bits image data of the was collected only a gain calibration-based correction. The imager wasdetector moved was collectedin with a gain nonuniformity calibration-based correction. The imagerindoor was moved continuously the only first 540 frames. Subsequently, while it was still facing the complex scene, continuously in the first 540 frames. Subsequently, whilewere it was2000 stillframes. facing the complex indoor scene, the FOV was repeatedly blocked by hand. In total, there the FOV repeatedly blocked byused hand. total, there wereparameters 2000 frames. The was selected algorithms were toIn correct the offset of the video sequence. All selectedused algorithms used the offset parameters the video All LMS The algorithms kal = 2, were T = 10, andtoacorrect four neighborhood filter. Asof the FPN wassequence. serious and LMSsignal algorithms used ksubmerged al = 2, T = 10, a four Asand theaFPN serious and the the was almost byand noise, theneighborhood ST–LMS used filter. Ts = 80 7 × 7was spatial correlation signal was almost submerged noise, the ST–LMS used Ts = 80 a 7 × 7 spatial correlation window. The parameters in our by algorithm were the same as those for and the simulated experiment. This window. The parameters algorithm speed were the as those formethod the simulated experiment. This experiment mainly tested in theour convergence andsame the deghosting of the tested algorithms experiment mainly tested the convergence speed and the deghosting method of the tested algorithms for real nonuniformity noise. for real nonuniformity Video S2 comparesnoise. the tested algorithms for a visual evaluation. It is apparent that our algorithm Video generated S2 compares the tested algorithms a visual evaluation. It is apparent that our algorithm effectively fewer ghosting artifactsfor than the other algorithms. When the imager remains effectivelydue generated fewer ghosting artifacts the other algorithms. Wheninthe remains stationary to the platform’s weak jitter, the than temporal variations of grayscale theimager same pixel will stationary due the to the platform’s weak jitter, temporalblurs variations of grayscale in of thethe same pixel be higher than threshold. In this case, the the Gated–LMS the details regardless intensity will higher than threshold. Inred thisbox case, blurs the regardless of the of thebegradient, just the as shown in the in the Gated–LMS top row of Figure 8b. details The spatial threshold of intensity of the gradient, just as shown in the red box in the top row of Figure 8b. The spatial threshold of ST–LMS retains some textures, but the details within the threshold are gradually degraded, as shown in the red box in the top row of Figure 8c. This phenomenon is more pronounced when the

Sensors 2018, 18, 211

11 of 15

ST–LMS retains some Sensors Sensors2018, 2018,18, 18,211 211

textures, but the details within the threshold are gradually degraded, as shown 11ofof15 15 11 in the red box in the top row of Figure 8c. This phenomenon is more pronounced when the FOV is FOV occluded by large of artifacts occurred inGated–LMS theGated–LMS Gated–LMS frequently occluded by hand: ahand: largeaanumber of ghosting artifacts occurred in the and FOVisisfrequently frequently occluded byhand: large number number of ghosting ghosting artifacts occurred in the and ST–LMS, as shown in the blue box in the bottom row of Figure 8b,c. Our algorithm preserved ST–LMS, as shown in the blue box in the bottom row of Figure 8b,c. Our algorithm preserved most and ST–LMS, as shown in the blue box in the bottom row of Figure 8b,c. Our algorithm preservedof most ofofthe image details, and Figure 8d shows that there were no obvious obvious degradation orghosting ghosting the image details, Figure shows there were no were obvious degradation or ghosting artifacts most the imageand details, and8d Figure 8dthat shows that there no degradation or artifacts during the imager and hand of theThe FOV. The effect visualof effect ofthe thecorrected corrected during the imager and hand occlusion of the FOV. visual the of corrected image artifacts during thejittering imagerjittering jittering and hand occlusion occlusion of the FOV. The visual effect image processed by our algorithm is satisfactory. processed by our algorithm is satisfactory. image processed by our algorithm is satisfactory.

(a) (a)

(b) (b)

(c) (c)

(d) (d)

Figure 8.8.NUC NUC performance comparison of the real real nonuniformity nonuniformityinfrared infraredsequence: sequence:(a) (a)un-corrected un-corrected Figure8. NUCperformance performancecomparison comparison of of the the real nonuniformity Figure infrared sequence: (a) un-corrected image; (b) Gated–LMS; (c) ST–LMS; (d) our image. The top row is the 906th frame, when theimager imager image; (b) Gated–LMS; (c) ST–LMS; (d) our The top row is the 906th frame, when image; (b) Gated–LMS; (c) ST–LMS; (d) our image. The top row is the 906th frame, whenthe the imager jittered toward the indoor scene. The bottom row is the 1965th frame, when the FOV was blocked by jitteredtoward towardthe theindoor indoorscene. scene.The bottom row is the 1965th 1965th frame, jittered frame, when when the the FOV FOVwas wasblocked blockedby by aahand. hand. a hand.

Figure999shows showsthe theroughness roughnesscurves of the the image image sequences Figure shows the roughness curves of imagesequences sequencesprocessed processedby bythe thetested testedalgorithms. algorithms. Figure processed by the tested algorithms. The figure shows that the roughness curves drop rapidly in the first 540 frames and that The figure figure shows shows that that the the roughness roughness curves curves drop rapidly rapidly in in the the first first 540 540 frames frames and and that thatthey theyare are The they are almost coincident. This further proves that the adaptive edge detection did not hinder the removal almost coincident. This further proves that the adaptive edge detection did not hinder the removal of almost coincident. This further proves that the adaptive edge detection did not hinder the removal of nonuniformity noise. Then, during the frequent occlusion of the FOV, the roughness curves appear nonuniformity noise. Then, during thethe frequent occlusion of of thethe FOV, thethe roughness curves appear to of nonuniformity noise. Then, during frequent occlusion FOV, roughness curves appear tochange change cyclical manner. When the hand was removed, image roughness became larger. In change in in ain cyclical manner. When thethe hand waswas removed, the the image roughness became larger. In this to aacyclical manner. When hand removed, the image roughness became larger. In this case, the roughness ofalgorithm our algorithm was the highest because ST–LMS and Gated–LMS caused case, the roughness of our was the highest because ST–LMS and Gated–LMS caused image this case, the roughness of our algorithm was the highest because ST–LMS and Gated–LMS caused image degradation. The roughness was reduced when the FOV was blocked by hand. The roughness degradation. The roughness was reduced when the FOV blocked by hand. roughness of our image degradation. The roughness was reduced when thewas FOV was blocked by The hand. The roughness of our algorithm was at lowest at this time because both ST–LMS and Gated–LMS resulted in ghosting algorithm was lowest this time because both ST–LMS and Gated–LMS resulted in ghosting artifacts. of our algorithm was lowest at this time because both ST–LMS and Gated–LMS resulted in ghosting artifacts. The experiment further shows that the adaptive spatial threshold our algorithm have The experiment further shows that the adaptive threshold makes ourmakes algorithm have a stronger artifacts. The experiment further shows that the spatial adaptive spatial threshold makes our algorithm have a stronger deghosting ability than ST–LMS and Gated–LMS, not the reducing the denoising ability than ST–LMS Gated–LMS, while not while reducing denoising ability. ability. adeghosting stronger deghosting ability thanand ST–LMS and Gated–LMS, while not reducing the denoising ability. 0.018 0.018

Gat ed-LMS ST Gat-LMS ed-LMS Ours ST -LMS Ours

0.016 0.016 0.014

0.014 0.012





0.012 0.01 0.01 0.008

0.008 0.006 0.006 0.004 0.004 0.002 0.002 0 0

0

0

200

200

400

400

600

800

600

800

1000 1200 Frames 1000 1200 Frames

1400

1600

1800

2000

1400

1600

1800

2000

Figure 9. Roughness versus frame number for the tested algorithms using real nonuniformity data. Figure 9. 9. Roughness Roughness versus versus frame frame number number for for the the tested tested algorithms algorithms using using real real nonuniformity nonuniformity data. data. Figure

Figure 10 shows the frame-by-frame variation of the correction matrix gradient ̅̅̅̅̅̅̅̅ edgeb and the ̅̅̅̅̅̅̅̅b and the Figure 10 shows the frame-by-frame variation of the correction matrix gradient ̅̅̅̅̅̅ adaptive threshold S. It can be seen that the curve of edgeb gradually rises to a certain edge point before adaptive threshold It canwhile be seen the curve of ̅̅̅̅̅̅ edge b gradually rises a certainthe point before stabilizing. For 481 S. frames, thethat temporal variation was lower than the to threshold, corrected stabilizing. For 481 frames, while the temporal variation was lower than the threshold, the corrected

Sensors 2018, 18, 211

12 of 15

Figure 10 shows the frame-by-frame variation of the correction matrix gradient edgeb and the adaptive threshold S. It can be seen that the curve of edgeb gradually rises to a certain point before Sensors 2018, 18, 211 12 of 15 stabilizing. For 481 frames, while the temporal variation was lower than the threshold, the corrected image noise was very weak, as indicated by the arrow in Figure 10. The S almost represented the noise image noise was very weak, as indicated by the arrow in Figure 10. The S almost represented the level evaluated by theby subjective visual. AfterAfter 481 481 frames, S drops sharply, before remaining at noiseaslevel as evaluated the subjective visual. frames, S drops sharply, before remaining Sensors 2018, 18, 211 12 of 15 around 20. Consequently, the spatial threshold S in our adaptively changed according to the at around 20. Consequently, the spatial threshold S inalgorithm our algorithm adaptively changed according real residual nonuniformity noise. to the real residual nonuniformity noise. image noise was very weak, as indicated by the arrow in Figure 10. The S almost represented the noise level as evaluated by the subjective visual. After 481 frames, S drops sharply, before remaining 1800 at around 20. Consequently, the spatial threshold S in our algorithm S adaptively changed according 1600 edgeb to the real residual nonuniformity noise. 1400 1800

1200

S edgeb

1600

1000 1400

800 1200

600 1000

400

800

200600 0400 0

200

400

200

600

800

1000 1200 Frames

1400

1600

1800

2000

Figure variables in algorithm during the real real nonuniformity nonuniformity 0 important Figure10. 10.The Thevariation variationof ofthe the important variables in1200our our algorithm during the 0 200 400 600 800 1000 1400 1600 1800 2000 Frames data correction. data correction. Figure 10. The variation of the important variables in our algorithm during the real nonuniformity correction. Thedata second real data data was FPA uncooled imager fromfrom North Guang Wei The second real was acquired acquiredby bya a384 384× 288 × 288 FPA uncooled imager North Guang [23]. The imager works at 50 FPS and the F-number of the optics is 0.75. The outdoor scene Wei [23]. The Thesecond imagerreal works at 50 FPS and the F-number of the optics is 0.75. The outdoor scenewas was data was acquired by a 384 × 288 FPA uncooled imager from North Guang Wei observedatat99p.m., p.m.,and anduncorrected uncorrectedraw rawdata data with with severe severe FPN FPN was was obtained. obtained. The imager moved observed The imager moved [23]. The imager works at 50 FPS and the F-number of the optics is 0.75. The outdoor scene was randomlywithout withoutany anyintentional intentionaljitter jitterand andocclusion. occlusion. In In total, total, there there were were 3315 3315 frames. frames. randomly observed at 9 p.m., and uncorrected raw data with severe FPN was obtained. The imager moved Compared with the previous experiment, the images in this group were noisier. Thus, the ST– Compared with the previous experiment, the images thisthere group were noisier. the ST–LMS randomly without any intentional jitter and occlusion. In in total, were 3315 frames.Thus, LMSTsused Ts The = 200. The other parameters in the the images algorithms as those used in the with the previous experiment, in this group were noisier. Thus, the previous ST– used =Compared 200. other parameters in the algorithms were thewere samethe as same those used in the previous experiment. Figure 11 shows some typical images during the calibration. It shows that the LMS used Ts = 200. The other parameters in the algorithms were the same as those used in the experiment. Figure 11 shows some typical images during the calibration. It shows that the scene-based scene-based non-uniformity correction performed well even when the original image was mostly previous experiment. Figure 11 shows some typical images during the calibration. It shows that the non-uniformity correction performed well even when the original image was mostly covered by noise. scene-based non-uniformity correction performed well even whenthat the original image was mostly also covered by However, the red boxes in Figure 11b,c show and ST–LMS However, thenoise. red boxes in Figure 11b,c show that Gated–LMS and Gated–LMS ST–LMS also produced some covered by noise. However, the red boxes in Figure 11b,c show that Gated–LMS and ST–LMS also the produced some ghosting, even when the imager maintained random motion. In this situation, ghosting, even when the imager maintained random motion. In this situation, the performance of our produced some even when imager maintained random performance of ourghosting, algorithm was still the better, as shown in Figure 11d.motion. In this situation, the algorithm was still better, as shown in Figure 11d. performance of our algorithm was still better, as shown in Figure 11d.

(a)

(a)

(b)

(b)

(c)

(c)

(d)

(d)

Figure 11. NUC performance comparison imagermoved moved randomly without intentional Figure 11. NUC performance comparisonwhen when the the imager randomly without any any intentional Figure 11. NUC performance comparison when the imager moved randomly without any intentional jitter and occlusion: (a) uncorrected image; (b) Gated–LMS; (c) ST–LMS; (d) our image. The top rowrow is jitter and occlusion: (a) uncorrected image; (b) Gated–LMS; ST–LMS; (d) our image. The top jitter and occlusion: (a) uncorrected image; (b) Gated–LMS; (c) ST–LMS; (d) our image. The top row is the 794th frame. The bottom row is the 3289th frame. the 794th frame. The bottom row is the 3289th frame. is the 794th frame. The bottom row is the 3289th frame. 5. Efficiency Analysis

5. Efficiency Analysis

Sensors 2018, 18, 211

13 of 15

5. Efficiency Analysis In order to analyze the efficiency of the algorithm, we calculated the runtime of the algorithms for a single image on Matlab (MathWorks, Natick, MA, USA). The computer’s CPU was Intel Core i7-3540M (3.00GHz). As the conditional statement in the algorithm renders the processing capacity different for each frame, we calculated the average processing time of 100 images, as shown in Table 1. Table 1. Matlab runtime comparison of the algorithms. Method

Gated–LMS

ST–LMS

Ours

Time(s)

0.0512

0.4627

0.7472

Obviously, the adaptive edge detection module added in our algorithm increased the computational complexity of the algorithm, rendering its PC time consumption significantly higher than that of the other two algorithms. To analyze the degree of efficiency degradation, we counted all the computational resources per pixel from Equation (13) to Equation (22). The resulting data used by the different algorithms—the comparison, adding/subtracting, multiplying, and dividing—is listed in Table 2. The desired image was obtained by a four neighborhood filter, and the spatial correlation window in ST–LMS was 7 × 7. The W equaled 7 in our algorithm. Table 2. The comparison of the computational resources per pixel.

Gated–LMS ST–LMS Ours

Logical Operation

Adding

Multiplication

Division

2 77 161

19 58 131

8 8 19

4 4 5

The table shows that our algorithm consumes more than the double ST–LMS’s consumption in logical operations, addition operations, and multiplication operations. These increased computational costs mainly concern Equations (13) and (20). Fortunately, our algorithm exhibits only one time-consuming division operation in excess of the other two algorithms. It derives from the local variance calculation in Equation (20), which is only used as comparison. Given this, the division can be optimized as a combination of multiplication and shift operations without affecting the result. 6. Conclusions In this paper, we present an improved learning rate rule which combines adaptive threshold edge detection with a temporal gate in the NN-based NUC algorithm. The adaptive edge detection threshold can be automatically adjusted according to the residual nonuniformity of the corrected image, so that the algorithm has the ability to continuously suppress ghosting artifacts without affecting the convergence speed. The proposed AST–LMS method was compared with Hardie et al.’s Gated–LMS and Fan et al.’s ST–LMS methods in the experiments by using infrared video sequences with simulated nonuniformity noise and real nonuniformity noise, respectively. The simulated experiment showed that when the imager was in stable motion, AST–LMS had an equivalent PSNR to Gated–LMS and ST–LMS. However, AST–LMS had a higher PSNR than ST–LMS and Gated–LMS when the imager was jittering. The real data experiments showed that our algorithm achieves a better deghosting ability while retaining the NUC performance, regardless of whether the imager has an intentional jitter or occlusion. The roughness index of AST–LMS is better than those of Gated–LMS and ST–LMS. These results show that the adaptive method in this paper is a reasonable option for the spatial threshold in the previous deghosting method. In addition, just like the analysis in the simulated experiment, the temporal

Sensors 2018, 18, 211

14 of 15

variation of the correction matrix gradient proposed in the algorithm can be used as a criterion for the convergence of the algorithm. The nonuniformity correction algorithm always runs in the embedded field-programmable gate arrays circuit of the infrared imager. According to our previous work on the real time nonuniformity correction algorithm [5], the added edge detection and adaptive threshold calculation in this paper will increase the power consumption of the circuit and the design difficulty of the pipeline. The proposed algorithm divides the nonuniformity correction process into denoising and deghosting stages. However, in the denoising stage, pixels with faint details are still likely to produce “ghosting” because their gradients are similar to nonuniformity noise, and consequently their correction parameter updating is not limited. Future work may focus on using machine learning methods to improve the accuracy of incident radiation estimation. Supplementary Materials: The following are available online at www.mdpi.com/1424-8220/18/1/211/s1, Video S1: Algorithms performance comparison with simulated nonuniformity. Video S2: Algorithms performance comparison with real nonuniformity. In Videos S1 and S2, the top row are the original image and the image processed by Gated–LMS, from left to right. The bottom row are images processed by ST–LMS and our method, from left to right. Acknowledgments: This work was supported by the National Natural Science Foundation of China under grant no. 61231014 and the Army Equipment Pre-Study Project of China under grant no. 30102040104. Author Contributions: Yiyang Li and Weiqi Jin conceived and designed the experiments; Yiyang Li and Xu Zhang performed the experiments; Yiyang Li, Jin Zhu, and Shuo Li analyzed the data; and Yiyang Li wrote the paper. Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2. 3.

4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

Perry, D.L.; Dereniak, E.L. Linear theory of nonuniformity correction in infrared staring sensors. Opt. Eng. 1993, 32, 1854–1859. [CrossRef] Milton, A.F.; Barone, F.R.; Kruer, M.R. Influence of nonuniformity on infrared focal plane array performance. Opt. Eng. 1985, 24, 245855. [CrossRef] Scribner, D.A.; Sarkady, K.A.; Kruer, M.R.; Caulfield, J.T.; Hunt, J.D.; Colbert, M.; Descour, M. Adaptive retina-like preprocessing for imaging detector arrays. In Proceedings of the IEEE International Conference on Neural Networks, San Francisco, CA, USA, 28 March–1 April 1993. Zuo, C.; Chen, Q.; Gu, G.; Qian, W. New temporal high-pass filter nonuniformity correction based on bilateral filter. Opt. Rev. 2011, 18, 197–202. [CrossRef] Li, Y.; Jin, W.; Li, S.; Zhang, X.; Zhu, J. A Method of Sky Ripple Residual Nonuniformity Reduction for a Cooled Infrared Imager and Hardware Implementation. Sensors 2017, 17, 1070. Chiang, Y.-M.; Harris, J.G. An analog integrated circuit for continuous-time gain and offset calibration of sensor arrays. J. Analog Integr. Circuits Signal Process 1997, 12, 231–238. [CrossRef] Hardie, R.C.; Hayat, M.M.; Armstrong, E.E.; Yasuda, B. Scene-based nonuniformity correction using video sequences and registration. Appl. Opt. 2000, 39, 1241–1250. [CrossRef] [PubMed] Boutemedjet, A.; Deng, C.; Zhao, B. Robust Approach for Nonuniformity Correction in Infrared Focal Plane Array. Sensors 2016, 16, 1890. [CrossRef] [PubMed] Vera, E.M.; Torres, S.N. Fast Adaptive Nonuniformity Correction for Infrared Focal-Plane Array Detectors. EURASIP J. Appl. Signal Process. 2005, 13, 1994–2004. [CrossRef] Rossi, A.; Diani, M.; Corsini, G. Bilateral filter-based adaptive nonuniformity correction for infrared focal-plane array systems. Opt. Eng. 2010, 49, 057003–057013. [CrossRef] Zhang, T.; Shi, Y. Edge-directed adaptive nonuniformity correction for staring infrared plane arrays. Opt. Eng. 2006, 45, 016402. [CrossRef] Torres, S.N.; San Martin, C.; Sbarbaro, D.; Pezoa, J.E. A neural network for nonuniformity and ghosting correction of infrared image sequence. Lect. Notes Comput. Sci. 2005, 3656, 1208–1216. Hardie, R.; Baxley, F.; Brys, B.; Hytla, P. Scene-based nonuniformity correction with reduced ghosting using a gated LMS algorithm. Opt. Express 2009, 17, 14918–14933. [CrossRef] [PubMed]

Sensors 2018, 18, 211

14. 15. 16. 17. 18. 19. 20. 21.

22. 23.

15 of 15

Rong, S.H.; Zhou, H.X.; Qin, H.L.; Lai, R.; Qian, K. Guided filter and adaptive learning rate based non-uniformity correction algorithm for infrared focal plane array. Infrared Phys. Technol. 2016, 76, 691–697. Fan, F.; Ma, Y.; Huang, J.; Liu, Z.; Liu, C.Y. A combined temporal and spatial deghosting technique in scene based nonuniformity correction. Infrared Phys. Technol. 2015, 71, 408–415. [CrossRef] Yu, H.; Zhang, Z.J.; Wang, C. An improved retina-like nonuniformity correction for infrared focal-plane array. Infrared Phys. Technol. 2015, 73, 62–72. [CrossRef] Martin, C.S.; Torres, S.; Pezoa, J.E. Statistical recursive filtering for offset nonuniformity estimation in infrared focal-plane-array sensors. Infrared Phys. Technol. 2008, 5745, 564–571. [CrossRef] Rogalski, A. Progress in focal plane array technologies. Prog. Quantum Electron. 2012, 36, 342–473. [CrossRef] Kaufman, L.; Kramer, D.M.; Crooks, L.E.; Ortendahl, D.A. Measuring signal-to-noise ratios in MR imaging. Radiology 1989, 173, 265–267. [CrossRef] [PubMed] Nowak, R.D. Wavelet-based Rician noise removal for magnetic resonance imaging. IEEE Trans. Image Process. 1999, 8, 1408–1419. [CrossRef] [PubMed] Aja-Fernández, S.; Vegas-Sánchez-Ferrero, G.; Martín-Fernández, M.; Alberola-López, C. Automatic noise estimation in images using local statistics. Additive and multiplicative cases. Image Vis. Comput. 2009, 27, 756–770. [CrossRef] Amer, A.; Dubois, E. Fast and reliable structure-oriented video noise estimation. IEEE Trans. Circuits Syst. Video Technol. 2005, 15, 113–118. [CrossRef] GWIR 0203X1A. Available online: http://www.gwic.com.cn/productdetail.html?id=8 (accessed on 22 October 2017). © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).