De-noising and retrieving algorithm of Mie lidar data ... - OSA Publishing

De-noising and retrieving algorithm of Mie lidar data based on the particle filter and the Fernald method Chen Li,1,2 Zengxin Pan,2 Feiyue Mao,3,4,5,6,* Wei Gong,2,4,5,8 Shihua Chen,1 and Qilong Min2,7 2

1 School of Mathematics and Statistics, Wuhan University, Wuhan 430072, China State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China 3 School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China 4 Collaborative Innovation Center for Geospatial Technology, Wuhan 430079, China 5 Hubei Collaborative Innovation Center for High-efficiency Utilization of Solar Energy, Wuhan 430079, China 6 School of Resources and Environmental Science, Wuhan University, Wuhan 430079, China 7 Atmospheric Sciences Research Center, State University of New York, Albany, New York 12203, USA 8 [email protected] * [email protected]

Abstract: The signal-to-noise ratio (SNR) of an atmospheric lidar decreases rapidly as range increases, so that maintaining high accuracy when retrieving lidar data at the far end is difficult. To avoid this problem, many de-noising algorithms have been developed; in particular, an effective denoising algorithm has been proposed to simultaneously retrieve lidar data and obtain a de-noised signal by combining the ensemble Kalman filter (EnKF) and the Fernald method. This algorithm enhances the retrieval accuracy and effective measure range of a lidar based on the Fernald method, but sometimes leads to a shift (bias) in the near range as a result of the over-smoothing caused by the EnKF. This study proposes a new scheme that avoids this phenomenon using a particle filter (PF) instead of the EnKF in the de-noising algorithm. Synthetic experiments show that the PF performs better than the EnKF and Fernald methods. The root mean square error of PF are 52.55% and 38.14% of that of the Fernald and EnKF methods, and PF increases the SNR by 44.36% and 11.57% of that of the Fernald and EnKF methods, respectively. For experiments with real signals, the relative bias of the EnKF is 5.72%, which is reduced to 2.15% by the PF in the near range. Furthermore, the suppression impact on the random noise in the far range is also made significant via the PF. An extensive application of the PF method can be useful in determining the local and global properties of aerosols. ©2015 Optical Society of America OCIS codes: (280.1100) Aerosol detection; (280.3640) Lidar.

References and links 1. 2. 3. 4. 5. 6.

V. A. Kovalev and W. E. Eichinger, Elastic Lidar: Theory, Practice, and Analysis Methods (Wiley-Interscience, 2004). W. Wang, W. Gong, F. Mao, and J. Zhang, “Long-Term Measurement for Low-Tropospheric Water Vapor and Aerosol by Raman Lidar in Wuhan,” Atmosphere 6(4), 521–533 (2015). F. Mao, W. Gong, S. Song, and Z. Zhu, “Determination of the Boundary Layer Top from Lidar Backscatter Profiles Using a Haar Wavelet Method over Wuhan, China,” Opt. Laser Technol. 49, 343–349 (2013). X. Cao, Z. Wang, P. Tian, J. Wang, L. Zhang, and X. Quan, “Statistics of Aerosol Extinction Coefficient Profiles and Optical Depth Using Lidar Measurement over Lanzhou, China since 2005–2008,” J. Quant. Spectrosc. Radiat. Transf. 122, 150–154 (2013). W. Sun, Y. Hu, B. Lin, Z. Liu, and G. Videen, “The Impact of Ice Cloud Particle Microphysics on the Uncertainty of Ice Water Content Retrievals,” J. Quant. Spectrosc. Ra 112(2), 189–196 (2011). M. Feiyue, G. Wei, and M. Yingying, “Retrieving the Aerosol Lidar Ratio Profile by Combining Ground- and Space-Based Elastic Lidars,” Opt. Lett. 37(4), 617–619 (2012).

#245540 (C) 2015 OSA

Received 8 Jul 2015; revised 19 Sep 2015; accepted 20 Sep 2015; published 1 Oct 2015 5 Oct 2015 | Vol. 23, No. 20 | DOI:10.1364/OE.23.026509 | OPTICS EXPRESS 26509

7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24.

R. Collis, “Lidar: A New Atmospheric Probe,” Q. J. R. Meteorol. Soc. 92(392), 220–230 (1966). J. D. Klett, “Stable Analytical Inversion Solution for Processing Lidar Returns,” Appl. Opt. 20(2), 211–220 (1981). F. G. Fernald, “Analysis of Atmospheric Lidar Observations: Some Comments,” Appl. Opt. 23(5), 652–653 (1984). F. Rocadenbosch, C. Soriano, A. Comerón, and J. M. Baldasano, “Lidar Inversion of Atmospheric Backscatter and Extinction-to-Backscatter ratios by Use of a Kalman Filter,” Appl. Opt. 38(15), 3175–3189 (1999). V. A. Kovalev, “Stable near-End Solution of the Lidar Equation for Clear Atmospheres,” Appl. Opt. 42(3), 585– 591 (2003). H. T. Fang and D. S. Huang, “Noise Reduction in Lidar Signal Based on Discrete Wavelet Transform,” Opt. Commun. 233(1-3), 67–76 (2004). H. T. Fang, D. S. Huang, and Y. H. Wu, “Antinoise Approximation of the Lidar Signal with Wavelet Neural Networks,” Appl. Opt. 44(6), 1077–1083 (2005). W. Gong, J. Li, F. Mao, and J. Zhang, “Comparison of Simultaneous Signals Obtained from a Dual-Field-ofView Lidar and Its Application to Noise Reduction Based on Empirical Mode Decomposition,” Chin. Opt. Lett. 9(5), 050101 (2011). P. Tian, X. Cao, J. Liang, L. Zhang, N. Yi, L. Wang, and X. Cheng, “Improved Empirical Mode Decomposition Based Denoising Method for Lidar Signals,” Opt. Commun. 325, 54–59 (2014). F. Mao, W. Gong, and C. Li, “Anti-Noise Algorithm of Lidar Data Retrieval by Combining the Ensemble Kalman Filter and the Fernald Method,” Opt. Express 21(7), 8286–8297 (2013). J. Su, Z. Liu, Y. Wu, M. P. McCormick, and L. Lei, “Retrieval of Multi-Wavelength Aerosol Lidar Ratio Profiles Using Raman Scattering and Mie Backscattering Signals,” Atmos. Environ. 79, 36–40 (2013). A. Doucet, S. Godsill, and C. Andrieu, “On Sequential Monte Carlo Sampling Methods for Bayesian Filtering,” Stat. Comput. 10(3), 197–208 (2000). M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, “A Tutorial on Particle Filters for Online Nonlinear/Non-Gaussian Bayesian Tracking,” IEEE Trans. Signal Process. 50(2), 174–188 (2002). W. Gong, F. Mao, and J. Li, “Oflid: Simple Method of Overlap Factor Calculation with Laser Intensity Distribution for Biaxial Lidar,” Opt. Commun. 284(12), 2966–2971 (2011). Y. Sasano, “Tropospheric Aerosol Extinction Coefficient Profiles Derived from Scanning Lidar Measurements Over Tsukuba, Japan, from 1990 to 1993,” Appl. Opt. 35(24), 4941–4952 (1996). H. Qiu, J. Lee, J. Lin, and G. Yu, “Wavelet Filter-Based Weak Signature Detection Method and Its Application on Rolling Element Bearing Prognostics,” J. Sound Vibrat. 289(4-5), 1066–1090 (2006). F. Luisier, T. Blu, B. Forster, and M. Unser, “Which Wavelet Bases Are the Best for Image Denoising?” in Optics & Photonics 2005, (International Society for Optics and Photonics, 2005), 59140E–59140E–59112. L. Huang, W. Gong, J. Li, F. Mao, and L. Lei, “Signal Splicing of Dual-Receiver Mie Scattering Lidar Inatmospheric Remote Sensing,” J. Remote Sens. 16, 15 (2012).

1. Introduction Lidar is a powerful active measurement that makes range-resolved observations of the Earth’s atmosphere, such as the vertical structure of clouds and aerosols, planetary boundary layer, water vapor and stratospheric ozone [1–4], which are difficult to determine using passive remote sensing measurements [5]. Lidar has been extensively used to study horizontal and vertical atmospheric structures and their time evaluations. A large number of lidar networks and satellite lidar systems have been established and served, such as the Micro-pulse Lidar Network, Cloud–Aerosol Lidar, and Infrared Pathfinder Satellite Observations, which significantly contribute to the research on local and global climatology [6]. The lidar signal is the amount of energy that reaches the lidar detector from the range during a given sampling time interval. For a Mie lidar signal, the two variables that need to be retrieved are the backscatter and extinction coefficient. Since the 1960s, several algorithms have been developed to obtain the atmospheric and aerosol parameters with a lidar signal [1, 7–11]. In the 1980s, Klett and Fernald provided stable inversion methods that have been extensively used and considered as a standard inversion method [8,9]. The Fernald twocomponent method, which is mathematically equivalent to the Klett one-component method, treats aerosol and molecular components separately, which is vital when neither component is dominant. As such, the Fernald two-component method is chosen in this work. In the retrieval, the signal-to-noise ratio (SNR) decreases rapidly with an increase in the range and attenuation of laser beam energy. Averages over replications in time and vertical directions are generally applied to increase the SNR and effective measuring range, but they sharply reduce the temporal or vertical resolution of the lidar measurement. To retrieve the signal with high accuracy and temporal and vertical resolution, many methods, such as the Monte Carlo average, Fourier, wavelet, and empirical mode

#245540 (C) 2015 OSA


decomposition (EMD) methods have been used for de-noising lidar signals [12–14]. Most denoising methods are difficult to use in de-noising raw signals, that is, P(r), where r is range. The magnitude range of the range-corrected signal X(r) [i.e., P(r)r2] is smaller than that of P(r). The raw lidar signal P(r) is a strongly nonlinear and non-stationary signal because of the strong attenuation caused by the 1/r2 effect. The range of the coefficient of variation can be larger than 107 or more orders of magnitude. This property leads to the inadequacy of the moving average and Fourier methods in de-noising the lidar signal [13,14]. Furthermore, wavelet analysis is beset by issues pertaining to cutoff frequency and basic wavelet function selection; the EMD-based automatic method of de-noising lidar data performs better than the wavelet method [15]. However, the noise level of X(r) is equivalent to e(r)r2, which rapidly increases as range increases, thereby causing de-noising methods based on X(r) to yield undesirable results, where e(r) is the noise of P(r). Thus, results based on the de-noised signal by the Fernald method can be seriously distorted. To avoid this problem, some approaches have been proposed to simultaneously de-noise the lidar signal and retrieve the corresponding backscatter coefficients [16,17]. A framework has previously been proposed by combining the ensemble Kalman filter (EnKF) and the Fernald two-component method [16]. Experiments of both synthetic and real signals illustrate that lidar signals are simultaneously and effectively de-noised with a corresponding retrieval result via this method. However, the bias caused by the forecast is not eliminated perfectly. Such results suggest that this method is inferior in the near range where aerosol is rapidly changing. In this study, a particle filter (PF) algorithm instead of the EnKF is used. The PF is also based on the Monte Carlo Markov Chain (MCMC) method, but the weights of the particles instead of their values (i.e., similar to the ensemble members in the EnKF) are updated by the measurements [18,19]. Synthetic experiments with simulated signals illustrate that the signal noise is suppressed, and the retrieval result can track the original with PF. From the results of 200 replicates for signals contaminated with random noise, the average SNR of the results of the PF method increases up to 27.26, while those for the standard Fernald and EnKF methods are 18.88 and 24.43, respectively. The effect of denoising by EnKF in the far range is close to that of averaging 64 signals, and the result of the PF is close to that of averaging 16 signals. A comparison of the results of the Fernald method with the results of EnKF and PF in the near range (i.e., from 0.3 km to 2 km) shows that the relative biases (shifts) are 5.72% and 2.15% on average, respectively. Thus, the PF method not only retrieves the backscatter coefficient in the near range more accurately than the other methods compared in this study, but also effectively reduces the noise in the far range. 2. Principles and methods Basic lidar principles may be described formally in a single-scattering lidar equation as [1, 9] P (r)=

{

}

r C ⋅ G ( r ) ⋅ [ β1 (r)+β 2 (r) ] ⋅ exp −2 [α1 (r)+α 2 (r) ] dr + e ( r ) , 2 0 r

(1)

where r is the altitude, P(r) denotes the instantaneous received power at range r, C is the lidar constant, G(r) is the overlap factor (i.e., the ratio of the received energy to the energy hits the telescope mirror) [20], β1(r) and β2(r) are the aerosol backscattering coefficient (ABC) and molecular backscattering coefficient, respectively, α1(r) and α2(r) are the aerosol and molecular extinction coefficient, respectively, and e(r) is the noise which can be considered Gaussian approximately. The Fernald method is the most common inversion method that can distinguish the contributions between molecules and aerosols, but is very sensitive to SNR. Thus, we propose a more robust de-noising algorithm to avoid this problem. 2.1 Flow of the retrieval and de-noising algorithm The alternative algorithm for de-noising signals and retrieving optical properties contains two parts: forward and backward, where the backward part indicates that the retrieval of backscatter is started from a selected reference height to the ground while the forward part is

#245540 (C) 2015 OSA


started from a selected reference height to the farther range. We discuss here only the backward part; the forward part can be established in a similar way. In this algorithm, the overlap factor is regarded as constant. As shown in Fig. 1, the backward part of this algorithm includes three key steps:

Fig. 1. Flow of the retrieval and de-noising algorithm.

(1) Particle sampling and forecast for range-corrected signal in the next range bin In accordance with the spatial correlation for ABC, β1(i + n),...,β1(i + 1), β1(i) and Eq. (1) are used to calculate Xratio(i). Xratio(i) is given as the ratio of the range-corrected signal in two contiguous range bins and is defined as X ratio (i) =

β1 (i)+β 2 (i) X (i) = ⋅ exp {−2 [α1 (i)+α 2 (i) ] Δr} , X (i-1) β1 (i-1)+β 2 (i-1)

(2)

where Δr is the vertical resolution and the altitude of bin i is i·Δr. In this step, the Monte Carlo method is first used with X(i) and σ·r2(i) as the mean and standard deviation of Gaussian distribution to sample a corresponding set of X j (i ) with its weight w j (i ) (j = 1,...,N, and N is the ensemble size or called as number of particles), where the σ is the standard deviation of the noise of the raw lidar signal. Second, β1(i-1) is obtained by linear extrapolation with β1(i + n),...,β1(i + 1), β1(i). For a lidar with high vertical resolution, β1(i-1) = β1(i) can be used for simplicity and without loss of generality. Third, Xratio(i) is derived by applying β1(i-1) and the given lidar ratio. Finally, X j (i ) and Xratio(i) are used to approximately forecast X j (i − 1) , which is applied in the next step as follows:

X j (i − 1) = X j (i ) / X ratio (i ),

(3)

(2) Data assimilation with PF Here, the PF is used to simultaneously estimate X (i − 1) and reduce noise. X j (i − 1) , w j (i − 1) , and X o (i − 1) are applied as the input parameters to update the w j (i − 1) in the PF,

where X o is the original signal which is used as ‘observation’ in the data assimilation algorithm and Pr( X o (i − 1) | X j (i − 1)) is conditional probability of X o (i − 1) given X j (i − 1) . Then, the weighted mean of X j (i − 1) is used as a more accurate estimation for X(i-1). This procedure is similar to the method in [16] in using PF instead of EnKF.

#245540 (C) 2015 OSA


(3) Lidar data retrieval by the Fernald method In this step, β1(i-1) is retrieved based on the estimated X(i) and X(i-1) by the standard Fernald method, after which Steps 1–3 are iterated until the whole β1 profile is retrieved. 2.2 De-noising by PF The PF is based on Monte Carlo methods, that is, sampling to approximate a distribution from its prior information. The representation of particles follow the distribution P( xk |z0:k ) , where zk is the observation of the model state xk at time k, and z0:k is the observation sequence from time 0 to k. The basic idea of PF is the representation of the distribution of the state variables with an ensemble of N particles and their own weights {xki , wki }iN=1 .The distribution of particles can be represented approximately as P ( xk | z1:k ) ≈  i =1 wki δ ( xk − xki ), N

(4)

where xki is sampled from q( xki | xki −1 , zk ) , and wki is updated from wki −1 by wki ∝ wki −1

p( zk | xki ) p( xki | xki −1 ) , q ( xki | xki −1 , zk )

(5)

where q( xki | xki −1 , zk ) is the distribution function for importance sampling and p( xki | xki −1 ) the conditional probability function of zk given xki . Usually, for simplicity we set q( xki | xki −1 , zk ) = p( xki | xki −1 ),

(6)

and xki is calculated from xki −1 with the model evolution [19]. Notice that wki is normalized as wki = w ki

(

N i =1

)

w ki ,

(7)

where w ki = wki −1

p ( zk | xki ) p ( xki | xki −1 ) , q( xki | xki −1 , zk )

(8)

and when k = 0, wki is usually set as 1/N for all i. In the PF, along with the model states evolution, a common problem called degeneracy phenomenon makes the algorithm defective, which means the weight of all but one particle is negligible (i.e., approaching 0) with a few iterations. The covariance of the importance weight increases and cannot be avoided [18]. Resampling is usually applied to the particles to avoid filter degeneracy, and the principle is to eliminate particles with small weights and concentrate on particles with large weights [19]. A common and easy method of estimating the degeneracy of the algorithm is N eff = 1



N i =1

( wki ) 2 ,

(9)

where wki is the normalized weight obtained using Eq. (7) and (8). Proving that range of Neff is [1, N] using Cauchy–Schwarz inequality is easy. Furthermore, the degeneracy of the particles is severe when Neff is small. Thus, the algorithm of the PF can be briefly described as (1) Sampling {xki , wki }iN=1 from q( xki | xki −1 , zk ),

#245540 (C) 2015 OSA


(2) Updating weights according to Eq. (7) and (8), (3) Calculating Neff by Eq. (9), if Neff < Nth, Resampling; otherwise, go to next step, (4) Estimating xˆk =  i =1 wki xki . N

2.3 Data retrieval by the standard Fernald method For a mono-wavelength Mie lidar, ABC can be retrieved via the two-component Fernald method only if the lidar ratio S1(i) and the aerosol backscatter coefficients β1(r) are assumed at a calibration range. Via the Fernald method, the total backscatter coefficients at altitude i·Δr can be given under those assumptions for ground-based lidars. The two-component Fernald method includes forward and backward schemes, which can be presented as [8,9] The forward scheme is given by X ( i + 1) exp  − A ( i ) 

β1 ( i + 1) =

X (i ) − S X ( i ) + X ( i + 1) exp  − A ( i )  Δr β (i ) 1 The backward scheme is given by

β1 ( i − 1) =

{

}

X ( i − 1) exp  A ( i − 1) 

− β 2 ( i + 1) ,

X (i ) + S X ( i ) + X ( i − 1) exp  A ( i − 1)  Δr β (i ) 1

{

}

− β 2 ( i − 1) .

(10)

(11)

where A ( i ) =  S1 ( i ) − S2 ( i )   β 2 ( i ) + β 2 ( i + 1)  Δr , β2(i) is the molecule backscatter coefficient defined in the US standard atmospheric model, and the molecule lidar ratio S2(i) = α2(i)/β2(i) = 8π/3. Usually, the aerosol lidar ratio S1(i) sharply changes because of the temporal and spatial variation of the physical and chemical properties of aerosols [6], but it is set as a constant that equals 50 sr in this study, following a previous study with long-term observations [21]. In the present study, the calibration altitude c is set at about 10 km, and β(c)/β2(c) = 1.05β2(c) is assumed for the retrieval experiments with real data. 3. Results and discussion

In this section, the performance of the anti-noise retrieval of lidar data based on EnKF and PF is tested with simulated and real signals, respectively. Real signals are measured by our ground-based lidar at a wavelength of 532 nm with a vertical resolution of 7.5 m. At first, the two algorithms are tested with a single simulated signal. 3.1 Experiments with simulated signals To test these algorithms, as illustrated in Fig. 2(a), a single simulated signal with 50 sr as the aerosol lidar ratio is simulated under ideal conditions with the same vertical resolution as the real signals measured by our ground-based lidar. A planetary boundary layer and a typical optically thick multi-layer are simulated at 0–2 km and 4.4–6.8 km, respectively. Figure 2(a) shows the true signal along with a Gaussian noise-contaminated signal, as well as signals denoised by EnKF and PF. Figure 2(b) shows the true backscatter coefficient and retrieved ABC by the standard Fernald, EnKF, and PF methods. In this way, the anti-noise effects of the EnKF and PF can be compared with the true values and signals in Figs. 2(a) and 2(b), respectively. The uncertainty of ABC retrieved by both the EnKF and PF method is much smaller than that of the standard Fernald method. In addition, the noise of the lidar signal has been significantly eliminated by these two methods. To compare the performance of the EnKF and PF, the detail of signal de-noising at the simulated multi-layer at 4.4–6.8 km are presented in Figs. 2(c) and 2(d). Biases exist in the result of the EnKF, which may have been

#245540 (C) 2015 OSA


caused by forecasts, whereas no biases exist for the PF. The PF method can track the true value more accurately.

Fig. 2. De-noising effect of the EnKF and PF methods. (a) True, noise-contaminated and two method de-noised signals, respectively. (b) True aerosol backscatter coefficient used in the simulation, and aerosol coefficients retrieved by the standard Fernald method, EnKF, and PF from noise-contaminated signals, respectively. (c) and (d) are enlarged subsection views of (a) and (b), respectively.

Fig. 3. (a) Mean and error bars (standard deviation) of the noise-contaminated and de-noised signals, respectively. (b) Mean and error bar of the aerosol backscatter coefficients retrieved by the standard Fernald method, EnKF, and PF methods. The error bars in (a) and (b) are shown with an interval of 40 range bins to avoid overlay. (c) and (d) are enlarged subsection views of (a) and (b), respectively.

To evaluate the behavior of these methods, signals retrieved and de-noised by these methods are compared with the true signal. In addition, 200 replicates of these tests are used

#245540 (C) 2015 OSA


to illustrate the performance and robustness of these algorithms. To further investigate the stability and effect of the de-noising process for the EnKF and PF, 200 noise-contaminated signals are simulated and processed. The means and standard deviations of the signals and retrieval results are calculated at different heights as shown in Fig. 3. Below the multi-layer, the standard deviations of all the methods are similar and negligible. This suggests that all these methods work well in this range because the SNR is high. In the multi-peak layer range, the standard deviations of the EnKF and PF are much lower than those of the standard Fernald method. Biases exist in the result of the EnKF, whereas no distinct biases exist in those of the PF. In the range over the multi-peak layer, the retrieval results of the EnKF and PF are close, and the standard deviations for these two methods are less than those of the standard Fernald method. This implies that the noise should be significantly reduced in this range because of the low SNR. The errors of the EnKF and PF are generally smaller than those of the standard Fernald method in this range, and both of the EnKF and PF can keep the layer’s structure and distinguish close layers well. The RMSE, R2, and SNR of all the 200 replicates of the processed contaminated signals are calculated, and the boxplots for these results are given in Fig. 4. The means and standard deviations of RMSE, R2, and SNR of all the replicates are calculated and shown in Table 1. The wavelet is used here for comparison since the wavelet method is widely used for signal de-noising [22], and DB4 (Daubechies wavelet of order 4) is chosen as one of the popular and standard wavelet bases [23]. EnKF and PF can effectively eliminate noise from the contaminated signal; RMSEs decrease, whereas R2 and SNR increase. The EnKF and PF methods enhance the 95% confidence interval of SNR to [23.82, 25.36] and [25.73, 28.79] and the mean of SNR by 29.40% and 44.46%, respectively. The mean of the RMSEs of the EnKF and PF methods is 52.55% and 38.14% of that of the Fernald method, respectively. Both of the EnKF and PF provide better performances of de-noising than the wavelet. Thus, the PF method generally has the best results according to the mean of the statistics, and the EnKF has the smallest standard deviations for RMSE and R2. As for the standard deviations of the SNR, the standard Fernald method is the lowest. This is possible due to that the EnKF introduces some sampling errors, but PF has a larger range of variation because of the process of sampling and resampling. Table 1. RMSE, SNR, and R2 of the results of four methods; 200 noise-contaminated signals are simulated and processed for these statistics Mean RMSE R2 SNR

Standard Fernald 4.27E-06 0.987 18.88

Standard deviation

Wavelet

EnKF

PF

2.79 E-06 0.994 22.57

2.24E-06 0.997 24.43

1.63E-06 0.998 27.26

Standard Fernald 1.36E-07 8.04E-04 0.27

Wavelet

EnKF

PF

1.51 E-07 5.97 E-04 0.47

7.97 E-07 2.57E-04 0.31

1.45E-07 3.39E-04 0.78

Fig. 4. Boxplots of the RMSE, SNR, and R2 of the results of four methods; 200 noisecontaminated signals are simulated and processed for this statistics

#245540 (C) 2015 OSA


3.2 Testing with real signals To further study the de-noising performance and practical utility of the EnKF and PF methods, several real cases were tested with signals measured by our ground-based lidar at 532 nm at 30.5° N and 114.3° E (Wuhan University) on November 30, 2008. The vertical and temporal resolutions are 7.5 m and 1 min, respectively. For consistency, the overlap of these observations unifies above 300 m, and the signals below 300 m are not shown in this paper. Figures 5(a)–5(c) depict the ABCs retrieved by the standard Fernald, EnKF, and PF methods. The patterns in the figure are similar, which indicate that both the EnKF and PF are as stable as the standard Fernald method. Figure 6(a) presents the means and standard deviations of the original signals and the de-noised signals with EnKF and PF, respectively. Figure 6(a) demonstrates that noise is eliminated by the EnKF and PF methods. The statistical differences of the real cases are not as significant as those of the simulated case, because the real signals are also affected by the variation of the atmospheric backscatter coefficient and the two-way transmittance as accorded to the lidar principle of Eq. (1), while the backscatter and the transmittance are constants in the simulation. Figure 6(b) depicts the statistical information of the ABCs retrieved by the standard Fernald, EnKF, and PF methods. On the one hand, in the range below 4 km, the standard deviations of the standard Fernald, EnKF, and PF methods are comparable. On the other hand, above 4 km, the standard deviations of the de-noising methods are smaller than those of the standard Fernald method, which conforms to the conclusion of the synthetic experiments. Furthermore, it shows that the standard deviation of the EnKF is smaller than that of the PF in the range above 6 km in Fig. 6(b).

Fig. 5. (a), (b), and (c) are the sequential aerosol backscatter coefficients retrieved by the standard Fernald, EnKF, and PF methods, respectively.

Fig. 6. (a) Mean and error bars (standard deviation) for the observational and the de-noised signals, which are normalized and range corrected, i.e., ln[X(r)], respectively. (b) Mean and error bars (standard deviation) of the ABC retrieved by the standard Fernald method and our methods. To avoid overlay, the error bars of (a) and (b) are presented with an interval of 40 bins apart. Moreover, the error bars above 8 km in (b) are hidden partly because negative numbers are not displayed in the figure with a logarithmic axis.

#245540 (C) 2015 OSA


To further compare the performance of the standard Fernald method and two de-noising methods, we show the results of six cases of aerosol backscatter retrieved by the three methods with the original signals. The six one-minute signals are the signals at 22:00 to 5:30 with a 1.5-hour interval. As shown in Figs. 7(a)–7(f), for one-minute signals, the aerosol backscatter retrieved from the signals is smooth below 4 km. However, the signals are severely perturbed by noise above 4 km because of the decrease of the 1/r2 effect and clearer atmosphere. The ABCs retrieved by both the EnKF and PF not only have smaller uncertainty above 4 km but also agree with the profile of the standard Fernald method below 4 km. The result of the EnKF method may be slightly different from our previous result [16] because we updated the background-noise removal and calibration methods in the new version of our lidar package.

Fig. 7. (a–f) ABC retrieved by the standard Fernald, EnKF, and PF methods from six signals at 22:00 to 5:30 with a 1.5-hour interval.

To compare the detail of retrieval results for the standard Fernald, EnKF, and PF methods, the detail of ABCs below 4 km are presented in Fig. 7 in the range 2–4 km and 0.5–2 km, respectively. Moreover, ABC changes largely as seen in Figs. 8(a)–8(f), which are enlarged subsection views of Figs. 7(a)–7(f). In comparison with the retrieved result of the one-minute profile, both results of EnKF and PF show over-smoothing in many peaks of the layer inflection points, whereas the PF method can more accurately track the one-minute result. In the range with a high signal quality below 2 km, the bias caused by the shift can be larger than the de-noising effects, as shown in Figs. 8(a)–8(c). Biases exist in the result of EnKF because of layer inflection, whereas no biases exist in the result of PF. That is, the de-noising process with the EnKF seems to be unnecessary, but the PF performs better in this range. In the range between 2 and 4 km, where the signal quality is still acceptable, noise still exists in the result retrieved with the standard Fernald method. Unlike the experiments with simulated signals, we have no true value with which to assess the performance of the methods through statistic information, such as RMSE, SNR, and R2. Therefore, we calculated the bias to assess the methods’ performance by assuming that the results of the standard Fernald method are the “true” values and that the random errors of the results are negligible in the high SNR range between 0.3 and 2 km. The relative bias (shift) of the results of the EnKF and PF are 5.72%

#245540 (C) 2015 OSA


and 2.15% on average, respectively, which demonstrates that the PF method can keep the layer’s structure and distinguish close layers much better than the EnKF method in this range.

Fig. 8. (a–f) Enlarged subsection views of Figs. 7(a)–7(f), respectively.

The mean of the standard deviation of the retrieval results above 2 km are described in Table 2. The wavelet is used in this case for comparison again, and the result of the wavelet is close to that of PF. In addition, the results retrieved by the standard Fernald method are calculated based on the averaged signals of 16 and 64 minutes, which are centered on the oneminute signal for comparison. For the far range, the result of the EnKF is close to that of the average of 64 signals, which may halve the standard error of the signal three times [16]. At the same time, the result of PF is close to that of the average of 16 signals. Overall, the result shows that the EnKF performs better than the PF in reducing noise above 4 km. Furthermore, we can combine near range results of PF method and far range results of EnKF method as alternative retrieval results. In the following experiment, we select 4−5 km as combining range by referring to the range for signal splicing in [24]. A joint is chosen at where the retrieval results of the PF and EnKF are closest to each other to avoid artificial mutation in the combining results. As shown in Table 2, the mean of the standard deviation of the combined results in the splicing range is between that of EnKF and PF, and the higher and lower results are the same as EnKF and PF, respectively. In this way, the combined retrieval and de-noising results eliminated the relative bias in the near range, but also significantly suppress the noise in the far range. In the range 2–4 km some minor details of the retrieval results are eliminated by the EnKF, whereas the PF method has kept those details. The over-smoothing as shown in Figs. 8(d)–8(f) can also occur in the far range for the EnKF by comparing results of the PF and wavelet. Thus, both the EnKF and PF methods can provide acceptable results, but the EnKF leads to a shift of a few bins because of the effects of the poor forecast, which can be still residual in the retrieval of a few lower bins, and the combined results can also be given for reference.

#245540 (C) 2015 OSA


Table 2. Mean of the standard deviation (10−8) for the retrieval backscatter coefficient in the range above 6 km Range

Standard Fernald

16 min

64 min

Wavelet

EnKF

PF

Combination

2–3km

3.069E-07

3.050E-07

2.984E-07

3.063E-07

2.968E-07

3.046E-07

3.046E-07

3–4km

8.036E-08

7.639E-08

7.305E-08

7.751E-08

7.433E-08

7.690E-08

7.690E-08

4–5km

6.136E-08

5.416E-08

4.800E-08

5.630E-08

4.754E-08

5.452E-08

5.253E-08

5–6km

4.735E-08

3.549E-08

2.386E-08

3.897E-08

2.622E-08

3.676E-08

2.622E-08

6–7km

5.212E-08

3.873E-08

2.540E-08

4.265E-08

2.742E-08

3.995E-08

2.742E-08

7–8km

5.561E-08

4.044E-08

2.502E-08

4.483E-08

2.668E-08

4.147E-08

2.668E-08

8–9km

5.713E-08

4.082E-08

2.262E-08

4.538E-08

2.461E-08

4.179E-08

2.461E-08

9–10km

5.920E-08

4.176E-08

2.382E-08

4.682E-08

2.655E-08

4.207E-08

2.655E-08

>10km

6.184E-08

4.399E-08

2.419E-08

4.967E-08

2.660E-08

4.393E-08

2.660E-08

4. Summary

Lidar data retrieval methods are essential for lidar data application, which needs to be not only insensitive to noise but can also accurately keep the real character of the atmosphere. In this study, an alternative de-noising method is proposed to avoid the over-smoothing of other methods based on the PF and the Fernald methods. The results of both simulated and real signals demonstrate that the performance of standard Fernald method is satisfactory in the near range but unacceptable in the far range. An average strategy will reduce the temporal or range resolution of the lidar signal, which is not fully appropriate for studying atmospheric structure and its time evaluation. Our previous method, combining EnKF and the standard Fernald method, can obtain effective retrieval accuracy at far range, but leads to a larger shift (bias) in the near range because of the over-smoothing the tendency change of aerosol variation. However, the simulated experimental results show that PF performs better than EnKF with a lower RMSE and higher SNR and R2. Retrieval and de-noising results with real signals illustrate that relative bias is eliminated using the PF in the near range, but also effectively suppresses the noise in the far range. Furthermore, we carried out an experiment by combining near range results of PF method and far range results of EnKF method, which result can be an effective alternative because which not only has small relative bias in the near range, but also has less random noise in the far range. Generally, the retrieval results show that the PF method can obtain more accurate backscatter coefficients than the other single methods, and combining results of different retrieving methods can be an effective alternative for future applications. Acknowledgments

This research is supported by the National Science Foundation of China (41127901, 61273215), the Program for Innovative Research Team in University of Ministry of Education of China (IRT1278), the National Science Foundation of Hubei province (2015CFA002), the China Postdoctoral Science Foundation (2015M570667), and the Fundamental Research Funds for the Central Universities (2042015kf0015).

#245540 (C) 2015 OSA
