Classification of Rice Heavy Metal Stress Levels

sensors Article

Classification of Rice Heavy Metal Stress Levels Based on Phenological Characteristics Using Remote Sensing Time-Series Images and Data Mining Algorithms Tianjiao Liu, Xiangnan Liu *, Meiling Liu and Ling Wu School of Information Engineering, China University of Geosciences, Beijing 100083, China; [email protected] (T.L.); [email protected] (M.L.); [email protected] (L.W.) * Correspondence: [email protected]; Tel.: +86-10-8232-3056; Fax: +86-10-8232-2095 Received: 31 October 2018; Accepted: 12 December 2018; Published: 14 December 2018







Abstract: Heavy metal pollution in crops leads to phenological changes, which can be monitored by remote sensing technology. The present study aims to develop a method for accurately evaluating heavy metal stress in rice based on remote sensing phenology. First, the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) was applied to blend Moderate Resolution Imaging Spectroradiometer (MODIS) and Landsat to generate a time series of fusion images at 30 m resolution, and then the vegetation indices (VIs) related to greenness and moisture content of the rice canopy were calculated to create the time-series of VIs. Second, phenological metrics were extracted from the time-series data of VIs, and a feature selection scheme was designed to acquire an optimal phenological metric subset. Finally, an ensemble model with optimal phenological metrics as classification features was built using random forest (RF) and gradient boosting (GB) classifiers, and the classification of stress levels was implemented. The results demonstrated that the overall accuracy of discrimination for different stress levels is greater than 98%. This study suggests that fusion images can be utilized to detect heavy metal stress in rice, and the proposed method may be applicable to classify stress levels. Keywords: heavy metal stress; time-series; remote sensing phenology; MODIS and Landsat; ensemble model; feature selection

1. Introduction The present situation of heavy metal pollution in China is severe, which not only poses a serious threat to agricultural product quality, and thus harms human health, but also leads to social instability [1]. The national soil survey jointly released by the Ministry of Land and Resources and the Ministry of Environmental Protection showed that the over-standard rate of heavy metals in cultivated land was 19.4%. According to statistics, more than 10 Mt of grain are lost every year in China due to heavy metal pollution [2,3]. Rice is the crop with the largest area in China and cadmium (Cd, and Cd hyperaccumulation), as one of the most toxic elements to the human body, has become an important limiting factor on rice quality. Cd has been extensively studied and its hazard to human health is regularly reviewed by international organizations such as the World Health Organization (WHO). Cd accumulates primarily in the kidneys, leading to renal tubular dysfunction and the formation of kidney stones. High intake of Cd can lead to disorder in calcium metabolism, which results in osteomalacia, osteoporosis, and painful bone fractures. There is sufficient evidence that long-term exposure to Cd increases the risk of lung cancer, kidney cancer, and prostate cancer [4,5]. In recent years, China has identified the prevention and control of heavy metal pollution in cultivated land as Sensors 2018, 18, 4425; doi:10.3390/s18124425

www.mdpi.com/journal/sensors

Sensors 2018, 18, 4425

2 of 19

the key task of development planning [6,7]. Therefore, it is of great significance to monitor heavy metal stress accurately and timely. Traditional methods for monitoring heavy metal contamination, such as sample collection and laboratory analysis, can obtain heavy metal concentrations accurately, and lay a good theoretical foundation for studying the remote sensing mechanism of heavy metal pollution. However, it is difficult to establish a wide range of contamination monitoring in the short term, and meanwhile, long-term field trials add complexity and cost. Remote sensing (RS), with the merits of huge observation scope, cost-effectiveness, real-time effects and non-destruction, has become an efficient method for monitoring heavy metal pollution in crops [8–11]. As the change in a plant’s physiological elements, such as chlorophyll content, cell structure, and water or nitrogen content, can be monitored by the reflection spectrum, some research on heavy metals in rice have been carried out to identify the relationship between sensitive spectral characteristics and heavy metal concentrations or physiological factors by establishing empirical or semi-empirical models. However, the spectral data only reflected the stress state of one or several growth stages, which may lead to randomness. Still some researchers established the assimilation framework based on RS and the crop growth models to achieve the dynamic monitoring of heavy metal stress from the aspects of physiological functions. In such studies, the dry weight of crop roots (WRT) is generally considered as a good indicator to evaluate stress levels, yet deficiency exists in reduced sensitivity to heavy metal stress as roots age. Many studies have shown that heavy metal cadmium (Cd) poisoning can induce short leaves, decreased chlorophyll content, serious yellowing, and delayed maturity. In addition, Cd stress affects the water content of rice by reducing water absorption [12–15]. Combining spectral, temporal, and spatial information, the phenological features extracted using remote sensing technology reflect continuous growth and the stress state of rice throughout the entire growth stage. Time-series and phenological characteristics have been proved to be useful to monitor heavy metal stress in rice [10,16–18]. Phenological detection is implemented through analysis of time series of remotely-sensed images [19–22]. One of the most commonly used datasets in phenology studies is collected by the Moderate Resolution Imaging Spectroradiometer (MODIS) sensors on the Terra and Aqua satellites. However, it is challenging to detect changes at small scales or in heterogeneous landscapes using moderate resolution data. Landsat data are insufficient from a temporal perspective due to cloud contamination or revisit cycle limitation. An alternative solution is to develop a high spatial and temporal resolution multi-source remote sensing data fusion method [22–25]. Zhu et al. proposed the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) algorithm to make use of the correlation to blend multi-source data and, meanwhile, to minimize systemic biases and enhance the accuracy of predicting reflectance in changing, heterogeneous landscapes [26]. Since the 1970s, many researchers have recognized the potential of multi-temporal satellite observations to provide information about the phenological development of natural vegetation and crops. VIs are designed to take advantage of spectral reflection/absorption characteristics of plants to enhance the vegetation signal and allow dependable spatial-temporal inter-comparisons in terrestrial photosynthetic activity and canopy structural variations [27,28]. Therefore, the time-series data of vegetation index with distinct seasonal rhythm are suitable for phenological study [29]. In previous phenological-based stress detection studies, indices reflecting greenness information (e.g., Normalized Difference Vegetation Index, NDVI; Enhanced Vegetation Index, EVI) or moisture information (e.g., Normalized Difference Water Index, NDWI) were used independently, which may ignore some critical information. Therefore, it is important to monitor heavy metal stress by combining greenness and moisture information. Because the original temporal vegetation index (VI) profiles include various noise components, such as aerosols and bidirectional reflectance distribution factors, noise reduction or fitting techniques, such as the Savitzky-Golay (S-G) filter, Fourier analysis, and asymmetric Gaussian and double logistic fitting are applied to reconstruct the original VI time series before application [30,31]. Some researchers have compared these smoothing models,

Sensors 2018, 18, 4425

3 of 19

and revealed that the Whittaker smoother (WS) had a consistently superior performance in most cases [32–34]. From the smoothed VI time profiles, phenological periods can be estimated by the derivation method [35,36], but this method is unable to accurately determine the transplanting period, and the algorithm that combines rice transplanting signals and agronomic rules provides a new method for the extraction of phenological stages [37–39]. Previous studies established one or several phenological indicators for heavy metal stress monitoring, making it difficult to effectively exploit phenological information. Moreover, the discriminative features have been constructed manually. Thus, we need to probe for the phenological characteristics and apply some automatic feature extraction methods to stress detection. In most cases, the NDVI or EVI was used empirically for estimation, which rarely determined the optimal VIs for stress estimation. Therefore, the key is to select the optimal discriminative features from this large feature set. Küçük et al. [40] applied support vector machine recursive feature elimination (SVM-RFE) to select the top four features for phenology monitoring. The result showed the feasibility of machine learning (ML) algorithms in feature selection. At present, ML algorithms, such as support vector machines (SVM), decision trees (DT) and random forests (RF) have been utilized in classification [41–45], and the distinction of heavy metal stress can be considered a classification problem. The goal of classification problems is to find a model that best predicts the desired data. The ensemble method involves multiple models and fuses them into a final model, which generally achieves better predictions [44]. Based on the analysis above, a scheme for stress evaluation was explored in this work. In addition to mining the annual temporal change in VIs using time series analysis methods, we took advantage of crop greenness, moisture condition, and corresponding stress characteristics during the whole growth period. In the following sections, we gave a detailed description of our stress level classification method and present its application in Zhuzhou City, Hunan Province, China using fusion images generated by MODIS and Landsat time-series datasets. 2. Materials 2.1. Study Area Our study area, ranging between 26◦ –28◦ North and 112.5◦ –114◦ East, is located in Zhuzhou City, Hunan Province, China. The region has a subtropical monsoon humid climate with abundant rainfall, and adequate light and heat for rice growth, and the predominant soil type is red soil with sufficient organic matter (2–3%). These climatic and soil conditions promote high yields of rice, making it an important grain commodity base. However, large areas have been contaminated by heavy metals due to sewage irrigation from Xiang Jiang River [46,47]. Previous studies have shown that Cd is the predominant pollutant in paddy soils that are watered from the Xiang Jiang River, which contains industrial wastewater [46,47]. The contaminated paddy fields have resulted in rice growth that is stressed by heavy metals. Six study sites were selected in the research area (Figure 1). According to the content of the heavy metal Cd, the heavy metal stress levels in rice at these six sites were classified as non-stress, moderate stress and severe stress, respectively (Table 1). The same rice type (Boyou 9083) is under intensive cultivation patterns in all sites to ensure adequate irrigation and sufficient fertilizers in paddy fields without pests, weeds, or other environmental issues.

Sensors 2018, 18, 4425

4 of 19

Table 1. Heavy metal Cd concentration of the six study sites. Study Sites A-1 A-2 C-1 C-2 D-1 D-2

Geographic Location 113◦ 120 113◦ 100 113◦ 140 113◦ 100 113◦ 060 113◦ 040

E 27◦ 470 E 27◦ 470 E 27◦ 370 E 27◦ 400 E 27◦ 450 E 27◦ 490

N N N N N N

Cd Concentration in Soil 0.84 0.84 2.25 2.31 3.27 3.54

Background Value

1.43

National Quality Standard

Pollution Level

0.3–1.0

None None Moderate Moderate Severe Severe

Note: The unit of Cd concentration is mg/kg. The quality standard of soil environment is used to evaluate pollution levels. Cd background value was obtained from the Hunan Institute of Geophysical and Geochemical Exploration, Sensors 2018, 18, xChina. FOR PEER REVIEW 4 of 19

Figure Zhuzhou, Hunan HunanProvince, Province,China. China. Figure1.1.Location Locationmap mapof ofthe thesix six study study sites in Zhuzhou,

2.2. Data Preparation 2.2. Data Preparation The experiment growing stages stagesin in2013. 2013.The Thedata dataincluded included The experimentwas wascarried carriedout outduring duringthe the entire entire rice rice growing remote sensing images and field measurements. We downloaded Landsat 8 OLI (Operational Land remote sensing images and field measurements. We downloaded Landsat 8 OLI (Operational Land Imager) and Landsat Mapper Plus) Plus)Level-2 Level-2surface surfacereflectance reflectanceproducts products Imager) and Landsat7 7ETM+ ETM+(Enhanced (Enhanced Thematic Thematic Mapper in in thethe study area from early May to late October from the United States Geological Survey (USGS) study area from early May to late October from the United States Geological Survey (USGS) EarthExplorer andthen thenperformed performedlayer layerstacking stackingand andregional regional EarthExplorer(http://earthexplorer.usgs.gov/), (http://earthexplorer.usgs.gov/), and clipping onon Landsat the Landsat-7 Landsat-7airborne airbornescan scanline linecorrector corrector clipping Landsatdata. data.InInaddition, addition,due due to to the the failure of the (SLC) inin May 2003, images.The TheMOD09A1 MOD09A1product productis is (SLC) May 2003,gapfill gapfillprocessing processingwas wasrequired required for for the the ETM+ ETM+ images. 8-day composite surface reflectance data withspatial spatialresolution resolutionofof500 500m.m.Zhuzhou ZhuzhouCity Cityisiscovered coveredby 8-day composite surface reflectance data with bytiles two(h28v06 tiles (h28v06 and h27v06) of MOD09A1 We obtained the from two National tiles fromAeronautics National two and h27v06) of MOD09A1 data. Wedata. obtained the two tiles Aeronautics and Space (NASA) Administration (NASA) (https://ladsweb.modaps.eosdis.nasa.gov/) and and Space Administration (https://ladsweb.modaps.eosdis.nasa.gov/) and conducted image conducted image mosaic and reprojection to MODIS data using Modis Reprojection Tool (MRT) mosaic and reprojection to MODIS data using Modis Reprojection Tool (MRT) software. Furthermore, in order to implement the ESTARFM the order MOD09A1 data in software. order to Furthermore, implement the ESTARFM algorithm, the order algorithm, of MOD09A1 data of layer stacking was layer stacking was adjusted to match the order of Landsat data. The ESTARFM algorithm is the key adjusted to match the order of Landsat data. The ESTARFM algorithm is the key to generating to generating synthetic requires at least two pairs of fine- and coarse-resolution imageson synthetic images, which images, requireswhich at least two pairs of fineand coarse-resolution images obtained obtained on the same date, as well as a set of coarse resolution images for prediction. The algorithm the same date, as well as a set of coarse resolution images for prediction. The algorithm implementation implementation includes four components: Search similar pixels based on two ETM+ or OLI includes four components: (1) Search similar (1) pixels based on two ETM+ or OLI images at different images at different times; (2) compute the weights (Wi) of all similar pixels; (3) decide the conversion times; (2) compute the weights (Wi) of all similar pixels; (3) decide the conversion coefficients (Ci) by coefficients (Ci) by linear regression; and (4) predict the fine-resolution reflectance from MOD09A1 linear regression; and (4) predict the fine-resolution reflectance from MOD09A1 image on the desired image on the desired date by Wi and Ci. Moreover, according to the pixel value interpretation of date by Wi and Ci. Moreover, according to the pixel value interpretation of quality assessment (QA) quality assessment (QA) information distributed by USGS, we masked out the pixels that might be information distributed by USGS, we masked out the pixels that might be affected by instrument affected by instrument artifacts or cloud contamination for subsequent research [48,49]. artifacts or cloud contamination for subsequent research [48,49]. We conducted a field survey in six study sites of the research area. In each study site, the latitude We conducted a field survey in six study sites of the research area. In each study site, the latitude and longitude coordinates at the center of each sample plot were measured by GPS. In each plot, the and longitude at the were centersimultaneously of each samplecollected plot wereand measured by GPS. In each plot, the rice rice samplescoordinates and soil samples preserved in sample bags and soil samples samples collectedofand in sample bags and soilEach boxes, boxes, and andsoil then sent towere the simultaneously laboratory for analysis thepreserved physicochemical characteristics. sampled plot contained four subplots, and we sampled 30 g of soil and a whole rice plant in each subplot. The soil of four subplots in each sample plot was mixed for heavy metal measurement; that is, the average heavy metal content of soil samples in each plot was taken as its heavy metal content. The heavy metal content in the soil was analyzed at the Chinese Academy of Agricultural Sciences. 0.5 g of soil samples were put into polytetrafluoroethylene digestion tube with 5 mL of hydrochloric

Sensors 2018, 18, 4425

5 of 19

and then sent to the laboratory for analysis of the physicochemical characteristics. Each sampled plot contained four subplots, and we sampled 30 g of soil and a whole rice plant in each subplot. The soil of four subplots in each sample plot was mixed for heavy metal measurement; that is, the average heavy metal content of soil samples in each plot was taken as its heavy metal content. The heavy metal content in the soil was analyzed at the Chinese Academy of Agricultural Sciences. 0.5 g of soil samples were put into polytetrafluoroethylene digestion tube with 5 mL of hydrochloric acid and 10 mL of nitric acid (excellent grade), and then placed overnight. On the next day, 2 mL of hydrofluoric acid and 2 mL of perchloric acid were added to digest the samples by heating. Finally, 2–3 mL of liquid was transferred to a 50 mL plastic volumetric flask for cooling and constant volume, and the metal content was determined by inductively coupled plasma mass spectrometer ICP-MS (Model: Agilent 7900, Agilent Technologies, Santa Clara, CA, USA). 3. Methods A method for accurately monitoring rice under heavy metal stress at the region scale was proposed (Figure 2). It focuses on the following procedures: (i) calculate vegetation indices and create time series; (ii) design phenological parameters based on changes in rice phenology under heavy metal stress using Matlab and Tsfresh package; (iii) select an optimal phenological feature subset from the original feature set, and establish an ensemble model based on machine learning algorithms to classify heavy metal stress levels in rice; and, (iv) evaluate the accuracy of classification results. 3.1. Construction of VI Time Series for Phenological Analysis Images with cloud cover of less than 30% in the research area were used as inputs of the ESTARFM algorithm, and a series of fusion images was obtained. Depending on the QA bands, we masked out the pixels that exhibited instrument artifacts or cloud contamination, and removed the images with more bad-quality rice pixels. Finally, we obtained 22 images, including original Landsat images and the synthetic images, which were combined to create time series data in chronological order. For each image, we calculated four vegetation indices—(a) NDVI, (b) EVI, (c) NDWI(1) and (d) NDWI(2)—using the land surface reflectance values of the blue (ρblue ), green (ρ green ), red (ρred ), NIR (ρnir ), SWIR1 (ρswir1 ), and SWIR2 (ρswir2 ) bands. NDVI and EVI were selected for their performance in detecting canopy greenness. We also applied NDWI(1) and NDWI(2), which are sensitive to leaf water and soil moisture [50–52]. The spectral indices were calculated using the following equations: NDVI = (ρnir − ρred )/(ρnir + ρred )

(1)

EVI = 2.5 ∗ (ρnir − ρred )/(ρnir + 6 ∗ ρred − 7.5 ∗ ρblue + 1)

(2)

NDWI(1) = (ρnir − ρswir1 )/(ρnir + ρswir1 )

(3)

NDWI(2) = (ρnir − ρswir2 )/(ρnir + ρswir2 )

(4)

The Whittaker smoother is based on penalized least squares [53]. It fits a discrete series to discrete data and puts a penalty on the roughness of the smooth curve [54]. The fitting effect, Q, depends on the fidelity to original data, the roughness of smoothed data, R, and the smoothing parameter, k. The aim of penalized least squares is to find the series, z, that minimizes Q. Suppose a noisy series y. The smoother z is, the more it will deviate from y. The larger the parameter k, the stronger the influence of R on the goal Q. Some researchers have found that setting the k value to 2 has a higher fitting accuracy [55,56]. Accordingly, we set the k value to 2 in this study. The smoothing program was run in Matlab software. The algorithm is extremely fast, provides continuous control over smoothness with only one parameter, and interpolates automatically. The filtering efficiently removed the negatively biased noise present in the original data, while preserving the overall shape of the curves showing vegetation growth and development [56]. It has shown a high potential for remotely sensed time series filter and phenological study [32,56]. Therefore, the WS was chosen in this research.

Sensors 2018, 18, 4425

6 of 19

Sensors 2018, 18, x FOR PEER REVIEW

6 of 19

Figure 2. Overview of methodology.

3.2. Designing Phenological Phenological Metrics 3.2. Designing Metrics from from Seasonal Seasonal Patterns Patterns of of VIs VIs We extracted four four key keyphenological phenologicalphases, phases,that that transplanting, tillering, heading, We extracted is, is, ricerice transplanting, tillering, heading, and and maturity. For each growth cycle, the heading date was first identified based on maximum value. maturity. For each growth cycle, the heading date was first identified based on maximum value. In In transplanting phase, rice paddy fieldsare area amixture mixtureofofwater waterand andgreen green rice rice plants, plants, and and NDWI thethe transplanting phase, rice paddy fields NDWI values EVI values; values; specifically, specifically, here values are are larger larger than than NDVI NDVI or or EVI here we we recognized recognized the the transplanting transplanting signals signals using EVI or NDVI [37–39]. [37–39]. We We selected using the the criteria criteria NDWI NDWI > > EVI or NDWI NDWI > > NDVI selected the the date date with with the the maximum maximum value value of of the the first first derivative derivative to to identify identify the the active active tillering tillering phase, phase, and and the the date date that that corresponded corresponded to to zero of the second derivative was considered to be the maturity stage [35,36]. Studies [12–17] have zero of the second derivative was considered to be the maturity stage [35,36]. Studies [12–17] have shown (1) when when the the rice rice is is exposed to heavy shown that: that: (1) exposed to heavy metal metal stress stress (e.g., (e.g., from from Cd), Cd), enzyme enzyme required required for for chlorophyll and chlorophyll contentcontent decreased, resultingresulting in chlorosis chlorophyll formation formationis inhibited, is inhibited, and chlorophyll decreased, insymptoms chlorosis in rice, which performed in the NDVI/EVI time-seriestime-series is the reduction maximum and minimum symptoms in rice, which performed in the NDVI/EVI is the of reduction of maximum and NDVI/EVI values; (2) the toxicity of heavy metals can influence the ability of organs and minimum NDVI/EVI values; (2) the toxicity of heavy metals can influence the abilitytoofaccept organs to convert photosynthetic products, resulting in reduced rate and the length of growth season; accept and convert photosynthetic products, resultinggrowth in reduced growth rate and the length of (3) the toxicity heavy metals can also lead to other changes, such as postponed growth season;of(3) the toxicity of heavy metals can phenological also lead to other phenological changes,heading such as date, decreased water content, delayed greening and maturation. Therefore, for unstressed rice, the postponed heading date, decreased water content, delayed greening and maturation. Therefore, for EVI or NDVI values at heading dates and magnitude of the EVI or NDVI change might be larger than unstressed rice, the EVI or NDVI values at heading dates and magnitude of the EVI or NDVI change for stressed rice,than and the time range from ofrange transplanting theofheading date might beheading shorter might be larger for stressed rice, andthe theend time from theto end transplanting to the compared rice. Moreover, changes water content in non-stressed rice in during the rice date mightto bestressed shorter compared to stressed rice.ofMoreover, changes of water content non-stressed growth period are generally different from those in stressed rice. We built phenological indicators rice during the rice growth period are generally different from those in stressed rice. We built based on the phenological differences under heavy differences metal stress. phenological indicators based on the phenological under heavy metal stress. We We calculated calculated the the annual annual average, average, maximum maximum and and minimum minimum values values of of VIs, VIs, and and extracted extracted some some phenological parameters with reference to TIMESAT [57]. Furthermore, two growth stages phenological parameters with reference to TIMESAT [57]. Furthermore, two growth stages were were determined the early early growth growth stage stage was was defined determined based based on on the the estimated estimated phenological phenological periods: periods: the defined as as the the period from estimated transplanting to heading date, and the late growth stage was defined as the period from estimated transplanting to heading date, and the late growth stage was defined as the period Some phenology-based phenology-based indicators indicators were were period from from the the estimated estimated heading heading date date to to the the maturity maturity date. date. Some designed Reed et et al. al. [27] [27] verified verified that that these these indicators indicators may may designed during during the the early early or or the the late late growth growth stage. stage. Reed not necessarily directly correspond to conventional, ground-based phenological events, but show not necessarily directly correspond to conventional, ground-based phenological events, but show

strong coincidence with expected phenological characteristics. It has been proved that phenology can be a practical indicator for heavy metal stress in rice plants [16,17]; the significance of these indicators

Sensors 2018, 18, 4425

7 of 19

strong coincidence with expected phenological characteristics. It has been proved that phenology can be a practical indicator for heavy metal stress in rice plants [16,17]; the significance of these indicators lies in the possibility to map out phenological changes in vegetation [58]. Basic details of the phenological signatures are listed in Table 2. To further mine the hidden phenological information in VI time series, we extracted features automatically by time-series feature extraction method. Time-series feature extraction is a time-consuming process because researchers have to consider the multifarious algorithms of signal processing and time-series analysis for identifying and extracting meaningful features from time series. The Python package Tsfresh accelerates this process by combining 63 time-series characterization methods, and automatically calculates a large number of time series characteristics (794 time series features by default) [59,60]. The Tsfresh package has a built-in filtering procedure, which mathematically removes the extracted null-value features [59,60]. In this study, we automatically obtained thousands of features employing the Tsfresh package. Table 2. List of phenological signatures manually established in this study. Feature Category

Parameters

Number of Parameters

Phenological signatures

Annual average, maximum and minimum values of VIs Base level, seasonal amplitude, seasonal integral, seasonal length and growth ratio VIs(heading)-VIs(maturity) VIs(heading)-VIs(tillering) T(heading)-T(transplantend); (NDWImax-NDWImin)/(VI(heading)-VI(maturity)) (NDWImax-NDWImin)/(VI(heading)-VI(tillering))

18 10 6 6 2 2 2 46

Total

Note: VIs refers to NDVI, EVI, NDWI(1), NDWI(2), NDVI2 +NDWI(2)2 and EVI2 +NDWI(1)2 . VIs(heading), VIs(maturity), VIs(tillering) refers to the VIs values in heading, maturity and tillering date respectively. VI(heading), VI(maturity) and VI(tillering) refers to the EVI or NDVI values in heading, maturity and tillering date respectively. T(heading) and T(transplantend) refers to the dates in heading and the end of transplant respectively.

3.3. Classification of Heavy Metal Stress Levels in Rice The hyper-dimensional feature space consisted of thousands of remote sensing features. It was necessary to apply an optimal feature selection strategy to this space to reduce redundancy and computation, and the optimal feature subsets inherited the original physical/mathematical meanings of the features [61]. The normalized feature selection scikit-learn exposes feature selection routines as objects that implement the transform method [62]. SelectKBest, as a class in the scikit-learn library, can be used for feature selection. We applied SelectKBest to obtain a preliminary feature subset. The analysis of variance (ANOVA) F-value is the default function of SelectKbest, which has been used in statistical discriminant analysis. The F-value is an statistic which estimates the significance of variables participating discriminant efficiency [63,64]. We implemented feature ranking based on the F-value, the large F-values were preferred, and then went backwards, until the desired number of features were obtained. Then, we executed feature selection through recursive features elimination with cross-validation (RFECV). RFE involves selecting features by recursively considering smaller and smaller sets of features, that is, acquiring the importance of each feature and pruning the least important features from the current feature set [40]. This procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached. RFECV performs RFE in a cross-validation loop to find the optimal features. As a result, an optimal feature subset was built for the classification task. We conducted classification and regression tree (CART) classification on the Landsat 8 OLI image on 17 September 2013, extracted the farmland areas, and then selected rice pixels based on a field survey and Google Earth. Consequently, 1838 rice pixels were obtained for research. Previous work has shown that using boosting and bagging ensemble classifiers achieved greater accuracy than using single classifiers, and was more stable and robust to noise in the training data [44,65]. The random forest (RF) classifier is a good example of the bagging method, and the gradient boosting (GB) classifier is based on the boosting method. We built an ensemble model for the classification of heavy metal stress levels using RF and GB classifiers, that is, firstly, we respectively exploited the two classifiers (RF and

Sensors 2018, 18, 4425

8 of 19

GB) to calculate the probabilities that each rice pixel belongs to each stress level, then averaged the probabilities obtained by these two classifiers and took these average values as the final probabilities of the pixel. In addition, in order to set the parameter values with the best classification effect in the classifiers, we exhaustively considered all parameter combinations and systematically traversed a variety of parameter combinations, and finally determined the best performance parameter values through cross-validation. 3.4. Accuracy Assessment We randomly split the original data (1838 rice pixels) into two categories (training set and test set) according to the ratio of 7:3. Training data were used to fit classification models, and testing data were used to evaluate the classification accuracy through the trained model. Cross-validation is a good performance evaluation method in the case of limited data. Here we used 3-fold cross-validation to divide the training set into three equal parts, two of which were used as training sets, the remaining one was used as a verification set. We obtained the prediction results for each validation set, then averaged the correct rates obtained from these three validation sets as the standard to measure the performance of the trained model. In the end, the distinction accuracy for heavy metal stress levels was assessed on the basis of indicators, including the overall accuracy, confusion matrix and the receiver operating characteristic (ROC) curve. Overall accuracy indicates the proportion of all predictions that are correct [66]. A confusion matrix gives a visual representation of the quality of the discrimination results [67]. The ROC is a graphical plot which illustrates the performance of classification model as the discrimination threshold is varied [68]. 4. Results 4.1. Rice Growth Trajectory under Different Heavy Metal Stress Figure 3a presents the fitting results of the WS model with smoothing parameter k = 2. It can be seen that the reconstructed curve is smooth and fits the original data well, which conforms to the growth and development of rice. The statistical measures, such as mean absolute deviation (MAE), agreement coefficient (AC), root mean square error (RMSE) and correlation coefficient, were utilized to evaluate the WS model. These four parameters were calculated for each rice pixel between the original data and fitted data, which are 0.0413, 0.8594, 0.0414 and 0.9516, respectively. The smaller the RMSE is, the better the fit would be, and the large correlation coefficient value indicates a close fit. Hence, the WS model performed well in filtering remote sensing time series. As shown in Figure 3b, during a rice growing cycle, the EVI values during the rice transplanting and tillering period could be considerably lower compared to NDWI values due to irrigation, and generally increased with tillering, then reached their peaks in the heading phase. Approximately 50 days after transplanting, most of the rice paddy fields were fully covered by the rice canopy, and NDWI values were lower than those of EVI. At the end of the maturation stage, the rice canopy has lower leaf and stem moisture content and more senescent leaves. Therefore, the EVI values gradually declined as rice was harvested. Similar to the EVI temporal profile, we observed a slight decrease before maturation, and a remarkable decrease after maturation, in the NDWI time series curve. Hence, the parameters based on single or combined variations of EVI/NDVI and NDWI during specific phenological periods could be designed to indicate the growth status of rice.

Sensors 2018, 18, 4425

9 of 19

Sensors 2018, 18, x FOR PEER REVIEW

9 of 19

Sensors 2018, 18, x FOR PEER REVIEW

9 of 19

Figure 3. (a) The raw EVIdata dataand andthe thereconstructed reconstructed EVI byby WS. (b)(b) The keykey Figure 3. (a) The raw EVI EVI time-series time-seriescurve curvefitted fitted WS. The phenology-based dates extracted from temporal profiles of EVI and NDWI(1). phenology-based dates from temporal profiles EVI and NDWI(1). Figure 3. (a) The rawextracted EVI data and the reconstructed EVIoftime-series curve fitted by WS. (b) The key phenology-based dates extracted from temporal profiles of EVI and NDWI(1).

curvesin inFigure Figure4a,b 4a,bwere were plotted plotted by by averaging averaging the at at thethe TheThe curves the VI VIvalues valuesofofrice ricepixels pixels corresponding study site, which showed growth characteristics of rice, as well as the difference corresponding study which growth of rice, as of well the difference The curves in site, Figure 4a,b showed were plotted by characteristics averaging the VI values riceaspixels at the information about rice growth growth under different heavy metal stressstress levels. TheasNDVI NDWI curves corresponding study site, which showed growth characteristics of rice, well as the difference information about rice under different heavy metal levels. Theand NDVI and NDWI generally follow a rising trendunder in thedifferent early growth stage, and decline in the late growth stage. information about rice growth heavy metal stress levels. The NDVI and NDWI curves curves generally follow a rising trend in the early growth stage, and decline in the late growthThe stage. distinct shape of the curves under different metal stress are mainly reflected in thestage. following follow rising trend in the early heavy growth decline in the late growth The The generally distinct shape ofathe curves under different heavystage, metaland stress are mainly reflected in the following aspects. First, the maximum EVI different and NDWI values instress severe are smaller than those in distinct shape of the curves under heavy arecurves mainly reflected the following aspects. First, the maximum EVI and NDWI values inmetal severe curves are smaller thanin those in moderate moderate stress and non-stress curves. Second, for the NDWI curves, the values of non-stress curves aspects. First, the curves. maximum EVI and NDWI values in severe curves smaller curves than those in stress and non-stress Second, for the NDWI curves, the values of are non-stress are larger are larger compared to those of moderate stress and severe stress in the early growth stage, and the moderate stress and non-stress curves. Second, for the NDWI curves, the values of non-stress curves compared to those of moderate and severe stresscurves in the in early growth stage,stage. and the severe stress severe stress curve drops the stress most among the three thein late Third, for the the are larger compared to those of moderate stress and severe stress thegrowth early growth stage, and curve drops the the most among the three curves in the late growth stage. Third, for the NDVI curves, NDVI values in the severe curves arethe generally smaller those in other two curves during severecurves, stress curve drops most among three curves inthan the late growth stage. Third, for the thethe values in severe curves are generally smaller than those in other two curves during the whole whole rice growth period. In addition, thegenerally phenological parameters extracted the non-stress NDVI curves, the values in severe curves are smaller than those in otherfrom two curves during ricecurve, growth period. Inlevel addition, phenological parameters extracted from the non-stress curve, such as base and seasonal integral, were greater than those from moderate severe the whole rice growth period. Inthe addition, the phenological parameters extracted from theand non-stress such as base level andlevel seasonal integral, were greater than those from moderate and severe stress stress curves. it and is still impossible to accurately distinguish the three stress levels based on curve, such asHowever, base seasonal integral, were greater than those from moderate and severe curves. However, it is still impossible to accurately distinguish the three stress levels based on this obvious difference information. Thus, it is necessary to extract more implicit phenological stress curves. However, it is still impossible to accurately distinguish the three stress levels based onthis information from time-series data. obvious difference information. Thus, it is necessary to extractto more implicit information this obvious difference information. Thus, it is necessary extract morephenological implicit phenological frominformation time-seriesfrom data. time-series data.

Figure 4. Schematic diagram of changes in (a) greenness and (b) moisture under different heavy metal Figure 4. Schematic diagram changesinin(a) (a)greenness greenness and different heavy metal stress Figure 4. levels. Schematic diagram ofofchanges and(b) (b)moisture moistureunder under different heavy metal stress levels. stress levels.

4.2. Identification of Optimal Number of Feature Subset

Identification of Optimal NumberofofFeature FeatureSubset Subset 4.2. 4.2. Identification of Optimal Number

A total of 9528 features were automatically extracted from the VI time-series data, and 3677 A total 9528 features were automatically extracted from thefrom VI package. time-series data, and 3677 features were by removing the null values from the Tsfresh SelectKBest A total ofofobtained 9528 features were automatically extracted the VIAfter time-series data, features were obtained by removing null greater valuesthe from the were Tsfresh package. After SelectKBest the features importance scores thannull 120 chosen create a preliminary andscreening, 3677 features werewith obtained bythe removing values fromto the Tsfresh package. screening, the features with importance greater than built 120 were chosenthan to create preliminary feature subset, including 1029 manually and signatures. We applied RF-RFECV After SelectKBest screening, the featuresscores withautomatically importance scores greater 120 awere chosen to feature subset, including 1029 manually and automatically built signatures. We applied RF-RFECV

create a preliminary feature subset, including 1029 manually and automatically built signatures.

Sensors 2018, 18, 4425 Sensors 2018, 2018, 18, 18, xx FOR FOR PEER PEER REVIEW REVIEW Sensors

10 of 19 10 of of 19 19 10

We applied RF-RFECV and GB-RFECV algorithms to respectively analyze the contribution of the and GB-RFECV GB-RFECV algorithms to respectively respectively analyze the 5contribution contribution of the the features to classification classification features to classification accuracy, as shownanalyze in Figures and 6. Figures 5afeatures and 6a to show the overall and algorithms to the of accuracy, as shown in Figures 5 and 6. Figures 5a and 6a show the overall classification accuracies for to classification accuracies RF-RFECV GB-RFECV in which theoverall features are incrementally added accuracy, as shown in for Figures 5 and 6.and Figures 5a and 6a show the classification accuracies for RF-RFECV and GB-RFECV inthe which theoffeatures features are incrementally added to the the Along classification and GB-RFECV which the incrementally to classification theRF-RFECV classification analysis. Whenin number featuresare equals 1, accuracyadded is the lowest. with the analysis. When the thethe number of features features equalssignificantly 1, accuracy accuracy is is the lowest. lowest. Along with the the increase increase of analysis. When number of equals 1, the Along with of increase of features, classification accuracy increased until the characteristic number features, the classification accuracy significantly increased until the characteristic number is greater features, the classification accuracy significantly increased until the characteristic number is greater is greater than 200. We can see from Figure 5a that the accuracy value is around 0.96 as the number than 200. 200.features We can can see see from Figure Figure 5a that the accuracy accuracy value is iscorrect aroundidentification 0.96 as as the the number number of selected selected than We from that the value around 0.96 of of selected is between 2005a and 1029. The maximum rate reaches 0.97, features is between 200 and 1029. The maximum correct identification rate reaches 0.97, features is between 200 and 1029. The maximum correct identification rate reaches 0.97, corresponding to the characteristic number of 260. Nevertheless, the classification accuracy obtained by corresponding to to the the characteristic characteristic number number of 260. 260. Nevertheless, Nevertheless, the the classification accuracy accuracy obtained obtained corresponding using all the 1029 features was 0.96. Therefore,ofinstead of using all the classification features in classification, a higher by using using all all the the 1029 1029 features features was was 0.96. 0.96. Therefore, Therefore, instead instead of of using using all all the the features features in in classification, classification, aa by accuracy can be achieved with using only 260 features. A similar trend is also presented in Figure 6a, higher accuracy accuracy can can be be achieved achieved with with using using only only 260 260 features. features. A A similar similar trend trend is is also also presented presented in in higher and we selected 206 optimal features with the GB-RFECV algorithm. Figures 5b and 6b provide Figure 6a, and we selected 206 optimal features with the GB-RFECV algorithm. Figures 5b and 6b Figure 6a, and we selected 206 optimal features with the GB-RFECV algorithm. Figures 5b and 6b detailed information about the about impact ofimpact selected featuresfeatures on the classification result.result. We can see the provide detailed detailed information information about the impact of selected selected features on the the classification classification result. We can can provide the of on We optimal feature subset selected by RF-RFECV and GB-RFECV; the features with importance scores see the the optimal optimal feature feature subset subset selected selected by by RF-RFECV RF-RFECV and and GB-RFECV; GB-RFECV; the the features features with with importance importance see greater 150 than account for 95%for and 87%, Some Some features, suchsuch as NDVI growth ratio scoresthan greater 150 account account 95% andrespectively. 87%, respectively. respectively. features, as NDVI NDVI growth scores greater than 150 for 95% and 87%, Some features, such as growth andratio moisture movement during tillering to heading, scored 139.73 and 145.83, respectively. However, and moisture moisture movement movement during during tillering tillering to to heading, heading, scored scored 139.73 139.73 and and 145.83, 145.83, respectively. respectively. ratio and these features were filtered out by the RFECV algorithm in order to reduce the redundancy between However, these features were filtered out by the RFECV algorithm in order to reduce the redundancy However, these features were filtered out by the RFECV algorithm in order to reduce the redundancy relevant features, so will not be discussed further. Additionally, the indicator corresponding to the between relevant features, so will not be discussed further. Additionally, the indicator corresponding between relevant features, so will not be discussed further. Additionally, the indicator corresponding maximum value, 696.41, which was automatically extracted by Tsfresh, is greater than the contribution to the the maximum maximum value, value, 696.41, 696.41, which which was was automatically automatically extracted extracted by by Tsfresh, Tsfresh, is is greater greater than than the the to of manually established indicators. contribution of manually manually established indicators. indicators. contribution of established

Figure 5. (a) (a) Overall classificationaccuracy accuracyusing using the the RF RF classifier classifier with respect totodifferent different feature setset Figure 5. (a) Overall classification classifierwith withrespect respectto different feature Figure 5. Overall classification accuracy using the feature set obtained by the RFECV algorithm. (b) The importance scores of optimal number of features set. obtained RFECV algorithm.(b) (b)The Theimportance importance scores of optimal obtained byby thethe RFECV algorithm. optimalnumber numberofoffeatures featuresset. set.

Figure 6. (a) (a) Overall classification accuracy using the GBclassifier classifierfor fordifferent differentfeature featureset setobtained obtainedby Figure 6. (a) Overall classification accuracy using the GB Figure 6. Overall classification accuracy using the GB classifier for different feature set obtained by the RFECV RFECV algorithm. (b) importance The importance importance scores of optimal optimal number of features features set. theby RFECV algorithm. (b) The scores of optimal number of of features set. the algorithm. (b) The scores of number set.

Sensors 2018, 18, x FOR PEER REVIEW Sensors 2018, 18, 4425

11 of 19 11 of 19

4.3. Classification Results of Stress Levels 4.3. Classification Results of Stress Levels Because n_estimators, min_samples_split and min_samples_leaf are critical input parameters in the ensemble model, the optimal parameter values must be determined. The ranges of parameters Because n_estimators, min_samples_split and min_samples_leaf are critical input parameters in were set as model, follows:the (n_estimators: 10–200), (min_samples_split: 3–9), The (min_samples_leaf: 2–6). the ensemble optimal parameter values must be determined. ranges of parameters Figure presents the(n_estimators: effect of different parameter values on the accuracy of the model; that is, were set7 as follows: 10–200), (min_samples_split: 3–9), (min_samples_leaf: 2–6). different parameter values leads to parameter different results, model notdifferent rise as Figure 7 presents the effect of different values onand the the accuracy of accuracy the model;does that is, parametervalues valuesleads increases. We traversed combinations, computed the parameter to different results, all andimportant the modelparameter accuracy does not rise asand parameter values accuracy through cross-validation. the combinations, important parameters in RFthe were determined as increases. We traversed all importantFinally, parameter and computed accuracy through follows: (n_estimators: 120), (min_samples_split: and the important cross-validation. Finally, the important parameters in9), RF(min_samples_leaf: were determined as 2); follows: (n_estimators: parameters in GB were determined as follows: (n_estimators: 110), (max_depth: 3). in Of GB the were 1286 120), (min_samples_split: 9), (min_samples_leaf: 2); and the important parameters randomly selected training samples, there 372 pixels3). in Of non-stress 680 pixels in moderate determined as follows: (n_estimators: 110), are (max_depth: the 1286 rice, randomly selected training stress rice, and are 234 372 pixels in severe stress rice, which were used for building therice, ensemble model. samples, there pixels in non-stress rice, 680 pixels in moderate stress and 234 pixelsThe in 3-fold cross validation was applied to evaluate the quality of the ensemble model, with an accuracy severe stress rice, which were used for building ensemble model. The 3-fold cross validation was of 0.988,to indicating we obtained good-quality ensemble model that can be used for classification applied evaluate that the quality of theaensemble model, with an accuracy of 0.988, indicating that we of stress alevels. At 30-mensemble spatial resolution, maps of the model with optimum obtained good-quality model that can be usedensemble for classification of stress levels.parameter At 30-m combination applied regional model scale were generatedparameter (Figure 8). Figure 8a–c show the spatial resolution, mapsatof the the ensemble with optimum combination applied at the discrimination results for non-stress, and severe stress, respectively. Among them, regional scale were generated (Figure moderate 8). Figure stress, 8a–c show the discrimination results for non-stress, the greenstress, part isand thesevere area that belongs to positive judgment; thegreen red part belongs to to moderate stress, respectively. Among them, the partisisthe the area area that belongs misjudgment. Comparing the classification results of different stress levels in Figure the 8, itclassification is found that positive judgment; the red part is the area that belongs to misjudgment. Comparing there are differentstress degrees forthat thethere threeare stress levels, however, the ensemble results of different levelsofinmisclassification Figure 8, it is found different degrees of misclassification model can achieve good results on the ensemble whole. model can achieve good results on the whole. for the three stress levels, however,

Figure Figure 7. 7. The The assessment assessment results results under under different differentparameter parametervalues. values.

Sensors 2018, 18, 4425 Sensors 2018, 18, x FOR PEER REVIEW

12 of 19 12 of 19

Figure and (c) (c) severe severe stress stress areas. areas. Figure8.8.Classification Classification results results for for (a) (a) non-stress, non-stress, (b) moderate stress, and

4.4. 4.4.Validation ValidationofofDiscrimination DiscriminationResults Results According thethe discrimination model established above,above, an accuracy assessment was conducted Accordingto to discrimination model established an accuracy assessment was base on test datasets, which included a total of 552 randomly selected rice pixels. The confusion conducted base on test datasets, which included a total of 552 randomly selected rice pixels. The matrix that compared predicted results and actual was generated togenerated evaluate classification confusion matrix thatthe compared the predicted resultsresults and actual results was to evaluate accuracy. Among the 154Among non-stress 150 were identified non-stress riceas(97% agreement). classification accuracy. the rice 154 pixels, non-stress rice pixels, 150 as were identified non-stress rice For 293 moderate-stress rice pixels, 290 were identified as moderate-stress rice (99% agreement). (97% agreement). For 293 moderate-stress rice pixels, 290 were identified as moderate-stress rice (99% From Figure 9a, theFigure severe9a, stress regionstress had 101 pixels for positive judgment 4 pixelsand misjudged, agreement). From the severe region had 101 pixels for positiveand judgment 4 pixels with a positive rate of 96%.rate An of overall accuracy 98.01% of was obtained when compared with the misjudged, with a positive 96%. An overallofaccuracy 98.01% was obtained when compared actual dataset. we can see from 9 that the 9discrimination effect of the ensemble with the actualFurthermore, dataset. Furthermore, we can Figure see from Figure that the discrimination effect of the model is better than that ofthan using RFofand GBRF separately. The discrimination accuracy for non-stress ensemble model is better that using and GB separately. The discrimination accuracy for and severe-stress level rice increased 1–3% andby2–3% and the overall for non-stress and severe-stress level ricebyincreased 1–3%respectively, and 2–3% respectively, andaccuracy the overall different categories showed an improvement of approximately 1–2%. Figure1–2%. 10 presents accuracystress for different stress categories showed an improvement of approximately Figure the 10 ROC and area under curve to evaluate thetoclassification The y-axis represents the presents the ROC and area(AUC) undervalues curve (AUC) values evaluate the results. classification results. The y-axis represents the rate at which a pixel is a positive class and is predicted to be a positive class. The x-

Sensors 2018, 18, 4425 Sensors 2018, 18, x FOR PEER REVIEW

13 of 19 13 of 19

rate represents at which a the pixel is aatpositive predicted to be a positive class.toThe represents the axis rate which aclass pixeland is aisnegative class and is predicted be ax-axis positive class. The rate at which a pixel is a negative class and is predicted to be a positive class. The closer the AUC value closer the AUC value is to 1, the better the differentiation effect. The AUC value here reached 0.98 is to 0.99, 1, thewhich better the differentiation effect. The AUC valuefor here reached 0.98 and 0.99, which means we and means we obtained a high accuracy classifying different heavy metal stress obtained a high accuracy for classifying different heavy metal stress categories. categories.

Figure 9. 9. Confusion Confusionmatrix matrixbetween betweenestimated estimatedand andactual actualstress stress categories categories based based on on (a) (a) the the ensemble ensemble Figure model, (b) (b) the the RF RF classifier classifier and and (c) (c) the the GB GB classifier. classifier. model,

Sensors 2018, 18, x FOR PEER REVIEW Sensors 2018, 18, 4425

14 of 19 14 of 19

Sensors 2018, 18, x FOR PEER REVIEW

14 of 19

Figure 10.10. The evaluation results based onROC ROCcurves curves and AUC values. Figure 10. The evaluation on ROC curves and AUC values. Figure The evaluationresults results based based on and AUC values.

5. 5. Discussion 5. Discussion We obtained a satisfactory performance classifying heavy metal stress levels in which We aasatisfactory performance in in classifying heavy metal stress levels in rice, rice, which can Weobtained obtained satisfactory performance classifying heavy metal stress levels in rice, which can be partly attributed to the effectiveness of spatio-temporal fusion images for detecting rice stress be partly attributed to the effectiveness of spatio-temporal fusion images for detecting rice stress can be partly attributed to the effectiveness of spatio-temporal fusion images for detecting rice stress andselection the selection of high-quality ricepixels. pixels. In In addition addition to rice pixels, the the accuracy in in and of rice to high-quality rice pixels, and the the selection of high-quality high-quality rice pixels. addition tohigh-quality high-quality rice pixels, the accuracy accuracy in reconstructing VI time series is dependent on the de-noising effect. The WS model performed well in well reconstructing series is is dependent dependenton onthe thede-noising de-noisingeffect. effect.The TheWS WSmodel model performed reconstructing VI time series performed well in de-noising, as well as being in accordance with the growth characteristics of rice, with RMSE and in de-noising, well beingininaccordance accordancewith withthe thegrowth growth characteristics characteristics of of rice, with RMSE and de-noising, asas well asasbeing and correlation coefficient of 0.0414 and 0.9516, respectively. The extracted four key phenological periods, correlation coefficient of 0.0414 and 0.9516, respectively. The extracted four key correlation coefficient of 0.0414 and 0.9516, respectively. The extracted phenological periods, namely transplanting, tillering, heading and maturation, which conformed to rice phenology and namely transplanting, tillering,can heading and maturation, which conformed rice namely transplanting, maturation, conformed rice phenology phenology and transplanting characteristics, be used to establish phenological indicators totodetect heavy meal and transplanting characteristics, can be used establish heavy transplanting usedbyto tothe establish phenological indicators totodetect detect heavy meal meal stress in rice. Other features extracted Tsfresh phenological package were indicators also appliedto this research. stress in features extracted the Tsfresh package werefeatures also applied stressTherefore, in rice. rice. Other Other features to this research. the capability and necessity by of extracting multi-dimensional were discussed. We plottedthe thecapability decision boundaries of three stress levels based on only twofeatures features (NDVI and EVI We Therefore, capability andnecessity necessity extracting multi-dimensional features were discussed. Therefore, and ofof extracting multi-dimensional were discussed. amplitude); these two features with high importance scores were used here. From Figure 11, we can EVI We plotted decision boundaries three stresslevels levelsbased basedon ononly onlytwo two features features (NDVI and plotted thethe decision boundaries of of three stress EVI see that the two features were able to discriminate different heavy metal stress to a certain extent. amplitude); importance scores were used here. From Figure 11, we see amplitude);these thesetwo twofeatures featureswith withhigh high importance scores were used here. From Figure 11, can we can However, many moderate stress pixels are mixed with unstressed pixels. Figure 5 showed that that the two were able discriminate differentdifferent heavy metal stress to astress certain However, see that the features two features weretoable to discriminate heavy metal to extent. a certain extent. classification accuracy was around 0.96 based on all 1029 features for RF or GB classifiers. Compared many moderate pixelsstress are mixed Figurepixels. 5 showed that5 classification However, manystress moderate pixelswith are unstressed mixed withpixels. unstressed Figure showed that with using all features in the classification, high accuracy cannot be maintained using only two accuracy was around 0.96 on all features RFfeatures or GB classifiers. Compared with using all classification accuracy wasbased around 0.961029 based on and allfor 1029 for RF or features GB classifiers. Compared features (Table 3). Hence, we acquired manually automatically designed to fully reflect features in the high accuracy cannot be maintained using only two (Tabletwo 3). with phenological using all classification, features in the classification, highachieved accuracy cannot be maintained usingmetal only information in VI time-series, and accurate discrimination forfeatures heavy Hence, we acquired manually and automatically designed features to fully reflect phenological features (Table 3). Hence, we acquired manually and automatically designed features to fully reflect stress levels using high-dimensional phenological features. information VI time-series, achievedand accurate discrimination for heavy metal phenologicalininformation in VIand time-series, achieved accurate discrimination for stress heavy levels metal using phenologicalphenological features. stress high-dimensional levels using high-dimensional features.

Figure 11. The decision boundaries of GB and RF classifiers.

Figure Figure 11. 11. The decision decision boundaries boundaries of of GB GB and and RF RF classifiers. classifiers.

Sensors 2018, 18, x FOR PEER REVIEW

15 of 19

Sensors 2018, 18, 4425

15 of 19

Table 3. Classification accuracy of RF and GB classifiers based on two phenological features.

Assessment Measures Random Forest Gradient Boosting Overall Accuracy 67.57% 66.12% Correct-Judgement Rate for Non-Stress Rice Assessment Measures Random 51.94% Forest Gradient 48.70% Boosting Correct-Judgement Rate for Moderate-Stress Rice 79.52% 78.16% Overall Accuracy 67.57% 66.12% Correct-Judgement Ratefor forNon-Stress Severe-Stress 57.14% 58.10% Correct-Judgement Rate Rice Rice 51.94% 48.70% Table 3. Classification accuracy of RF and GB classifiers based on two phenological features.

Correct-Judgement Rate for Moderate-Stress Rice 79.52% 78.16% Correct-Judgement Rate for Severe-Stress Rice 57.14% 58.10% Manually designed indicators were based on the critical phenological periods, which would

leave out important phenological information at other stages. The time-series curves of different stress levels contain visible and invisible difference information; without Tsfresh, allwhich complex and Manually designed indicators were based on the critical phenological periods, would implicit would haveinformation to be calculated by hand. TheThe extracted features basedofon Tsfresh leave outcharacteristics important phenological at other stages. time-series curves different can belevels used contain to buildvisible modelsand thatinvisible performdifference classification tasks on the time series [59]. the stress information; without Tsfresh, all Therefore, complex and automatically extracted characteristics able tobycompensate for the limitations of theon manually implicit characteristics would have to bewere calculated hand. The extracted features based Tsfresh designed indicators some extent. Further classification quantitative analysis by[59]. comparing the can be used to buildtomodels that perform tasks onwas theconducted time series Therefore, classification results obtained using bothwere the able manually and automatically designed with the automatically extracted characteristics to compensate for the limitations of features the manually those obtained using either the manually or automatically features only, and the the designed indicators to some extent. Further quantitative analysisdesigned was conducted by comparing classification accuracy reached 0.972, which was greater than the individual accuracies, which were classification results obtained using both the manually and automatically designed features with those 0.918 andusing 0.963,either respectively. In summary, the best stress level estimation results were using obtained the manually or automatically designed features only, and theobtained classification the manually and0.972, automatically designed features together. Due to the negative impact of some nonaccuracy reached which was greater than the individual accuracies, which were 0.918 and 0.963, essential features on classification results, we carriedresults out primary selection bythe the SelectKBest respectively. In summary, the best stress level estimation were obtained using manually and program and designed further screening the REFCV algorithm. As seen in Section the optimalfeatures feature automatically features by together. Due to the negative impact of some4.2, non-essential subset producedresults, the highest classification accuracy. The feature method employed in this on classification we carried out primary selection by theselection SelectKBest program and further study not by only faster machine learning models, but optimal also achieved the best discrimination screening thedeveloped REFCV algorithm. As seen in Section 4.2, the feature subset produced the results.classification accuracy. The feature selection method employed in this study not only developed highest order to allow the classification model tothe have generalization fasterIn machine learning models, but also achieved beststronger discrimination results. and achieve better results, we combined RF classifiers and to performed the classification process voting. That In order to allowGB theand classification model have stronger generalization and by achieve better is, the probabilities each pixel belonging to different categories werebypredicted by GB results, we combinedofGB andrice RF classifiers and performed the stress classification process voting. That is, andprobabilities RF, and thenofthe predicted results of thetotwo classifiers were averaged. We took theby above two the each rice pixel belonging different stress categories were predicted GB and features (thethe amplitudes NDVIofand EVI)classifiers as examples plot the classification probability for the RF, and then predictedof results the two weretoaveraged. We took the above two features ensemble model. 12 presents the averaged probabilities that rice pixels belong different (the amplitudes of Figure NDVI and EVI) as examples to plot the classification probability for thetoensemble stress categories. Yellow cells representprobabilities the area with the highest All points falling in the model. Figure 12 presents the averaged that rice pixelsprobability. belong to different stress categories. yellow cells area represent are considered to have largest probabilities, with the stress corresponding Yellow the area with the highest probability. All points fallingcategory in the yellow area are to the maximum taken as the with finalthe category. Section corresponding 4.4 described to thethe classification considered to have probability the largest probabilities, stress category maximum capacity of taken these as three in detail. Furthermore, Figure 9 demonstrated in an of intuitive way probability the classifiers final category. Section 4.4 described the classification capacity these three that the ensemble had the Figure best performance in determining boundaries theensemble stress classes. classifiers in detail.model Furthermore, 9 demonstrated in an intuitive way thatofthe modelIt has the been verified that the in classification of ensemble was greater of thethat RF and had best performance determiningability boundaries of the model stress classes. It hasthan beenthat verified the GB classifiers.ability of ensemble model was greater than that of the RF and GB classifiers. classification

Figure Figure 12. 12. The The probabilities probabilities that that rice rice pixels pixels belong belong to to different differentstress stressclasses classesfor forthe theensemble ensemblemodel. model.

Sensors 2018, 18, 4425

16 of 19

6. Conclusions This study applied the ESTARFM to fuse Landsat data with MODIS data, and then generated multi-temporal synthetic images to detect Cd stress on rice growing at regional scale. A scheme was built by considering the estimation of heavy metal stress levels as a classification problem. Firstly, 3677 features were obtained based on manual establishment and automatic extraction. Secondly, optimal feature selection was employed, in which 260 signatures of the RF classifier and 206 signatures of the GB classifier were selected using the proposed feature selection method. Finally, an ensemble model using RF and GB classifiers was utilized due to its capability for improving accuracy. The result of the scheme described above showed that: (1) The synthetic images offer promise for the accurate detection of rice under Cd stress; (2) the feature selection model is effective, with the stress level estimation accuracy achieved using the optimal features shown to be higher than that using all features; (3) the distinction accuracy based on a combination of the features extracted manually and automatically was higher than that based on either manual establishment or automatic extraction; and, (4) the discrimination accuracy obtained by the ensemble model was higher than that considering the RF and GB classifiers separately. In future studies, the response mechanism of rice phenology to the concentration of heavy metals will be further explored. Additionally, it is necessary to consider the crop varieties, various cultivation practices, climatic conditions, and growth environments in different areas, and take these characteristics as important input parameters of the model. The classification model of heavy metal stress should be optimized by conducting validation at more sites. Author Contributions: T.L. designed and performed the experiments and wrote the paper; X.L. supervised the research and provided significant comments and suggestions; M.L. gave suggestions about the experiment; and L.W. contributed to the review of the manuscript. Funding: This research was funded by the National Natural Science Foundation of China (Nos. 41371407, 41601473 and 41701387). Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2. 3. 4. 5.

6. 7.

8. 9.

10.

Zhang, X.; Zhong, T.; Liu, L.; Ouyang, X. Impact of Soil Heavy Metal Pollution on Food Safety in China. PLoS ONE 2015, 10, e0135182. [CrossRef] [PubMed] Jianming, X.U.; Meng, J.; Liu, X.; Shi, J.; Tang, X. Control of Heavy Metal Pollution in Farmland of China in Terms of Food Security. Bull. Chin. Acad. Sci. 2018, 33, 153–159. Zhao, J.X.; Yin, P.C.; Yue, R. Research Progress of Status, Source, Restoration Technique of Heavy Metals Pollution in Cropland of China. J. Anhui Agric. Sci. 2018, 46, 19–21. Järup, L. Hazards of heavy metal contamination. Br. Med. Bull. 2003, 68, 167–182. [CrossRef] [PubMed] Liu, X.; Zhong, L.; Meng, J.; Fan, W.; Zhang, J.; Zhi, Y.; Zeng, L.; Tang, X.; Xu, J. A multi-medium chain modeling approach to estimate the cumulative effects of cadmium pollution on human health. Environ. Pollut. 2018, 239, 308–317. [CrossRef] [PubMed] Chen, W.; Yang, Y.; Xie, T.; Wang, M.; Peng, C.; Wang, R. Challenges and Countermeasures for Heavy Metal Pollution Control in Farmlands of China. Acta Pedol. Sin. 2018, 55, 261–272. Wanlu, L.; Binbin, X.; Qiujin, S.; Xingmei, L.; Jianming, X.; Brookes, P.C. The identification of ‘hotspots’ of heavy metal pollution in soil-rice systems at a regional scale in eastern China. Sci. Total Environ. 2014, 472, 407–420. Deng-Wei, W.; Yun-Zhao, W.; Hong-Rui, M. Review on Remote Sensing Monitoring on Contaminated Plant. Remote Sens. Technol. Appl. 2009, 24, 238–245. Jin, M.; Liu, X.; Zhang, B. Evaluating Heavy-Metal Stress Levels in Rice Using a Theoretical Model of Canopy-Air Temperature and Leaf Area Index Based on Remote Sensing. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2017, 10, 1–11. [CrossRef] Liu, M.; Wang, T.; Skidmore, A.K.; Liu, X. Heavy metal-induced stress in rice crops detected using multi-temporal Sentinel-2 satellite images. Sci. Total Environ. 2018, 637–638, 18–29. [CrossRef]

Sensors 2018, 18, 4425

11. 12. 13. 14. 15. 16. 17. 18.

19. 20. 21. 22. 23. 24.

25. 26. 27. 28. 29.

30. 31.

32.

33.

17 of 19

Ren, H.Y.; Zhuang, D.F.; Pan, J.J.; Shi, X.Z.; Wang, H.J. Hyper-spectral remote sensing to monitor vegetation stress. J. Soils Sediments 2008, 8, 323–326. [CrossRef] Barcelo, J.; Poschenrieder, C. Plant water relations as affected by heavy metal stress: A review. J. Plant Nutr. 1990, 13, 1–37. [CrossRef] Cheng, S. Effects of Heavy metals on plants and resistance mechanisms. Environ. Sci. Pollut. Res. 2003, 10, 256–264. [CrossRef] Dias, M.C.; Monteiro, C.; Moutinho-Pereira, J.; Correia, C.; Gonçalves, B.; Santos, C. Cadmium toxicity affects photosynthesis and plant growth at different levels. Acta Physiol. Plant. 2013, 35, 1281–1289. [CrossRef] Nagajyoti, P.C.; Lee, K.D.; Sreekanth, T.V.M. Heavy metals, occurrence and toxicity for plants: A review. Environ. Chem. Lett. 2010, 8, 199–216. [CrossRef] Liu, S.; Liu, X.; Liu, M.; Wu, L.; Ding, C.; Huang, Z. Extraction of Rice Phenological Differences under Heavy Metal Stress Using EVI Time-Series from HJ-1A/B Data. Sensors 2017, 17, 1243. Liu, T.; Liu, X.; Liu, M.; Wu, L. Evaluating Heavy Metal Stress Levels in Rice Based on Remote Sensing Phenology. Sensors 2018, 18, 860. [CrossRef] [PubMed] Tian, L.; Liu, X.; Zhang, B.; Liu, M.; Wu, L. Extraction of Rice Heavy Metal Stress Signal Features Based on Long Time Series Leaf Area Index Data Using Ensemble Empirical Mode Decomposition. Int. J. Environ. Res. Public Health 2017, 14, 1018. [CrossRef] Cao, R.; Chen, J.; Shen, M.; Tang, Y. An improved logistic method for detecting spring vegetation phenology in grasslands from MODIS EVI time-series data. Agric. Forest Meteorol. 2015, 200, 9–20. [CrossRef] Melaas, E.K.; Friedl, M.A.; Zhu, Z. Detecting interannual variation in deciduous broadleaf forest phenology using Landsat TM/ETM + data. Remote Sens. Environ. 2013, 132, 176–185. [CrossRef] Sakamoto, T.; Yokozawa, M.; Toritani, H.; Shibayama, M.; Ishitsuka, N.; Ohno, H. A crop phenology detection method using time-series MODIS data. Remote Sens. Environ. 2005, 96, 366–374. [CrossRef] Walker, J.J.; Beurs, K.M.D.; Wynne, R.H.; Gao, F. Evaluation of Landsat and MODIS data fusion products for analysis of dryland forest phenology. Remote Sens. Environ. 2012, 117, 381–393. [CrossRef] Gao, F.; Masek, J.; Schwaller, M.; Hall, F. On the blending of the Landsat and MODIS surface reflectance: Predicting daily Landsat surface reflectance. IEEE Trans. Geosci. Remote Sens. 2006, 44, 2207–2218. Hilker, T.; Wulder, M.A.; Coops, N.C.; Linke, J.; Mcdermid, G.; Masek, J.G.; Gao, F.; White, J.C. A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODIS. Remote Sens. Environ. 2009, 113, 1613–1627. [CrossRef] Weng, Q.; Fu, P.; Gao, F. Generating daily land surface temperature at Landsat resolution by fusing Landsat and MODIS data. Remote Sens. Environ. 2014, 145, 55–67. [CrossRef] Zhu, X.L.; Jin, C.; Feng, G.; Chen, X.H.; Masek, J.G. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions. Remote Sens. Environ. 2010, 114, 2610–2623. [CrossRef] Reed, B.C.; Brown, J.F.; Vanderzee, D.; Loveland, T.R.; Merchant, J.W.; Ohlen, D.O. Measuring phenological variability from satellite imagery. J. Veg. Sci. 1994, 5, 703–714. [CrossRef] Reed, B.C.; White, M.; Brown, J.F. Remote Sensing Phenology; Springer: Dordrecht, The Netherlands, 2003; pp. 365–381. Cong, N.; Piao, S.; Chen, A.; Wang, X.; Lin, X.; Chen, S.; Han, S.; Zhou, G.; Zhang, X. Spring vegetation green-up date in China inferred from SPOT NDVI data: A multiple model analysis. Agric. Forest Meteorol. 2012, 165, 104–113. [CrossRef] Mo, Y.; Momen, B.; Kearney, M.S. Quantifying moderate resolution remote sensing phenology of Louisiana coastal marshes. Ecol. Model. 2015, 312, 191–199. [CrossRef] Pan, Z.; Huang, J.; Zhou, Q.; Wang, L.; Cheng, Y.; Zhang, H.; Blackburn, G.A.; Yan, J.; Liu, J. Mapping crop phenology using NDVI time-series derived from HJ-1 A/B data. Int. J. Appl. Earth Obs. Geoinf. 2015, 34, 188–197. [CrossRef] Atkinson, P.M.; Jeganathan, C.; Dash, J.; Atzberger, C. Inter-comparison of four models for smoothing satellite sensor time-series data to estimate vegetation phenology. Remote Sens. Environ. 2012, 123, 400–417. [CrossRef] Hird, J.N.; Mcdermid, G.J. Noise reduction of NDVI time series: An empirical comparison of selected techniques. Remote Sens. Environ. 2009, 113, 248–258. [CrossRef]

Sensors 2018, 18, 4425

34.

35. 36. 37.

38.

39.

40.

41.

42.

43.

44.

45. 46.

47.

48.

49. 50. 51.

52. 53.

18 of 19

Vilela, M.; Borges, C.C.; Vinga, S.; Vasconcelos, A.T.R.; Santos, H.; Voit, E.O.; Almeida, J.S. Automated smoother for the numerical decoupling of dynamics models. BMC Bioinform. 2007, 8, 305. [CrossRef] [PubMed] Ling, Y.; Geng, L.N.; Jing, Y.S.; Rong-Chen, H.U.; Sun, X.; Bureau, J.M. Study on Phenology Extraction of Paddy Rice Based on Different Filtering Methods. Sci. Technol. Eng. 2014, 14, 1671–1815. Sun, H.S.; Huang, J.F.; Peng, D.L. Detecting major growth stages of paddy rice using MODIS data. J. Remote Sens. 2009, 6, 1122–1137. Dong, J.; Xiao, X.; Menarguez, M.A.; Zhang, G.; Qin, Y.; Thau, D.; Biradar, C.; Iii, B.M. Mapping paddy rice planting area in northeastern Asia with Landsat 8 images, phenology-based algorithm and Google Earth Engine. Remote Sens. Environ. 2016, 185, 142–154. [CrossRef] Zhang, G.; Xiao, X.; Dong, J.; Kou, W.; Jin, C.; Qin, Y.; Zhou, Y.; Wang, J.; Menarguez, M.A.; Biradar, C. Mapping paddy rice planting areas through time series analysis of MODIS land surface temperature and vegetation index data. Isprs J. Photogramm. Remote Sens. 2015, 106, 157–171. [CrossRef] Zhou, Y.; Xiao, X.; Qin, Y.; Dong, J.; Zhang, G.; Kou, W.; Jin, C.; Wang, J.; Li, X. Mapping paddy rice planting area in rice-wetland coexistent areas through analysis of Landsat 8 OLI and MODIS images. Int. J. Appl. Earth Obs. Geoinf. 2016, 46, 1–12. [CrossRef] Küçük, Ç.; Ta¸skın, G.; Erten, E. Paddy-Rice Phenology Classification Based on Machine-Learning Methods Using Multitemporal Co-Polar X-Band SAR Images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2017, 9, 2509–2519. [CrossRef] Chen, J.; Chen, J.; Liao, A.; Cao, X.; Chen, L.; Chen, X.; He, C.; Han, G.; Peng, S.; Lu, M. Global land cover mapping at 30 m resolution: A POK-based operational approach. ISPRS J. Photogramm. Remote Sens. 2015, 103, 7–27. [CrossRef] Du, P.; Samat, A.; Waske, B.; Liu, S.; Li, Z. Random Forest and Rotation Forest for fully polarized SAR image classification using polarimetric and spatial features. ISPRS J. Photogramm. Remote Sens. 2015, 105, 38–53. [CrossRef] Luo, J.; Li, X.; Ma, R.; Li, F.; Duan, H.; Hu, W.; Qin, B.; Huang, W. Applying remote sensing techniques to monitoring seasonal and interannual changes of aquatic vegetation in Taihu Lake, China. Ecol. Indic. 2016, 60, 503–513. [CrossRef] Miao, X.; Heaton, J.S.; Zheng, S.; Charlet, D.A.; Liu, H. Applying tree-based ensemble algorithms to the classification of ecological zones using multi-temporal multi-source remote-sensing data. Int. J. Remote Sens. 2012, 33, 1823–1849. [CrossRef] Schneider, A. Monitoring land cover change in urban and peri-urban areas using dense time stacks of Landsat satellite data and a data mining approach. Remote Sens. Environ. 2012, 124, 689–704. [CrossRef] Chen, D.; Guo, H.; Li, R.; Li, L.; Pan, G.; Chang, A.; Joseph, S. Low uptake affinity cultivars with biochar to tackle Cd-tainted rice—A field study over four rice seasons in Hunan, China. Sci. Total Environ. 2016, 541, 1489–1498. [CrossRef] [PubMed] Lei, M.; Tie, B.Q.; Song, Z.G.; Liao, B.H.; Lepo, J.E.; Huang, Y.Z. Heavy metal pollution and potential health risk assessment of white rice around mine areas in Hunan Province, China. Food Secur. 2015, 7, 45–54. [CrossRef] Linghong, K.E.; Wang, Z.; Song, C.; Zhenquan, L.U. Reconstruction of MODIS LST Time Series and Comparison with Land Surface Temperature (T) among Observation Stations in the Northeast Qinghai-Tibet Plateau. Prog. Geogr. 2011, 30, 819–826. Wang, Q.; Huang, C.; Liu, G.; Liu, Q.; He, L.I.; Chen, Z. Cloud Shadow Identification Based on QA Band of Landsat 8. J. Geo-Inf. Sci. 2018, 20, 89–98. Gao, B.C. NDWI—A normalized difference water index for remote sensing of vegetation liquid water from space. Remote Sens. Environ. 1996, 58, 257–266. [CrossRef] Huete, A.; Didan, K.; Miura, T.; Rodriguez, E.P.; Gao, X.; Ferreira, L.G. Overview of the radiometric and biophysical performance of the MODIS vegetation indices. Remote Sens. Environ. 2002, 83, 195–213. [CrossRef] Xiao, X.; Boles, S.; Liu, J.; Zhuang, D.; Frolking, S.; Li, C.; Salas, W.; Iii, B.M. Mapping paddy rice agriculture in southern China using multi-temporal MODIS images. Remote Sens. Environ. 2005, 95, 480–492. [CrossRef] Eilers, P.H.C. A Perfect Smoother. Anal. Chem. 2003, 75, 3631–3636. [CrossRef] [PubMed]

Sensors 2018, 18, 4425

54.

55. 56. 57. 58. 59. 60. 61. 62. 63. 64.

65. 66. 67. 68.

19 of 19

Atzberger, C.; Rembold, F. Estimation of inter-annual winter crop area variation and spatial distribution with low resolution NDVI data by using neural networks trained on high resolution images. In Proceedings of the Remote Sensing for Agriculture, Ecosystems, and Hydrology XI, Berlin, Germany, 1–3 September 2009; pp. 170–175. Atzberger, C.; Eilers, P.H.C. Evaluating the Effectiveness of Smoothing Algorithms in the Absence of Ground Reference Measurements; Taylor & Francis, Inc.: Abingdon, UK, 2011; pp. 3689–3709. Atzberger, C.; Eilers, P.H.C. A time series for monitoring vegetation activity and phenology at 10-daily time steps covering large parts of South America. Int. J. Digit. Earth 2011, 4, 365–386. [CrossRef] Jönsson, P.; Eklundh, L. TIMESAT—A program for analyzing time-series of satellite sensor data. Comput. Geosci. 2004, 30, 833–845. [CrossRef] Eklundha, L.; Jönssonb, P. TIMESAT 3.1 Software Manual. Available online: http://web.nateko.lu.se/ timesat/docs/timesat3_1_1_SoftwareManual.pdf (accessed on 13 December 2018). Christ, M.; Braun, N.; Neuffer, J.; Kempa-Liehr, A.W. Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh—A Python package). Neurocomputing 2018, 307, 72–77. [CrossRef] Christ, M.; Kempaliehr, A.W.; Feindt, M. Distributed and parallel time series feature extraction for industrial big data applications. arXiv, 2016; arXiv:1610.07717. Cai, J.; Luo, J.; Wang, S.; Yang, S. Feature selection in machine learning: A new perspective. Neurocomputing 2018, 300, 70–79. [CrossRef] Varnek, A. 14. Compound Classification Using the Scikit-Learn Library; John Wiley & Sons, Ltd.: Hoboken, NJ, USA, 2017; pp. 223–239. Hikaru, M.; Seiichiro, A.; Tomomasa, N.; Yoshifumi, O. Accurate and robust gene selection for disease classification using a simple statistic. Bioinformation 2008, 3, 68–71. Wang, X.; Nagashima, T.; Fukuta, K.; Okada, Y.; Sawai, M.; Tanaka, H.; Uozumi, T. Statistical method for classifying cries of baby based on pattern recognition of power spectrum. Int. J. Biom. 2010, 2, 113–123. [CrossRef] Defries, R.S.; Chan, J.C.W. Multiple criteria for evaluating machine learning algorithms for land cover classification from satellite data. Remote Sens. Environ. 2000, 74, 503–515. [CrossRef] Ping, Y.P.; Zang, S.Y. Crop Identification Based on MODIS NDVI Time-series Data and Phenological Characteristics. J. Nat. Resour. 2016, 31, 503–513. Lewis, H.G.; Brown, M. A generalized confusion matrix for assessing area estimates from remotely sensed data. Int. J. Remote Sens. 2001, 22, 3223–3235. [CrossRef] Davis, J.; Goadrich, M. The relationship between Precision-Recall and ROC curves. In Proceedings of the International Conference on Machine Learning (ICML 0 06), New York, NY, USA, 25–29 June 2006; pp. 233–240. © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).