remote sensing - MDPI

remote sensing Article

Pole-Like Road Furniture Detection and Decomposition in Mobile Laser Scanning Data Based on Spatial Relations Fashuai Li *, Sander Oude Elberink and George Vosselman

ID

Faculty of Geo-Information Science and Earth Observation, University of Twente, PO BOX 217, 7514 AE Enschede, The Netherlands; [email protected] (S.O.E.); [email protected] (G.V.) * Correspondence: [email protected]; Tel.: +31-68-491-3147 Received: 16 February 2018; Accepted: 28 March 2018; Published: 30 March 2018







Abstract: Road furniture plays an important role in road safety. To enhance road safety, policies that encourage the road furniture inventory are prevalent in many countries. Such an inventory can be remarkably facilitated by the automatic recognition of road furniture. Current studies typically detect and classify road furniture as one single above-ground component only, which is inadequate for road furniture with multiple functions such as a streetlight with a traffic sign attached. Due to the recent developments in mobile laser scanners, more accurate data is available that allows for the segmentation of road furniture at a detailed level. In this paper, we propose an automatic framework to decompose road furniture into different components based on their spatial relations in a three-step procedure: first, pole-like road furniture are initially detected by removing ground points and an initial classification. Then, the road furniture is decomposed into poles and attachments. The result of the decomposition is taken as a feedback to remove spurious pole-like road furniture as a third step. If there are no poles extracted in the decomposition stage, these incorrectly detected pole-like road furniture—such as the pillars of buildings—will be removed from the detection list. We further propose a method to evaluate the results of the decomposition. Compared with our previous work, the performance of decomposition has been much improved. In our test sites, the correctness of detection is higher than 90% and the completeness is approximately 95%, showing that our procedure is competitive to state of the art methods in the field of pole-like road furniture detection. Compared to our previous work, the optimized decomposition improves the correctness by 7.3% and 18.4% in the respective test areas. In conclusion, we demonstrate that our method decomposes pole-like road furniture into poles and attachments with respect to their spatial relations, which is crucial for road furniture interpretation. Keywords: mobile laser scanning; road furniture; detection; decomposition; poles; attachments; spatial relations

1. Introduction Road safety has been one of the major focuses in public safety concerns for many years. In the 2015 global status report on road safety, the total number of worldwide road traffic deaths remains unacceptably high at 1.25 million per year [1]. To conduct road safety inspections in Europe, a road infrastructure safety management directive is adopted by the European Union [2]. The EU will eventually have a role in the safety management of the roads belonging to European transportation networks—which is set to encompass 90,000 km of motorway and high-quality roads by 2020—through safety audits at the design stage and regular safety inspections of the network [3]. Similar policies also have been made in the U.S. In 2014, the U.S. Department of Transportation developed a system named

Remote Sens. 2018, 10, 531; doi:10.3390/rs10040531

www.mdpi.com/journal/remotesensing

Remote Sens. 2018, 10, 531

2 of 28

Model Inventory of Roadway Elements (MIRE) to inventory road furniture and improve roadway safety [4]. Road safety can be enhanced by the inventory of road furniture which is strongly related to road furniture detection [5]. Another rising general interest is autonomous driving, which facilitates driving safety and makes life more convenient. In addition, it enables people such as the aged and disabled who cannot drive to use a vehicle [6]. Although autonomous driving systems do not fully rely on 3D precise maps, it is still crucial for the improvement of the safety and stability of automatic driving systems. Road furniture detection plays an important role in both road furniture inventory and 3D highly precise mapping, which consists of road detection, curbstone detection, pole-like road furniture interpretation, and so forth. Currently, the road furniture inventory mainly relies on visual inspection or semi-manual interpretation, which is time-consuming and tedious. To facilitate this procedure, methods for automatic road furniture detection relying on high-quality data are needed. Tools for capturing three-dimensional road scene data are Mobile Mapping Systems (MMS) which have been developed rapidly in recent years. They are often mounted on a vehicle and mainly composed of four parts: Light Detection and Ranging (LiDAR) sensors that collect 3D point clouds, cameras that capture 2D imagery data, an accurate Global Positioning System (GPS) that records the position of the previous two sensors, and an Inertial Measure Unit (IMU) that measures the pose of the sensors. Compared with optical images acquirement, Mobile Laser Scanning (MLS) data collection is not restricted by the illumination conditions such as good weather and daytime. Moreover, MLS data can be obtained rapidly and accurately. Significant progress has been made on research related to road furniture inventory in the laser scanning data field, which includes road detection and modelling [7–9], curbstones mapping [10], railway modelling [11,12], pole-like road furniture interpretation [13–21], tree inventory [22], and building detection [23–26]. In the urban scene, MLS data analysis has become popular and essential for urban road environment analysis. Individual road furniture can be interpreted as a single object or as multiple connected parts. For instance, a streetlight with an attached traffic sign can be recognized as a streetlight at the object level. It can also be interpreted as multiple connected parts: a pole connected to a traffic sign and a streetlight. Numerous studies have been carried out on road furniture interpretation on the level of single objects, while only a little attention has been paid to the more detailed segmentation of road furniture based on spatial relations between object parts. Such a detailed segmentation, however, is necessary since it provides more detailed partition information for the interpretation of road furniture. The objective of this paper is to propose an automatic framework for road furniture decomposition. The rest of the paper is organized as follows. Section 2 provides a review of related work on road furniture interpretation and state of art of point cloud semantics segmentation techniques. In Section 3, our method is described in three main stages: the initial road furniture detection, decomposition, and final road furniture detection. We introduce the test sites, analyze the experimental results, and give a comparison to other work in Section 4. The conclusions that are drawn and possible future work can be found in Section 5. 2. Related Work The reliability of MLS systems has improved remarkably in recent years. It is reported that the accuracy of the advanced mobile laser scanning system is as high as 5 mm under a 30 m range [27]. Its ability to acquire road objects’ 3D structural information, which 2D optical cameras cannot directly capture, is highlighted. An important factor of MLS data is the varying point density which impacts the performance of 3D point cloud interpretation algorithms. This is because it is difficult to find a reasonable neighboring size for feature extraction under these conditions. There are two elements that affect the point density of MLS data. The first one is the distance between scanners and recorded objects, which causes the change of the point density along the scanline direction. The latter one is the vehicle speed, which adds to the variation of the point density between the different scan lines.

Remote Sens. 2018, 10, 531

3 of 28

Accounting for these properties, numerous studies on MLS data interpretation have been performed. Work on 3D objects interpretation is summarized in two categories: unsupervised methods and supervised methods. Much research in recent years has focused on road furniture interpretation in the point cloud by using unsupervised methods [13,14,16,17,28,29]. Pu et al. [14] proposed a percentile-based method to recognize pole-like structures from mobile laser scanning data for road inventory studies. Firstly, they used road parts to partition the unorganized MLS data. Then they divided the point clouds into three parts: ground points, points connected to the ground, and off-ground points. Finally, they used knowledge-based methods to recognize the above-ground segments. The method was able to detect 61–87% of poles in the point clouds. Among all the subclasses, the detection rate of trees was the lowest (29.5–63.5%). However, for this method, there are problems with the detection of jointly connected pole-like objects such as trees connected with pole-like objects. Li and Oude Elberink [16] optimized the method of Reference [14] by adding reflectivity information. Because of the use of reflectance information and pulse count information, the rate of street sign detection and tree detection is largely improved. However, connected pieces of road furniture cannot be detected and recognized in these two methods. Lehtomäki et al. [13] represents an early attempt which detects vertical-pole objects by using scan line segmentation and cylinder fitting. At first, they extracted potential sweeps from the data, removing long segments and keeping short segments. Then the segments are grouped based on their profile information and horizontal plane position before these clusters that belong to one pole are merged by distance and orientation check. Finally, these clusters are classified into poles and non-poles according to their properties, such as height, shape, and orientation. The detection rate of the poles is 77.7% and the correctness is 81.0%. It is hard for this method to detect some complex pole-like objects that contain many points in the outer cylinder such as slanted pole-like road furniture or traffic signs that consist of many signboards. With a similar cylinder model mask, Cabo et al. [17] proposed a voxel-based algorithm to detect pole-like road furniture objects from MLS data. In order to make the point clouds more uniform, they first voxelized the point clouds into grids. Then they analyzed the three-dimensional information using two concentric cylinders. Finally, pole-like objects were classified from point clouds. Although this method has acquired pretty good results, there are still some limitations such as the detection of poles that are too close to bushes or guardrails. Nurunnabi et al. [28] utilized a robust diagnostic principal component analysis (RDPCA) in combination with region growing to segment road furniture into different parts. However, saliency features used in this method strongly rely on a dense point cloud. Several scholars have put effort into road furniture recognition by introducing supervised approaches [18,30–36]. Golovinsky et al. [30] utilized a normalized cut method to localize objects of interest. Different machine learning techniques in combination with shape features were used to classify these above-ground objects into sixteen categories. In the work of Munoz et al. [31], a functional gradient approach was proposed to label mobile mapping data by using Max-Margin Markov Networks (M3Ns). This method was tested for both 3D point cloud classification and geometric surface estimation in 2D images. However, this method still found it hard to separate and classify conjunctions which contain multiple objects. Different from these methods mentioned above, Huang and You [18] proposed a method in combination with a Supported Vector Machine (SVM) to classify road furniture into four categories. In order to localize pole-like objects, they implemented point cloud slicing, clustering, pole seed generation, and bucket augmentation. In the following stage, ground points are removed and the pole-like objects were extracted. In the end, six features are trained by the SVM to classify road furniture into four types. The detection rate of pole-like road furniture was 75%. Soilán et al. [34] used mobile laser scanning data in combination with images to recognize traffic signs. At the first stage, the ground points were removed by using height and intensity constraints.

Remote Sens. 2018, 10, 531

4 of 28

Traffic signs were extracted with a reflectance threshold estimated by a Gaussian Mixture Model (GMM). Then, geometric parameters of the extracted traffic signs were generated. Based on these geometric inventories, these extracted components were projected onto the corresponding camera systems, after which traffic signs can be found on the corresponding 2D images. In the end, Histogram of Oriented Gradients (HOG) features trained by the SVM are adapted to recognize traffic signs. More than 85% of pole-like traffic signs were detected. However, the detection of traffic signs in this methpd strongly relies on the reflectance information. Hackel et al. [35] described an efficient and effective method for point-wise semantic classification, which can deal with point clouds captured by LiDAR or derived from photogrammetric reconstruction with high-density variations. Instead of computing optimal neighborhoods for each point, they down-sample the entire point cloud to generate a multi-scale pyramid with decreasing point density and compute features for every voxel at every scale level. Then these features are trained and discriminant ones are selected. Finally, Random Forest (RF) is used for semantically labeling. As a drawback, RF cannot make use of contextual information. The precision of pole-like road furniture recognition is less than 35%. The Bag of Words (BoW) and Deep Boltzmann Machine (DBM) methods are applied by Yu et al. [36] to detect and recognize traffic signs in mobile laser scanning (MLS) data. Here, the authors first constructed a visual word vocabulary by using features encoded by a DBM model to detect traffic signs. Similar to Li et al. [29], they separated the poles and traffic signs which are projected to 2D images afterward. Lastly, these cropped traffic sign images were recognized by a pre-trained DBM model. More than 90% of pole-like road furniture was detected in the high-quality MLS data. A 3D convolutional neural network was proposed by Huang and You [33] to interpret urban scenes into seven categories. They first used small grids to densely voxelize the original point cloud. Then feature maps in combination with a convolutional neural network are trained to label the point clouds. 87% of pole-like road furniture were identified. Compared to unsupervised methods, supervised methods are more flexible in multi-class identification problems. Normally, supervised methods need a lot of training data. For one-class detection problems, unsupervised methods are more practical, especially when there is a limited amount of training data. How to leverage these two methods or combine them still needs to be explored. In this paper, we use a knowledge-driven method—in which generic rules are defined—to detect and decompose road furniture. All in all, significant progress has been achieved on 3D object classification and 3D scene labeling. Nevertheless, no attention has been paid to the road furniture partitioning based on shape constraints of different components, which is the innovation presented by this paper. 3. Methodology In this section, the three main stages of our framework are presented: the initial road furniture detection (Section 3.1), decomposition (Section 3.2), and the final road furniture detection (Section 3.3). Firstly, road furniture were initially extracted from unorganized mobile laser scanning points. In the following stage, these detected road furniture objects were decomposed into poles and attachments based on their spatial relations. Road furniture detection is then refined by the feedback of road furniture decomposition (Section 3.3). In addition, we introduced an approach to evaluate the result of the decomposition which is explained in Section 3.4. The workflow of our framework is illustrated in Figure 1.

Remote Sens. 2018, 10, 531

5 of 28

Figure 1. The workflow of our framework.

3.1. Initial Detection of Road Furniture The main objective of this paper is to segment road furniture within a certain distance from laser scanners. It is difficult to segment long distance points that are captured with scanners because of their low point density. Therefore, we defined a distance to remove points that are far away from the road (trajectory line), which also helps to reduce the computation cost. At this stage, we started with pre-processing, which divided the unstructured data into roadblocks, removed the ground points, and obtained the above-ground components. Then, an initial classification was performed to remove dynamic objects and detect building components. In the last step, we proposed a slice-based method to extract pole-like road furniture, trees, and pole-like road furniture connected with trees and excluded objects inside buildings by occlusion analysis. The workflow of the initial pole-like road furniture detection is as shown in Figure 2.

Remote Sens. 2018, 10, 531

6 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

6 of 28

Unorganized point cloud Cut into blocks based on trajectory dataset Ground points removal Aboveground components

Section 3.1.1

Dynamic objects removal

Facades detection

Section 3.1.2

Slice-based pole detection

Occlusion analysis

Vegetation

Pole-like road furniture

Pole-like road furniture connected with trees

Section 3.1.3 Figure 2. 2. The The workflow workflow of of the the initial initialroad roadfurniture furnituredetection. detection. Figure

3.1.1. Pre-Processing Pre-Processing 3.1.1. The original original mobile mobile laser laser point point cloud cloud files files are are often often very very large large and and result result in in computation computation and and The memoryproblems problems when processed ingo. one To circumvent these difficulties, split the memory when processed in one To go. circumvent these difficulties, we split the we unorganized unorganized point cloud into roadthe parts along the line as Reference In point cloud into road parts along trajectory linetrajectory as described in described Referencein[14]. In this [14]. paper, thislength paper,and the length and widthpart of aare road part areas specified as 50 40 m, respectively. the width of a road specified 50 m and 40 m m,and respectively. Since oneSince pieceone of piece of road furniture could be separated into two parts, the length of the overlapping zone between road furniture could be separated into two parts, the length of the overlapping zone between two two neighboring to 5 m. neighboring road road partsparts is setistoset 5 m. Ground points points are are the the main main connection connection between between different different above-ground above-groundobjects. objects. In In order order to to Ground separate different objects, ground points should be removed first. After splitting the point cloud into separate different objects, ground points should be removed first. After splitting the point cloud roadroad parts, the ground points werewere removed in each roadroad part.part. As the roadroad surfaces are are smooth, the into parts, the ground points removed in each As the surfaces smooth, height difference between all nearby points within a certain neighborhood waswas small. In contrast, for the height difference between all nearby points within a certain neighborhood small. In contrast, above-ground objects, there was a large height difference between all nearby points within a certain for above-ground objects, there was a large height difference between all nearby points within a certain neighborhood.We Wecalculated calculatedthe the height difference as the difference between the maximum height neighborhood. height difference as the difference between the maximum height and and the minimum height within a point’s neighborhood [37]. In this paper, the neighborhood the minimum height within a point’s neighborhood [37]. In this paper, the neighborhood size wassize set was to large be quite (e.g., 100 nearest neighboring points) the density high point density of to be set quite (e.g.,large 100 nearest neighboring points) because of because the high of point of the ground the ground points. When the point cloud is very sparse,beitset should beThe set lower. height threshold difference points. When the point cloud is very sparse, it should lower. heightThe difference threshold was set to be 0.15 m. Based on this property, the ground points were removed. Compared to surface growing, this method proved to be more efficient.

Remote Sens. 2018, 10, 531

7 of 28

was set to be 0.15 m. Based on this property, the ground points were removed. Compared to surface growing, this method proved to be more efficient. After the ground points are removed, the remaining above-ground points were clustered by conducting a connected component analysis. This resulted in the above-ground objects for the initial classification. 3.1.2. Initial Classification During the collection of MLS data, many moving objects or pedestrians were scanned. These unrelated objects should be removed to mitigate false positive detection of road furniture. As mentioned in the introduction, our objects of interest were road-side objects which were assumed to have traffic functionalities. However, large building façades also frequently occur in road environments. Consequently, it is necessary to remove them to reduce the computational effort for road furniture detection. On the other hand, buildings façades can provide useful information for pole-like road furniture detection. For instance, some pole-like objects inside buildings can be eliminated by using façade information. In this research, dynamic objects, buildings, and fences that are not related to road furniture were labeled thusly in this period and removed before decomposition. In the initial classification, we use the method described in Reference [37] to remove dynamic objects. A component was labeled as a dynamic object if more than 90% of the points only had k-neighbors from the same scanner. This can only be used for the data collected by multiple laser scanners. A detailed explanation is given in Reference [37]. A large number of building points in the point cloud would lead to an unnecessarily high computation time. Compared to other road objects, a façade plane is perpendicular to the ground, its area is large, and it is above a certain height. Similar to the method in References [23,24], the orientation, height, and area of the façade were selected as distinctive features to detect the façade components. In this step, surface growing was utilized to extract the planes from the components. Based on an area and angle threshold, the large vertical planes were retained afterward. 3.1.3. Slice-Based Pole-Like Road Furniture Detection In this step, pole-like road furniture, vegetation, and pole-like road furniture connected to vegetation were extracted. As known, many types of vegetation have small branches and massive leaves. When a laser pulse hits small branches or leaves, this pulse usually splits into multiple pulses before reaching the receiver sensor. In contrast, most points of other objects exclusively have a single pulse count attribute [38]. Consequently, the ratio of points with a multi-pulse count is useful for tree detection. However, there are points which belong to the edges of road furniture with multiple counts as well. If we use the number of returns, the points belong to the edges of above-ground objects are also labeled with multiple returns. Therefore, the value of the return number was utilized instead of the number of returns as a feature to extract trees. In our research, the ratio Rpc of points with the first return in above-ground components was also used as a feature to detect trees. In order to extract pole-like road furniture, Pu et al. [14] proposed a framework which allows for the general interpretation of road furniture. Their work represents an early attempt to utilize a percentile-based algorithm to detect pole-like road furniture. However, it has difficulty in detecting road furniture with a large number of attached components. For instance, it has difficulty extracting traffic lights that are connected to many traffic signs and street signs. To overcome these difficulties, a slice-based method was presented to detect pole-like road furniture. Occlusion analysis was used to remove pole-like objects behind façades afterward. The workflow of this method is as shown in Figure 3.

Remote Sens. 2018, 10, 531

8 of 28

Figure 3. The workflow of detecting pole-like road furniture.

First, every individual above-ground component was cut into horizontal slices (Figure 4). Then a connected component analysis was performed for every slice to produce separated components. The center point of every separated slice component was computed and a 2D connected component analysis was applied to connect the components which were very close to each other in the horizontal plane. Here, three constraints were applied to compute the number of pole-slices for the connected slice components in every individual above-ground component. The first constraint was the displacement ds of the center points of two neighboring slices (Figure 4). There should be no large displacement between the two neighboring slices. The second one was the difference of the diameter of two neighboring slices. The diameter of a slice was the 2D largest distance between two points in this slice. The difference of diameters between the two neighboring slices should be small (e.g., 0.2 m), assuming that a part of the pole will have no attached objects. The third constraint was the diameter d of a slice (Figure 4). The diameter of a slice should be smaller than a pre-determined threshold. Finally, the number of pole slices was checked for every individual above-ground component. If the number of pole-slices was larger than a specified threshold (set to 3) and the ratio Rpc was smaller than a threshold (0.05), this component was labeled a pole-like road furniture. If the number of pole slices was larger than a specified threshold and the ratio Rpc was larger than the threshold, this component was labeled a pole-like road furniture connected to trees. If both of them were smaller than their corresponding thresholds, this component was labeled trees.

Remote Sens. 2018, 10, 531

9 of 28

Figure 4. The slice-based detection of pole-like road furniture.

Among the detected pole-like road furniture candidates, there were incorrectly detected objects which were located behind façades. To exclude them, an occlusion analysis is performed. In the step of the initial classification, façade planes were obtained through surface growing. Then these façade planes were computed as constraints to determine if these pole candidates were located outside of the façade planes (Figure 5). If they were positioned outside of the façade, they were labeled as pole-like road furniture, otherwise, they were not.

Figure 5. Occlusion analysis.

Remote Sens. 2018, 10, 531

10 of 28

3.2. Road Furniture Decomposition ToSens. address the problem that some road furniture had multiple functions, we presented10a ofnew Remote 2018, 10, x FOR PEER REVIEW 28 method to decompose road furniture into different components based on their spatial relations. analyzed thethe relationship between attachments and vertical and We analyzed relationship between attachments and vertical andhorizontal horizontalpoles. poles.Therefore, Therefore,ititisis crucial to know which points belong to which pole. The remaining points have spatial crucial to know which points belong to which pole. The remaining points have spatialrelations relationstoto eitherhorizontal horizontalor orvertical vertical poles. poles. Based Based on on these these relations, either relations, we we presented presented an an optimal optimalsegmentation segmentation procedure. The overall workflow of this method is as shown in Figure 6. There is a large procedure. The overall workflow of this method is as shown in Figure 6. There is a largevariety varietyofof structures and shapes in pole-like road furniture. It is difficult to extract all the poles using structures and shapes in pole-like road furniture. It is difficult to extract all the poles usingaasingle single method.Therefore, Therefore, we we first first identified identified some some key key properties method. properties of of the the road road furniture, furniture,based basedon onwhich whichthe the most suitable pole extraction method was selected (Section 3.2.1). Next, the poles were removed and most suitable pole extraction method was selected (Section 3.2.1). Next, the poles were removed and theattached attachedcomponents componentswere wereseparated separated into into different different parts parts (Section the (Section 3.2.2). 3.2.2). In Inthe theend, end,the therule-based rule-based splitting and merging were carried out to refine the decomposition. splitting and merging were carried out to refine the decomposition. Road furniture point cloud

Preidentification

2D point density based method

RANSAC line fitting method

Slice cutting based method

Pole-extraction optimization Section 3.2.1

Poles removal and connected components analysis Rule-based splitting and merging Section 3.2.2

Figure 6. The flow chart of pole-like road furniture decomposition. Figure 6. The flow chart of pole-like road furniture decomposition.

3.2.1. 3.2.1.Pole-Extraction Pole-Extraction Most furniturecomprise compriseofofpoles poles and other attached components. These Mostpole-like pole-like road road furniture and other attached components. These polespoles are are the link between different attached components. The motivation of this step is to extract the link between different attached components. The motivation of this step is to extract this this link.link. A Aframework frameworkfor forpole poleextraction extraction is presented in our previous work [29]. is presented in our previous work [29]. The Thestructures structuresand andshapes shapesof ofpole-like pole-likeroad roadfurniture furniturevary varyaalot, lot,which whichmakes makesititdifficult difficultto touse usea single method toto extract attachmentsconnected connectedwith withpoles, poles, a single method extractpoles. poles.For Forexample, example,in inthe thecase case of of many many attachments ititisisdifficult to directly extract poles based on their linear features. Thus, 2D point density can be difficult to directly extract poles based on their linear features. Thus, 2D point density can be adopted as a feature to extract poles. We categorized the pole-like road furniture into three typical adopted as a feature to extract poles. pole-like road furniture into three typical types. was road furniture with many attachments (the left in Figure 7). The7). second types.The Thefirst firsttype type was road furniture with many attachments (theimage left image in Figure The one was one roadwas furniture with horizontal poles (the middle Figurein7),Figure and the type was second road furniture with horizontal poles (theimage middleinimage 7), third and the third normal road furniture (the right image in Figure 7). For these three different types of road furniture, type was normal road furniture (the right image in Figure 7). For these three different types of road three corresponding pole extraction proposed. furniture, three corresponding pole methods extractionwere methods were proposed.

Remote Sens. 2018, 10, x FOR PEER REVIEW Remote Sens. 2018, 10, 531

11 of 28 11 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

11 of 28

Figure 7. The three types of pole-like road furniture items. Figure 7. The The three types ofpole-extraction, pole-like road road furniture Figure 7. types of pole-like items. In order to select the method tothree be used for we first did the pre-identification to get information from the road furniture such as the length and width of its bounding box. To obtain In order order to to select select the method method to be be used used for for pole-extraction, pole-extraction, we we first first did did the the pre-identification pre-identification to to In the knowledge of the the structure ofto the road furniture, we cut pole-like road furniture into slices and get information information from from the the road road furniture furniture such such as the the length length and and width of of its its bounding bounding box. box. To To obtain obtain get computed the width of every slice. In theasprevious phase,width we already cut the above-ground the knowledge of the structure of the road furniture, we cut pole-like road furniture into slices and the knowledge ofslices the structure the road pole-like road furniture intoinformation slices and components into to check of if they werefurniture, pole-like. we Wecut do not use the previous slicing computed the width of every slice. In the previous phase, we already cut the above-ground components computed width of previous every slice. In inthe we already cut stage the above-ground because thethe width of the slicing theprevious pole-like phase, road furniture detection is rather large into slices to into check if they were if pole-like. Wepole-like. do not use previous slicing information because the components slices to check were Wethe dotogether. not use the slicing information (e.g., 0.3 m) because it was used tothey connect small fragments In previous this phase, the width was set width of the previous slicing in the pole-like road furniture detection stage is rather large (e.g., 0.3 m) because the width of the previous slicing in the pole-like road furniture detection stage is rather large to 0.1 m. The bounding box of every individual road furniture is then calculated. The median width because it was usedittowas connect small fragments together. In this phase, thephase, widththe was set towas 0.1set m. (e.g., m) because used connect small fragments In this width of the0.3 slices can be obtained byto the statistics of every slice.together. The maximum variation of the distance The bounding box of every individual road furniture is then calculated. The median width of the slices to 0.1 m. The box of every individual furniture is thencan calculated. The median between the bounding points of one road furniture itemroad in the XY direction be calculated based width on its canthe beslices obtained by obtained the statistics of every slice. The maximum variation of the distance between the of can be by the statistics of every slice. The maximum variation of the distance bounding box. points of the one points road furniture item furniture in the XY direction canXY be direction calculatedcan based on its bounding box. between of one road in the calculated based on the its According to the properties of roaditem furniture calculated in the be pre-processing phase, According to the properties of road furniture calculated in the pre-processing phase, bounding box. corresponding pole extraction method was selected (Figure 8). If more than half of the pole the corresponding poleproperties extraction of method was selected (Figure in 8).the If more than half of the pole According to the road furniture calculated pre-processing phase, component consists of attached components, a 2D point density-based method was utilized. This the can component consists ofextraction attached components, a 2D point density-based method was utilized. This be corresponding method was selected (Figure 8). If more than half of theacan pole be determined pole by comparing the smallest quartile width and median width. If there was large determined by comparing the smallest quartile width and median width. If there was a large difference, component consists of attached components, 2D point density-based was utilized. This can difference, we believed that there were manya components attached tomethod the pole. Otherwise, Random we believed that there were many components attached to the pole. Otherwise, Random Sample be determined by comparing the smallest quartile and median width.poles If there a large Sample Consensus (RANSAC)-based line fitting waswidth utilized if the horizontal werewas included in Consensuswe (RANSAC)-based linewere fitting wascomponents utilized if the horizontal poles were included in the difference, believed that many attached to the pole. Otherwise, Random the road furniture. These canthere be detected by checking the bounding box of the piece of road furniture. road furniture. These can be detectedline by fitting checking the bounding of the piece of road furniture. Sample Consensus (RANSAC)-based utilized if the box horizontal poles were in If the length or width of the bounding box in thewas horizontal direction was larger than theincluded threshold, If the length or width of the bounding box in the horizontal direction was larger than the threshold, the furniture. These canfurniture be detected by checking boundingpoles box ofwas the made. piece ofFor road furniture. theroad decision that this road likely containedthe horizontal normal road the decision that this road furniture likely contained horizontal poleswas waslarger made. Forthe normal road If the length or width of the bounding box in the horizontal direction than threshold, furniture, there were more robust features to extract poles than using 2D point density features. If furniture, there were robust features to extract poles than using 2D point densityFor features. If there the decision that thismore road furniture likely horizontal poles was made. there were not many attachments to the contained road furniture, the line fitting was morenormal robust road than were not many attachments to the road furniture, the line fitting was more robust thanfeatures. extracting furniture, features to Consequently, extract poles than usingthe 2Dslice point density If extractingthere pointswere withmore high robust 2D point density. we used cutting based method pointswere with not highmany 2D point density. Consequently, we used the slice cutting basedmore method to fitthan the there attachments to the road furniture, the line fitting was robust to fit the center lines of poles and perform pole extraction. Finally, when there were no attachments center linespoints of poles and perform poledensity. extraction. Finally, when no attachments to the pole, extracting high 2D for point Consequently, wethere usedwere the slice cutting based method to the pole, there with was no need decomposition. there was no need for decomposition. to fit the center lines of poles and perform pole extraction. Finally, when there were no attachments to the pole, there was no need for decomposition. Pole-like street furniture Pole-like point cloud street furniture point cloud Many attachments?

Yes

2D point density based method 2D point

Many attachments? No

Yes

Horizontal No pole?

Yes

RANSAC line fitting

Yes No

RANSAC line fitting Slice cutting based method

Horizontal pole?

density based method

Slice cutting Figure 8. The pole extraction No method selection. Figure 8. The pole extraction method selection. based method

Figure 8. The pole extraction method selection.

Remote Sens. 2018, 10, 531 Remote Sens. 2018, 10, x FOR PEER REVIEW

12 of 28 12 of 28

The is given The brief brief explanation explanation of of these these three three pole pole extraction extraction methods methods is given as as follows. follows. In In the the 2D 2D point point density-based method, first calculated aa Remote Sens. 2018, 10, x FORwe PEER REVIEW 12 ofused 28 density-based method, we first calculated the the 2D 2D point point density density around around every every point. point. Then Then we we used region growing method to extract the clusters of points with high 2D point density (Figure 9a) which region growing method to extract the clusters of points with high 2D point density (Figure 9a) which The brief explanation of these three poleextraction extraction methods is given as follows. In the 2D pointfirst were recognized as poles. were recognized as poles. In In the the second second pole pole extraction method, method, points points with with high high linearity linearity were were first density-based method, we first calculated the 2D point density around every point. Then we used a extracted the eigenvalues eigenvalues of of every every point’s point’s neighboring neighboring points. points. Then extracted by by calculating calculating the Then the the RANSAC RANSAC region growing method to extract the clusters of points with high 2D point density (Figure 9a) which algorithm algorithm was was adopted adopted to to extract extract lines lines from from these these points points with with high high linearity. linearity. Poles Poles were were extracted extracted by by were recognized as poles. In the second pole extraction method, points with high linearity were first detecting points within aa radius to the fitted lines (Figure (Figure 9b). 9b). In the slice cutting method, we first detecting points within radius to the fitted lines In the slice cutting method, we first extracted by calculating the eigenvalues of every point’s neighboring points. Then the RANSAC extracted the center lines by carrying out the RANSAC algorithm with center points of cut extracted thewas center linesto byextract carrying the RANSAC algorithm with the the center of the the algorithm adopted linesout from these points with high linearity. Poles werepoints extracted by cut slices. Similar to the second method, the poles were then extracted based on the distance between slices. Similar to the second method, thefitted poles were then 9b). extracted based on themethod, distance detecting points within a radius to the lines (Figure In the slice cutting webetween first the points andand fitted lineslines (Figure A more detailed description of these three methods by by Li the extracted points 9c). Aout more description of these three methods is given thefitted center lines(Figure by 9c). carrying the detailed RANSAC algorithm with the center pointsisofgiven the cut et al. (2016). Li et al. (2016). slices. Similar to the second method, the poles were then extracted based on the distance between the points and fitted lines (Figure 9c). A more detailed description of these three methods is given by Li et al. (2016).

(a)

(b)

(c)

Figure pole extraction. (a) Pole Pole extraction extraction using the the 2D 2D point method;(c) (b) pole pole (a)9. (b)using Figure 9. The The pole extraction. (a) point density-based density-based method; (b) extraction using using the the RANSAC-based RANSAC-based line line fitting fitting method; method; (c) (c) pole-extraction pole-extraction using using the the slice cutting extraction cutting Figure 9. The pole extraction. (a) Pole extraction using the 2D point density-based method;slice (b) pole based method. method. based extraction using the RANSAC-based line fitting method; (c) pole-extraction using the slice cutting based method.

Poles extracted extracted by the first two methods (2D point density method and RANSAC RANSAC line line fitting the firstFor twothe methods (2Ddensity-based point density method, method and RANSAC fitting method)Poles wereextracted often notby accurate. 2D some points attachments For the 2D point point density-based method, points ofline attachments method) were often not accurate. For the 2D point density-based method, some points of attachments that are near the poles have a high 2D point density. They can thereby be categorized as poles. that are near the poles have a high 2D point density. They can thereby be categorized as When poles. thatpoints are near the poles linearity have a high 2D point density. They can thereby categorized poles. points with high linearity were not extracted accurately enough, thebe center ofas the poles When with high were not extracted accurately enough, the lines center lines of When thecannot poles points withcorrectly high linearity were not extracted accurately enough, the center lines of the poles cannot be estimated using the RANSAC algorithm in the second method. If a pole is inaccurately cannot be estimated correctly using the RANSAC algorithm in the second method. If a pole is be estimatedacorrectly the RANSAC algorithm in the method. If a pole if is inaccurately generalized line, thisusing causes imprecise extraction ofsecond the pole. For example, generalized inaccuratelyto generalized to a line,the this causes the imprecise extraction of the pole. Forthe example, if the generalized to a line, this causes the imprecise extraction of the pole. For example, if the generalized center lines inclined towards the towards street side 10a),(Figure the part10a), of the that farwould away generalized center lines inclined the(Figure street side thepoints part of thewould pointsbe that center lines inclined towards the street side (Figure 10a), the part of the points that would be far away from the center lines and belong to belong the poles would not be extracted in this stage. To tackle these be far away center and to the poles not be extracted in this To tackle from the from centerthe lines and lines belong to the poles would notwould be extracted in this stage. To stage. tackle these problems, pole extraction was optimized by re-estimating the center lines of the Similar to the these problems, pole extraction optimized by re-estimating the center of poles. the Similar poles. Similar problems, pole extraction waswas optimized by re-estimating the center lines oflines the poles. to the to slice cutting, we cut the extracted points of poles into slices, computed their center points, and used the slice slicecutting, cutting,we wecut cutthe theextracted extractedpoints points poles into slices, computed their center points, used ofof poles into slices, computed their center points, andand used the RANSAC algorithm to extract the lines from these center points. By doing this, poles can thethe RANSAC algorithm these center centerpoints. points.ByBy doing this, poles be RANSAC algorithmtotoextract extractthe the lines lines from from these doing this, poles can can be be extracted more accurately byby using the precisely fitting center lines. The re-estimated center lines extracted more accurately by using the precisely fitting center lines. The re-estimated center lines extracted more accurately using the precisely fitting center lines. The re-estimated center lineswere were shown in in Figure 10b. were Figure 10b. shown inshown Figure 10b.

(a)

(a)

Figure 10. Cont.

Remote Sens. 2018, 10, 531

13 of 28

Remote Sens. Sens. 2018, 2018, 10, 10, xx FOR FOR PEER PEER REVIEW REVIEW Remote

13 of of 28 28 13

(b) (b) Figure 10. The extracted center lines of poles. The redlines linesare arethe the extracted center ofpole. the pole. Figure 10. 10. The The extracted extracted center center lines lines of of poles. poles. The The red red extracted center lineslines of the the Figure lines are the extracted center lines of pole. (a) The extracted center lines thestreet streetside sideand and optimized center (a) The extracted center lineswere wereinclined inclined towards towards the (b)(b) thethe optimized center lines.lines. (a) The extracted center lines were inclined towards the street side and (b) the optimized center lines.

3.2.2. Decomposition into Poles 3.2.2. Decomposition into Polesand andAttachments Attachments 3.2.2. Decomposition into Poles and Attachments In order order to separate separate theattachments attachmentswhich which are are connected totothe the poles, wewe removed the extracted extracted In order to separate the areconnected connectedto the poles, removed the extracted In to the attachments which poles, we removed the poles and performeda aaconnected connected component component analysis analysis [29]. As As the the components can be be very close toclose poles performed connected component [29]. As the components can very poles andand performed analysis [29]. components can beclose veryto each other, the maximum distance and the size of the nearest neighborhood should be chosen each other, the maximum distance and the size of the nearest neighborhood should be chosen to each other, the maximum distance and the size of the nearest neighborhood should be chosen properly. Consideringthe thedistance distance between between the the scanlines and the point distribution on aaonsingle single properly. Considering the distance between point distribution on properly. Considering the scanlines scanlinesand andthe the point distribution a single scanline, the the neighborhood neighborhood size size cannot cannot be be very very small. small. Otherwise, Otherwise, points points on on different different scanlines scanlines would would scanline, scanline, the neighborhood size cannot be very small. Otherwise, points on different scanlines would not be be connected. connected. The The maximum maximum distance distance between between two two scanlines scanlines here here is is 0.05 0.05 m. m. For aa point, point, the the not not be connected. The maximum distance between two scanlines here is 0.05 m. For For a point, the number number of its neighboring points was typically 10 within the distance of 0.05 m in a single scanline. number of its neighboring points was typically 10 within the distance of 0.05 m in a single scanline. of its neighboring points was typically 10 within the distance of 0.05 m in a single scanline. Here, Here, we we set set the the maximum maximum for for the the connected connected components components to to 0.15 0.15 m m and and the the neighborhood neighborhood size size to to 15 15 Here, we set the maximum forcomponents the connected components to 0.15 mThe andbest themethod neighborhood size to 15 points. points. The separated are as shown in Figure 11. for each piece of road points. The separated components are as shown in Figure 11. The best method for each piece of road Thefurniture separatedwas components are as shown in Figure 11. The best method for piece of road furniture furniture was chosen automatically. automatically. Figure 11a shows shows the partitioning of each the remainder remainder of points points chosen Figure 11a the partitioning of the of waswhich chosen automatically. Figure 11a shows the partitioning of the remainder of points which which were were obtained obtained from from aa 2D 2D point point density-based density-based pole pole extraction extraction method. method. Pole Pole extraction extraction in in Figure Figurewere obtained from 2D pointline density-based pole extraction method. Pole extraction in Figure uses the 11b uses uses theaRANSAC RANSAC line fitting method method and pole extraction extraction in Figure Figure 11c,d utilized utilized the slice slice11b cutting 11b the fitting and pole in 11c,d the cutting based method. method. RANSAC line fitting method and pole extraction in Figure 11c,d utilized the slice cutting based method. based

(a) (a)

(b) (b)

(c) (c)

(d) (d)

Figure 11. Examples of the separated components. The separated components after (a) 2D point Figure 11. Examples Examples of the the separated separated components. Figure 11. of components. density-based pole extraction, (b) RANSAC line fitting based pole extraction, (c) and (d) slice cutting Because of occlusion, occlusion, low low point point density, density, and and noisy noisy points, points, it it is is sometimes sometimes difficult difficult to to connect connect all all based pole-extraction. Because of the points that belong to the same attachment by leveraging the parameters of the aforementioned the points that belong to the same attachment by leveraging the parameters of the aforementioned Because of occlusion,analysis. low point density,are and noisy in points, it 11. is to connect connected component analysis. Examples are marked in Figure 11.sometimes To address addressdifficult this problem, problem, we all connected component Examples marked Figure To this we the defined points that belong to the same attachment by leveraging the parameters of the aforementioned defined a set of generic rules to split and merge the components. We first performed merging rules a set of generic rules to split and merge the components. We first performed merging rules connected Examples areThen marked Figure 11. connected To addressto problem, wewere defined for the the component componentsanalysis. for horizontal horizontal poles. Then the in components connected tothis vertical poles were for components for poles. the components vertical poles analyzed to see see whether theyand could be merged merged or split. split. Then, Then,We thefirst detached components wererules checked a setanalyzed of generic rules to split merge the components. performed merging for the to whether they could be or the detached components were checked on whether whether they could be bepoles. merged withthe their nearby components. components. on could merged with their nearby components forthey horizontal Then components connected to vertical poles were analyzed to As shown shown in the thebe redmerged circle of oforFigure Figure 11b, one one component attached to to aa were horizontal poleon canwhether be As in red circle horizontal pole can be see whether they could split. 11b, Then, thecomponent detached attached components checked separated into two parts during the connected component analysis. If two components were attached separated into two parts during the connected component analysis. If two components were attached they could be merged with their nearby components. to the the same same horizontal horizontal pole pole and and their their positions positions overlap overlap in in XY-plane, XY-plane, these these two two components components should should to As shown in the red circle of Figure 11b, one component attached to a horizontal pole can be be merged. The merging analysis of the horizontal pole attached components was repeated until no be merged. The merging analysis of the horizontal pole attached components was repeated until no separated into two parts during the connected component analysis. If two components were attached such components components were were found. found. Figure Figure 12 12 shows shows the the merged merged components components after after applying applying the the merging merging rule. rule. such

to the same horizontal pole and their positions overlap in XY-plane, these two components should be merged. The merging analysis of the horizontal pole attached components was repeated until no such components were found. Figure 12 shows the merged components after applying the merging rule.

Remote Sens. 2018, 10, 531

14 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

14 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

14 of 28

Figure12. 12. The frame is the in image the merged after applyingafter the merging rule. Figure Theleft left frame is zoomed the zoomed in of image of thecomponent merged component applying the merging rule. Another situation was that the attachments might be connected to each other because of noisy points imprecise poleisthat extraction. An example ismerged shown in the Figure 11c. this situation, wenoisy Another was the attachments might becomponent connected toapplying eachInother because of Figureor 12.situation The left frame the zoomed in image of the after the merging rule. increase the width of the pole extraction. If one component can be split into two parts and they are points or imprecise pole extraction. An example is shown in the Figure 11c. In this situation, we increase not at the same height,was thisthat component was splitmight into two parts. This is similar Another situation tomethod each other because oferosion noisy the width of the pole extraction. Ifthe oneattachments component can be be connected split into two parts and theytoare not at the followed dilation.pole extraction. An example is shown in the Figure 11c. In this situation, we points or by imprecise same height, this component was split into two parts. This method is similar to erosion followed Suppose 𝑪𝒗𝒆𝒓𝒕𝒊of (𝑷the is a component is attached to vertical poles the width is setare to increase the width extraction. which If one component can be split intowhen two parts and they 𝒘 ) pole by dilation. {𝒘𝟏 ,split }—where 𝒘 for extraction. increased 𝒘 =was 𝒘𝟐 , ⋯into , 𝒘𝒏two is the width of the cut slice 𝒊— not at pole the same height,We this component parts. 𝒘 This method is similar to erosion 𝒊 Suppose Cverti attached poles the width is set w ) is a component and performed a(Pconnected component which analysis.isOnce 𝑪𝒗𝒆𝒓𝒕𝒊to (𝑷𝒘vertical ) becomes the when two components— followed by dilation. to w𝑪for pole We increased wnot =at w1same , w2 , ·height, , wnthey wiseparated is width the width }—where (𝑷𝒘′ )𝒊extraction. and 𝑪𝒗𝒆𝒓𝒕𝒊 )a𝒋 —and they are should be into 𝑪𝒗𝒆𝒓𝒕𝒊 (𝑷𝒘(𝑷 ) 𝒘′ is component which is{the attached to· ·vertical poles when the is settwo toof the 𝒗𝒆𝒓𝒕𝒊Suppose cut slice performed a connected component analysis. Once C P becomes the two ( ) parts. This extraction. operation will until increment of 𝒘 reaches a predefined value. In this paper, {𝒘 }—where w 𝒘 fori—and pole Wecontinue increased 𝒘 =the , 𝒘 , ⋯ , 𝒘 𝒘 is the width of the cut slice 𝒊— verti 𝟏 𝟐 𝒏 𝒊 it is empirically set to 0.15 m for the splitting analysis. components—C P and C P —and they are not at the same height, they should be ( ) ( ) and performed a connected component analysis. Once 𝑪 (𝑷 ) becomes the two components— w0 i w0 j verti verti 𝒗𝒆𝒓𝒕𝒊 𝒘 There are single attachments which might be separated into two parts (Figure 11a). This case 𝑪 (𝑷 ) and 𝑪 (𝑷 ) —and they are not at the same height, they should be separated into two separated 𝒗𝒆𝒓𝒕𝒊 into 𝒘′ 𝒊two parts. 𝒗𝒆𝒓𝒕𝒊 This 𝒘′ 𝒋 operation will continue until the increment of w reaches a predefined occurs when theitdistance for connection is not large enough during the connected parts. This operation continue until the0.15 increment of 𝒘 reaches a predefined value. Incomponent this paper, value. In this paper, iswill empirically set to m for the splitting analysis. analysis. In this case, we conducted a merging analysis. We continued increasing the width ofThis the case it is empirically setattachments to 0.15 m for the splitting analysis. There are single which might be separated into two parts (Figure 11a). ″ ″ pole There extraction to 𝒘 . Here, we set 𝒘 to be the normal size of traffic signs, for example, 0.5 m. Once single attachments which might be enough separatedduring into two (Figurecomponent 11a). This case occurs when theare distance for connection is not large theparts connected analysis. the twowhen components 𝑪𝒗𝒆𝒓𝒕𝒊 (𝑷 )𝒊 and 𝑪𝒗𝒆𝒓𝒕𝒊is(𝑷not are wrapped their connected vertical pole and occurs the distance for enough by during the connected component 𝒘″ connection 𝒘″ )𝒋large In this case, we conducted a merging analysis. We continued increasing the width of the pole extraction once the two components are in the same height,analysis. the connection line of their center points comes analysis. In this case,00 we conducted a merging We continued increasing the width ofvery the to w00pole . Here, set w Then to bethey theshould normal size of traffic signs, for example, 0.5will m.0.5 Once the two near to thewe pole together. merging analysis be iterated extraction toline. 𝒘″. Here, we set 𝒘″ tobe bemerged the normal size ofThe traffic signs, for example, m. Once components C P and C P are wrapped by their connected vertical pole and once 00 00 ( ) ( ) verti verti w 𝑪 w The until nocomponents components were split as shown theand red the ican j components ″ )𝒊 and the two (𝑷𝒘merged. 𝑪𝒗𝒆𝒓𝒕𝒊 (𝑷𝒘″ )𝒋 are wrapped byand theirmerged connected verticalinpole 𝒗𝒆𝒓𝒕𝒊be two components are in the same height, the connection line of their center points comes very circlethe of Figure 13. once two components are in the same height, the connection line of their center points comes very near to thenear pole Then they should be merged together. TheThe merging analysis to line. the pole line. Then they should be merged together. merging analysiswill willbe beiterated iterated until until no components can be merged. The components were and splitmerged and merged as shown in the no components can be merged. The components were split as shown in the red red circle of circle Figure 13. of Figure 13.

Figure 13. The components connected with vertical poles are split (left) and merged (right) after applying the fitting rules. Figure 13. The componentsconnected connected with with vertical split (left) andand merged (right) after after Figure 13. The components verticalpoles polesare are split (left) merged (right) applying the fitting rules. applying the fitting rules.

Remote Sens. 2018, 10, 531

15 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

15 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

15 of 28

Components poles (Figure (Figure 11d) 11d)were weredescribed described detached Componentsthat thatwere were not not connected connected to poles as as detached components because of the low point density or occlusion. It is assumed that if one detached components because the low density occlusion. is assumed that if one Components thatofwere not point connected to poles (Figure It11d) were described as detached component and an were thesame same height, on Itsame the andifconnection their connection component and anattachment attachment were atatthe height, on the side, side, andthat their line components because of the low point density or occlusion. is same assumed one detached was pole, they should bebe one component merged together. Then, thisthis merged linecomponent wasvery veryclose close tothis this pole, they should one component and merged together. Then, merged andtoan attachment were at the same height, on and the same side, and their connection line component was continue withbe the detached components merging analysis. detached was very close toadded this pole, they should one component and merged together. Then,The this merged component was added totocontinue with the detached components merging analysis. The detached components were merged as illustrated in Figure 14. componentwere was merged added toascontinue withinthe detached components illustrated Figure 14. components merging analysis. The detached components were merged as illustrated in Figure 14.

Figure14. 14. The The detached detached components Figure componentsmerging. merging. Figure 14. The detached components merging.

Final Detection Pole-LikeRoad RoadFurniture Furniture 3.3.3.3. Final Detection ofofPole-Like 3.3. Final Detection of Pole-Like Road Furniture The results thedecomposition decompositionwere were imported imported as road furniture detection The results ofof the as ititisisfeedback feedbackfor forthe the road furniture detection stage.The If there was no pole extracted from a road furniture item such as the pillars of buildings, this results of the decomposition were imported as it is feedback for road furniture detection stage. If there was no pole extracted from a road furniture item such as the pillars of buildings, this road road furniture itemnowould not be labeled pole-like. For item example, thethe pillars of of buildings canthis be stage. Ifitem there was pole extracted aasroad furniture such as buildings, furniture would not be labeled asfrom pole-like. For example, the pillars ofpillars buildings can be detected detected as a pole-like roadnot furniture item as at the detection When a pillar is decomposed road furniture item would be labeled pole-like. For stage. example, thesuch pillars of buildings can be as a pole-like road furniture item at the detection stage. When such a pillar is decomposed by using by using as RANSAC lineroad fitting, this pillar probably notstage. be extracted because theispercentage of detected a pole-like furniture itemwill at the detection When such a pillar decomposed RANSAC line fitting, this pillar will probably not be extracted because the percentage of points which points which belongline to the polethis is low when are many withbecause a high linearity from the by using RANSAC fitting, pillar will there probably not bepoints extracted the percentage of belong the pole is when there are many points with a high linearity from the edges of façades edgestoof façades orlow fragments. points which belong to the pole is low when there are many points with a high linearity from the or fragments. edges of façades or fragments. 3.4. Road Furniture Decomposition Evaluation 3.4. Road Furniture Decomposition Evaluation 3.4. Road Furniture Evaluation In this section,Decomposition an approach was presented to evaluate the accuracy of the decomposition. Based In this section, an approach was presented the Based on their spatial relations, the components of to pole-like road furniture were manually labeled. For on In this section, an approach was presented toevaluate evaluate theaccuracy accuracyof ofthe thedecomposition. decomposition. Based their spatial relations, the oflabeled pole-like road and furniture were manually labeled. For15. example, example, street signs cancomponents be manually “pole attachments”, as shown in Figure on their spatial relations, the components of as pole-like road furniture were manually labeled. For street signs can be manually labeled as “pole and attachments”, as shown in Figure 15. example, street signs can be manually labeled as “pole and attachments”, as shown in Figure 15.

Figure 15. Examples of the reference data. Figure15. 15. Examples Examples of Figure ofthe thereference referencedata. data.

Remote Sens. 2018, 10, 531

16 of 28

In order to assess the results of the decomposition, we used a two-level evaluation method. One is Remote evaluation Sens. 2018, 10, x FOR 16 of 28 point-based andPEER theREVIEW other one is a component-based evaluation. We used completeness and correctness to quantify the evaluation. In order to assess the results of the decomposition, we used a two-level evaluation method. One In the point-based evaluation, we first selected the corresponding true positive points of the is point-based evaluation and the other one is a component-based evaluation. We used completeness manually labeled ground truth component. and correctness to quantify the evaluation.We matched the corresponding decomposition result and In truth the point-based evaluation, we firstthe selected thedecomposed correspondingcomponent true positivein points the every ground component by selecting largest this of ground truth manually ground truth component. We matched corresponding decomposition component. The labeled true positive points in an attachment arethethe points belonging to bothresult this and attachment every ground truth component by selecting the largest decomposed component in this ground truth and its corresponding manually labeled attachment. Then the completeness of every component component. The true positive points in an attachment are the points belonging to both this attachment was computed as the ratio of the number of points of the largest segment in this component to the and its corresponding manually labeled attachment. Then the completeness of every component was numbercomputed of pointsasof this component, = TP/ TP is the to number of correctly (TP +inFN ). component the ratio of the numberCompleteness of points of the largest segment this the number decomposed points in the manually𝑪𝒐𝒎𝒑𝒍𝒆𝒕𝒆𝒏𝒆𝒔𝒔 labeled components, is the number incorrectly decomposed of points of this component, = 𝑻𝑷/(𝑻𝑷FN + 𝑭𝑵) . 𝑻𝑷 is the of number of correctly points in the manually labeled components, 𝑭𝑵 is16, the of incorrectly points indecomposed the manually labeled components. For example, in Figure thenumber true positive component of decomposed the manually component labeled components. example,by in Figure the true the streetlight head points is the in decomposed which For is marked the red16,circle 1 inpositive the left figure. component of the streetlight head is the decomposed component which is marked by the red circle 1 Thus, the completeness of this component can be computed as the ratio of the number of points which in the left figure. Thus, the completeness of this component can be computed as the ratio of the are in the overlap of thewhich component red circle 1 in with the left figure and number of points are in thelabeled overlap with of thethe component labeled the red circle 1 inthe the component left labeled with 1 in the rightwith figure, the number of points of the thenumber component labeled with figure green and thecircle component labeled greentocircle 1 in the right figure, to of points of thecircle component labeled the green circle 1 in the right figure. the green 1 in the rightwith figure.

Figure Figure 16. The components andmanually manually labeled components 16.decomposed The decomposed components (left) (left) and labeled components (right). (right). The correctness of a separated attachment is the number of true positive points divided by the

Thenumber correctness of of a separated attachment theexample, numberinofFigure true positive points divided of points this separated attachment.isFor 16, the correctness of this by the numbercomponent of pointsisofcomputed this separated attachment. example, 16, of the as the ratio of the numberFor of points which in areFigure the overlap thecorrectness component of this labeled with the red as circle in the of leftthe figure and theof component labeledare with theoverlap green circle 1 incomponent the component is computed the1 ratio number points which the of the right figure, to the number of points of the component labeled with a green circle 1 in the right figure. labeled with the red circle 1 in the left figure and the component labeled with the green circle 1 in the 𝑪𝒐𝒓𝒓𝒆𝒄𝒕𝒏𝒆𝒔𝒔 = 𝑻𝑷/(𝑻𝑷 + 𝑭𝑷) . 𝑭𝑷 is the number of incorrectly decomposed points in the right figure, to the number of points of the component labeled with a green circle 1 in the right figure. components produced by our algorithm. Those points which are labeled with the green circle 1 in the Correctness = TP/(TP + FP). FP is the number of incorrectly decomposed points in the components right figure are included as well in the segment which is labeled with the red circle 1 in the left figure. produced byInour algorithm. Those points which aresegment labeledinwith greenlabeled circle 1ground in thetruth right figure the component-based evaluation, the largest everythe manually are included as well the segment which is labeled with circle 1Then in the figure. the component wasinselected as the corresponding segment of the this red component. weleft calculated completeness and correctness of every by using point-based evaluation. the In the component-based evaluation, the component largest segment in every manually labeled Ifground truth completeness was lower than the threshold 𝒐𝒗𝒆𝒓_𝒅𝒆𝒄𝒐 , this component was labeled as over𝒕𝒉 component was selected as the corresponding segment of this component. Then we calculated the decomposed. If the correctness was lower than the threshold 𝒖𝒏𝒅𝒆𝒓_𝒅𝒆𝒄𝒐𝒕𝒉 , this component was completeness and correctness of every component by using point-based evaluation. If the completeness was lower than the threshold over_decoth , this component was labeled as over-decomposed. If the correctness was lower than the threshold under_decoth , this component was labeled as underdecomposed. Otherwise, this component was an applicable decomposed component. Both over_decoth and under_decoth were defined to be 0.6. For example, in the left image of Figure 16, component 2

Remote Sens. 2018, 10, 531

17 of 28

labeled with a green circle is under-decomposed with low correctness. In contrast, component 3 is over-decomposed with low completeness. 4. Study Area and Experimental Result To test the performance of the proposed framework, two datasets—one in Enschede and one in Paris—are chosen. The Enschede dataset was used as a test case because there are many types of road furniture and it is a typical Dutch city. The Paris dataset was a benchmark dataset and Paris is a French city. The shapes of the road furniture in these two test sites are different because they are from two different countries. The test of the presented framework on the Enschede dataset is given in Sections 4.1 and 4.2 gives the experimental results of the Paris dataset. The analysis of the results is explained in the corresponding sections. In Section 4.3, the limitations and applicability are analyzed. 4.1. Enschede 4.1.1. Test Site This dataset was collected in Enschede, a medium-sized city located in the east part of the Netherlands. It was acquired by TopScan GmbH in December 2008. The Optech LYNX mobile mapping system was used to capture the laser scanning data. This system consists of two rotating laser scanners, which were mounted on the backside of a moving vehicle. Their directions are perpendicular and make a 45◦ angle with the driving direction. The frequency of both scanners was 100 kHz. The platform was driven at a maximum speed of 50 km/h. In this dataset, the coordinates plus two attributes—the pulse count and reflectance strength—are available. The Enschede MLS data are about 1.25 km long, consisting of 19,945,716 points and 151 pieces of road furniture. The point density ranges from 35 points per square meter to 350 points per square meter. The test area was partitioned into 25 blocks along the trajectory line, with each block 50 m long and 40 m wide. 4.1.2. Results The road furniture detection of a single road part in the Enschede dataset is as shown in Figure 17. The height difference was firstly estimated for every point in this road part. The height difference threshold between all nearby points within a certain neighborhood in order to filter ground points was set to 0.15 m. The neighboring size for the height difference calculation was set to be high (for example, 100) due to the high point density of the road surfaces. Then points with a small height difference were categorized as ground points (Figure 17a) and removed. The filtered ground points were colored white. Subsequently, the above-ground components (Figure 17a) were produced by removing the ground points and the connected component analysis. The dynamic objects were labeled (Figure 17b) and eliminated from the above-ground components afterward. Most of them were cycling bicycles, pedestrians, and moving vehicles. Because of the angles and occlusions, several objects behind building façades were scanned by only one laser scanner. This leads to these objects behind building façades being labeled as dynamic objects. Although this was incorrect, it does not affect our results because we would like to eliminate these objects behind the façades anyway. Then the façades were recognized (Figure 17b). More detailed information on this process was explained in Section 3.1.2. At the end of the detection phase, the slice-based poles were analyzed. Vegetation, pole-like road furniture, and road furniture connected with vegetation were detected as in Figure 17b. Due to the close proximity, some vegetation connected with road furniture was also detected. Two trees connected with one streetlight were colored as brown in this figure. Their separation was as shown in Figure 17c. The detected pole-like road furniture were colored in purple. The detected pole-like road furniture were as shown in Figure 18a. False positively detected road furniture inside buildings were excluded after the occlusion analysis, shown in Figure 18b.

Remote Sens. 2018, 10, 531

18 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

18 of 28

(a)

(b)

(c) Figure 17. The pole-like road furniture detection from the Enschede dataset. (a) ground points (white Figure 17. The pole-like road furniture detection from the Enschede dataset. (a) ground points points) and above-ground connected components; (b) detected vegetation (light green), façade (light (white points) and above-ground connected components; (b) detected vegetation (light green), blue), dynamic objects (light yellow), detected pole-like road furniture (purple), and pole-like façade (light blue), dynamic objects (light yellow), detected pole-like road furniture (purple), furniture connected to vegetation (brown); and (c) the separation of pole-like road furniture and pole-like furniture connected to vegetation (brown); and (c) the separation of pole-like road connected to trees. furniture connected to trees.

Remote Sens. 2018, 10, 531

19 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW Remote Sens. 2018, 10, x FOR PEER REVIEW

19 of 28 19 of 28

(a) (a)

(b) (b) Figure 18.18. The toexclude excludeincorrectly incorrectly detected pole-like objects buildings. Figure Theocclusion occlusionanalysis analysis to detected pole-like objects insideinside buildings. (a) Figure 18. The occlusion analysis to exclude incorrectly detected pole-like objects inside buildings. (a) (a) The detected pole-likeroad roadfurniture furniturebefore beforeocclusion occlusion analysis; pole-like road furniture after detected pole-like analysis; (b)(b) pole-like road furniture after The detected pole-like road furniture before occlusion analysis; (b) pole-like road furniture after occlusion analysis. occlusion analysis. occlusion analysis.

At At thethe stage of of decomposition, waspaid paidtotocomplex complex structures rather stage decomposition,particular particular attention attention was structures rather thanthan At the stage of decomposition, particular attention was paidcomplex to complex structures rather road than poles. Therefore, we removed the bare and only structural pole-like barebare poles. Therefore, we removed the bare polespoles and only complex structural pole-like road furniture bare poles. Therefore, we removed the bare poles and only complex structural pole-like road furniture were thethe input for the decomposition test.dataset, In this dataset, 115 pole-like road were identified asidentified the inputasfor decomposition test. In this 115 pole-like road furniture furniture were identified as the input for the decomposition test. Inofthis dataset, 115road pole-like road furniture were detected, given in Figure 19a. The final results the pole-like furniture were detected, given in Figure 19a. The final results of the pole-like road furniture decomposition can furniture were detected, given in Figure 19a. The final results of the pole-like road furniture decomposition be found in Figure 19b. be found in Figurecan 19b. decomposition can be found in Figure 19b.

(a) (a) Figure 19. Cont.

Remote Sens. 2018, 10, 531

20 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

20 of 28

(b) Figure 19.19.The road furniture furnituredetection detection decomposition the Enschede Figure Thepole-like pole-like road andand decomposition from from the Enschede dataset.dataset. (a) byby the component number); (b) (b) thethe decomposition (a) The The identified identified pole-like pole-likeroad roadfurniture furniture(colored (colored the component number); decomposition result after applying optimizationand andrules. rules. result after applying optimization

4.1.3. Performance Analysisand andParametric Parametric Sensitivity Sensitivity Analysis 4.1.3. Performance Analysis Analysis For the evaluation of pole-like road furniture detection, we manually inspected every detected For the evaluation of pole-like road furniture detection, we manually inspected every detected pole-like road furniture. The completeness and correctness of pole-like road furniture detection are pole-like road furniture. The completeness and correctness of pole-like road furniture detection are as shown in Table 1. The correctness is 96.0% and the completeness is 94.7%. In total, 149 pole-like as shown in Table 1. The correctness is 96.0% and the completeness is 94.7%. In total, 149 pole-like road furniture items were detected, 6 of which were incorrectly recognized. Among these false road furniture items were there detected, of which were incorrectly recognized. Among these positive detected entities, were 6small garbage boxes and thin trees. Small garbage boxes werefalse positive detected entities, there were small garbage boxes and thin trees. Small garbage boxes were partially scanned and they were similar to short poles such as road piles. Thin trees without many partially scanned and were they also wereeasily similar to shortaspoles suchroad as road piles.This Thinwas trees without many branches and leaves recognized pole-like furniture. because it was branches leaves were also easily recognized as pole-like road This was because difficultand to distinguish them from pole-like road furniture only from the furniture. structure information. Future it wasresearch difficult to distinguish themthe from pole-like roadinfurniture onlywith from structure information. could be to investigate color information combination thethe point cloud. Two polelike road furniture items were not identified because of the noisyinpoints and low point Four Future research could be to investigate the color information combination with density. the point cloud. traffic boardsroad are identified pole-like furniture. Although their is obviously nonTwo pole-like furniture as items wereroad not identified because of thestructure noisy points and low point pole-like, they can still be categorized as road furniture because of their salient traffic functionalities. density. Four traffic boards are identified as pole-like road furniture. Although their structure is Nevertheless, we strictlythey consider the be detected traffic boards negative results. obviously non-pole-like, can still categorized as roadasfurniture because of their salient traffic The accuracy evaluation of the decomposition of the pole-like roadboards furniture from the Enschede functionalities. Nevertheless, we strictly consider the detected traffic as negative results. dataset is given in Table 2. The completeness is above 79.5% and the correctness is 92.3%. Compared The accuracy evaluation of the decomposition of the pole-like road furniture from the Enschede to our previous work in which pole extraction optimization and rules were not applied, the dataset is given in Table 2. The completeness is above 79.5% and the correctness is 92.3%. Compared to performance improved significantly in the point-wise evaluation. Due to the high point density of our previous work in which pole extraction optimization and rules were not applied, the performance this dataset, the components were already well-separated in the point level. The ratio of the correctly improved significantly in the point-wise Due which to the proves high point density of this decomposed components increases fromevaluation. 72.2% to 79.5%, the improvement of dataset, the thedecomposition components were already well-separated in the point level. The ratio of the correctly decomposed after the addition of pole extraction optimization and rules for the decomposition part. components increases 72.2% to 79.5%,we which proves the improvement of the22decomposition In order to setfrom proper parameters, chose one block which contained pieces of roadafter thefurniture addition to of train pole extraction optimization and for thephase, decomposition part.of the maximum the parameter settings. In therules detection the threshold In orderoftopole-like set proper parameters, we chose onea block which The contained pieces ofwas road furniture diameter slices was obtained by using grid search. optimal22threshold 0.50 m. Then the we used the trained parameter detect pole-like phase, of to train parameter settings. In thetodetection phase,road the furniture. thresholdInofthe thedecomposition maximum diameter we alsoslices use a was grid obtained search to obtain the optimal ratio of the of every slice for pole0.50 extraction. A we pole-like by using a grid search. Thewidth optimal threshold was m. Then set of parameters {0.25, 0.5, 0.75, 1.0, 1.25, 1.50} was tested on the sampling data. A value of 0.75 used the trained parameter to detect pole-like road furniture. In the decomposition phase, we also be thetooptimal useproved a grid to search obtainparameter. the optimal ratio of the width of every slice for pole extraction. A set of

parameters {0.25, 0.5, 0.75, 1.0, 1.25, 1.50} was tested on the sampling data. A value of 0.75 proved to be the optimal parameter.

Remote Sens. 2018, 10, 531

21 of 28

Table 1. The accuracy of the road furniture detection in the two test sites. Test Site

Enschede, NL

Visual interpretation Correctly detected (before/after the feedback) Total detected (before/after the feedback) Correctness (before/after the feedback) Completeness (before/after the feedback)

151 145/143 154/149 94.2%/96.0% 96.0%/94.7%

Table 2. The accuracy evaluation of the decomposition in the Enschede dataset Enschede, NL

Previous Framework

Current Framework

Point wise

Completeness Correctness

79.5% 92.3%

83.2% 92.3%

Component wise

Visual inspection Correctly decomposed Over-decomposed Under-decomposed Correctness

327 236 55 58 72.2%

327 260 34 54 79.5%

4.2. Paris 4.2.1. Test Site The test data of Paris (France) was captured in January 2013 by a Stereopolis II system, an MLS system developed by the French National Mapping Agency IGN. This is the IQmulus & TerraMobilita Contest dataset [39]. This dataset contains MLS data from a dense urban environment, composed of 300 million points, covering approximately 10 km. In addition, it is an accessible benchmark dataset. In this dataset, there are two available attributes: reflectance and pulse count information. The Paris MLS data was acquired in 10 zones. Compared with other zones, most road furniture was in zone 7. For this reason, it was selected as the test data zone. This test area is about 0.43 km long, consisting of 13,776,061 points and containing 132 pieces of road furniture. The test area was partitioned into 13 blocks along the trajectory line. We designed each block to be 40 m long and 30 m wide because the Paris streets are fairly narrow. The point density ranged from 72 points per square meter to 500 points per square meter. Compared with the Enschede dataset, the point distribution of the road furniture in the Paris dataset was quite uneven. This was because the scanning pattern was different. The Paris dataset was collected by only one laser scanner. 4.2.2. Results The original point cloud of one road part was as shown in Figure 20. The detected ground points, above-ground components, and façade points were as shown in Figure 20a. The detected pole-like road furniture before and after occlusion analysis are illustrated in Figure 20b,c. Some entities behind the façades were detected as vegetation. The reason for this was that these objects were collected with multiple echo count information when the laser pulses passed through the windows and hit objects behind the windows. As this dataset was captured by using only one laser scanner, dynamic objects were not detectable by our framework.

Remote Sens. 2018, 10, 531

22 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

22 of 28

(a)

(b) Figure 20. Cont.

Remote Sens. 2018, 10, 531

23 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

23 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

23 of 28

(c) (c)

Figure 20. The pole-like road furniture detection from the Paris dataset. (a) the ground points and Figure 20. The pole-like road furniture detection from the Paris dataset. (a) the ground points and above-ground components; (b)furniture detected façade (light blue) and furniture road (purple). The Figure 20. Thecomponents; pole-like road detection from the Paris dataset. (a)furniture the(purple). ground points and above-ground (b) thethe detected façade (light blue) and road The entities entities behind were incorrectly detected as furniture road furniture theroad left green frame; (c) the label above-ground components; (b) the detected façade (light blue) and furniture (purple). The behind façadesfaçades were incorrectly detected as road in thein left green frame; (c) the label of of incorrectly detected pole-like objects inside the buildings in the left green frame were redressed entities behind façades were incorrectly detected as road furniture in the left green frame; (c) the label incorrectly detected pole-like objects inside the buildings in the left green frame were redressed after after occlusion analysis.pole-like objects inside the buildings in the left green frame were redressed of incorrectly detected occlusion analysis. after occlusion analysis.

In this dataset, 41 pole-like road furniture items were identified as poles with attachments (see In this dataset, 41 pole-like road furniture items were identified as poles with attachments Figure and 97 were identified as furniture bare poles. The were detected poles with attachments were used(see to In21a), this dataset, 41 pole-like road items identified as poles with attachments (see Figure 21a), and 97 were identified as bare poles. The detected poles with attachments were quantify theand decomposition results. pole-like furniture decomposition Figure 21a), 97 were identified as The bare poles. Theroad detected poles with attachmentsresult were was used as to used to quantify the decomposition results. The pole-like road furniture decomposition result was as depicted in Figure 21b. quantify the decomposition results. The pole-like road furniture decomposition result was as depicted in Figure 21b. depicted in Figure 21b.

(a) (a) Figure 21. Cont.

Remote Sens. 2018, 10, 531

24 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

24 of 28

(b) Figure pole-likeroad roadfurniture furniture decomposition from the Paris dataset. The detected Figure21. 21. The The pole-like decomposition from the Paris dataset. (a) The(a) detected polepole-like road furniture (colored component number); decomposition result after applying like road furniture (colored by by component number); (b)(b) thethe decomposition result after applying optimizationand andrules. rules. optimization

4.2.3. Analysis 4.2.3.Performance PerformanceAnalysis Analysisand andParametric Parametric Sensitivity Sensitivity Analysis ByBymeans of pole-like pole-likeroad roadfurniture furnituredetection detectioninin the Paris meansofofvisual visualinspection, inspection, the the performance performance of the Paris dataset wasevaluated. evaluated.Table Table33shows shows that that the correctness was 95.4% dataset was correctness was was91.3% 91.3%and andthe thecompleteness completeness was 95.4% thisdataset. dataset.In Intotal, total, there there were were 138 138 pole-like pole-like road ininthis road furniture furnitureitems itemsdetected, detected,1212ofofwhich whichwere were incorrectlyrecognized. recognized.Among Among these these false positive positive detected incorrectly detectedentities, entities,most mostofofthem themwere werepedestrians pedestrians andentities entitieswhose whosepoint pointclouds clouds were were partially partially acquired. acquired. Pedestrians road and Pedestriansare aresimilar similartotopole-like pole-like road furniture, especially in a noisy dataset. Another factor is that some road objects like fences were only furniture, especially in a noisy dataset. Another factor is that some road objects like fences were only partiallyscanned scannedand andare aresimilar similar to to pole-like pole-like road road furniture furniture in partially in appearance. appearance.ItItisisstill stilla achallenging challenging tasktotodistinguish distinguishthem. them. Six Six pole-like pole-like road road furniture ofof the low task furniture items itemswere werenot notidentified identifiedbecause because the low point density. Similar to the Enschede dataset, we also selected the optimal parameters by using a point density. Similar to the Enschede dataset, we also selected the optimal parameters by using a grid grid search in the Paris dataset. The optimal threshold of the maximum diameter of the pole-like search in the Paris dataset. The optimal threshold of the maximum diameter of the pole-like slices was slices was determined to be 0.30 m. The optimal ratio of the width of every slice for pole extraction determined to be 0.30 m. The optimal ratio of the width of every slice for pole extraction was 0.50. was 0.50. The performance of our previous work and the current framework is illustrated in Table 4. The performance of our previous work and the current framework is illustrated in Table 4. Compared with the Enschede dataset, the quality of the Paris dataset is lower. For example, in the Compared with the Enschede dataset, the quality of the Paris dataset is lower. For example, in the Paris dataset, the point distribution was uneven because of the usage of a single scanner. Because the Paris dataset, the point distribution was uneven because of the usage of a single scanner. Because the decomposition method required a high-quality point cloud, the performance of our previous work decomposition method required a high-quality point cloud, the performance of our previous work tested on the Paris dataset was not satisfying. As shown in Table 4, the results of the decomposition tested on the Paris dataset was not satisfying. As shown in Table 4, the results of the decomposition were obtained we did did not not utilize utilizepole poleextraction extractionoptimization optimization were obtainedby byour ourprevious previouswork work [29] [29] in in which which we and additionofofpole poleextraction extraction optimization rules indecomposition the decomposition part, andrules. rules.With With the the addition optimization andand rules in the part, the the completeness was improved to be above 80%, the correctness was improved to be 92.6%, and completeness was improved to be above 80%, the correctness was improved to be 92.6%, and the ratethe rate of correctly decomposed components enhanced to 72.0%. results prove the effectiveness of correctly decomposed components waswas enhanced to 72.0%. The The results prove the effectiveness of ofthe theproposed proposedframework. framework. Table 3. The accuracy evaluation of the decomposition in the Paris dataset. Table 3. The accuracy evaluation of the decomposition in the Paris dataset. Test Site Test Site

Paris, FR Paris, FR Visual interpretation 132 Visual interpretation 132 Correctly thethe feedback) 126/126 Correctlydetected detected(before/after (before/after feedback) 126/126 Total the feedback) 142/138 Totaldetected detected(before/after (before/after the feedback) 142/138 Correctness (before/after the feedback) 88.7%/91.3% Correctness (before/after the feedback) 88.7%/91.3% Completeness (before/after the feedback) 95.4%/95.4% Completeness (before/after the feedback) 95.4%/95.4%

Remote Sens. 2018, 10, 531

25 of 28

Table 4. The accuracy evaluation of the decomposition in the Paris dataset. Paris, FR Completeness Correctness

RemotePoint Sens. 2018, wise10, x FOR PEER REVIEW

Previous Framework

Current Framework

69.0% 90.0%

80.0% 92.6%

25 of 28

Table 4. TheVisual accuracy evaluation of the decomposition inspection 125 in the Paris dataset.

125 Correctly decomposed 67 90 Paris, FR Previous Framework Current Framework Component wise Over-decomposed 24 12 Completeness 69.0% 80.0% 29 Under-decomposed 45 Point wise Correctness 90.0% 92.6% Correctness 53.6% 72.0% Visual inspection 125 125 Correctly decomposed 67 90 4.3. Discussion Component wise Over-decomposed 24 12 Under-decomposed 29 Our carried out research aimed to decompose pole-like45road furniture into poles and attachments Correctness 53.6% 72.0%

based on their spatial relations such as connectivity. In this study, we developed a new framework to detect decompose pole-like road furniture automatically. As our results show, pole-like road 4.3.and Discussion furniture were detected accurately and automatically. Approximate 90% of pole-like road furniture Our carried out research aimed to decompose pole-like road furniture into poles and attachments items were detected and more than 90% was correctly detected in both test sites. The correctness of based on their spatial relations such as connectivity. In this study, we developed a new framework road furniture was 92.3% and completeness was 83.3%Asinour Cabo et al. [17].pole-like In Yang et al. [32], to detect detection and decompose pole-like road furniture automatically. results show, road their achieved correctness completeness of roadApproximate furniture detection was road higher than 90.0%. furniture were detected and accurately and automatically. 90% of pole-like furniture items were detected and more thanframework 90% was correctly detected in both sites. The correctness is of 93.6%. The average completeness of our is 95.0% and the test average correctness road furniture detection waswith 92.3%the andcurrent completeness 83.3% et al. [17]. In Yang etroad al. [32], Our framework is competitive state was of the art in inCabo the field of pole-like furniture theirThe achieved correctnessof and completeness ofwas roadabove furniture detection higher thanof90.0%. The detection. completeness decomposition 80% and thewas correctness decomposition average completeness of our framework is 95.0% and the average correctness is 93.6%. Our was above 90% in both experimental areas. Besides that, we introduced an automatic method framework is competitive with the current state of the art in the field of pole-like road furniture to evaluate theThe accuracy of the decomposition ofand thethevisual inspection mentioned detection. completeness of decomposition wasinstead above 80% correctness of decomposition was in our previous work. above 90% in both experimental areas. Besides that, we introduced an automatic method to evaluate accuracy of the decomposition instead of the is visual in our previous All work. A the significant advantage of this framework the inspection adoptionmentioned of occlusion analysis. pole-like A significant advantage of this framework is the adoption of occlusion analysis. All pole-like objects inside buildings were eliminated in the Enschede test case, and only four pole-like objects objects inside buildings were eliminated in Paris the Enschede testThese case, and four were pole-like objectsclose to inside buildings were not filtered out in the test site. fouronly objects located inside buildings were not filtered out in the Paris test site. These four objects were located close to building façades whose normal direction was almost parallel to the trajectory direction. Our occlusion building façades whose normal direction was almost parallel to the trajectory direction. Our analysisocclusion was notanalysis able to was copenot with such situations. Thesituations. pole-extraction optimization, defined merging, able to cope with such The pole-extraction optimization, and splitting rules also improved the performance of the decomposition. It enhances theItratio of defined merging, and splitting rules also improved the performance of the decomposition. enhances the ratio components of correctly decomposed components by 7.3% insite the Enschede testin site and 18.4% correctly decomposed by 7.3% in the Enschede test and 18.4% the Paris test area. in the Paris test area. Some well-decomposed road furniture is shown in Figure 22. Some well-decomposed road furniture is shown in Figure 22.

Figure Thecorrect correct decomposition results. Figure 22.22.The decomposition results.

Even though the accuracy of pole-like road furnituredetection detection is is already problems Even though the accuracy of pole-like road furniture alreadyhigh, high,some some problems still still remain. In both of the two test sites, small booths supported by pillars were recognize pole- road remain. In both of the two test sites, small booths supported by pillars were recognize as as pole-like like road furniture. These booths have pillars whose appearance is very similar to poles. The detection furniture. These booths have pillars whose appearance is very similar to poles. The detection algorithm algorithm was not good enough to discriminate the difference. Therefore, additional structural was notrelations good enough to discriminate the difference. Therefore, additional structural relations will will be added later to address this problem. In the slicing-based pole-like road furniture be added later to address this problem. In theroad slicing-based pole-like furniture detection detection step, trees connected to the pole-like furniture were detected.road We have yet been unable step, trees connected to the pole-like road furniture were detected. We have yet been unable to thoroughly to thoroughly separate them (Figure 17d). separate them (Figure 17d).

Remote Sens. 2018, 10, 531

26 of 28

Remote Sens. 2018, 10, x FOR PEER REVIEW

26 of 28

The remaining problems with decomposition are divided into two types. The first one is the remaining problems with decomposition are divided into twodistribution, types. The first is the problemThe with the data itself. Specifically, it involves point density, point andone noisy points. problem with the data itself. Specifically, it involves point density, point distribution, and noisy In Figure 23a, the two connected traffic signs should have been separated. If there are many noisy points. In Figure 23a, the two connected signs should beenthese separated. If there are many by points around one road furniture item, ittraffic is even harder to have separate attached components noisy points aroundUnder one road furniture item, it is even harder to separate these attached components visual interpretation. such circumstances, our decomposition algorithm does not work. If the by visual interpretation. Under such circumstances, our decomposition algorithm does not work.in If the point density is extremely low, points cannot form an individual component. For example, the point density is extremely low, points cannot form an individual component. For example, in the Paris dataset, a few traffic lights were documented by several points, which led to the fragmented Paris dataset, a few traffic lights were documented by several points, which led to the fragmented unstructured components during the decomposition (Figure 23b). In contrast, most of the traffic unstructured components during the decomposition (Figure 23b). In contrast, most of the traffic signs signs were decomposed as individual components owing to their adequate point density. Therefore, were decomposed as individual components owing to their adequate point density. Therefore, the the point pointdensity densitycannot cannot very low. The point spacing between points intrajectory the trajectory direction bebe very low. The point spacing between points in the direction is is normally higher than 0.05 m. The point spacing was about 0.02 m within the scanlines. The point normally higher than 0.05 m. The point spacing was about 0.02 m within the scanlines. The point spacing in these two directions was different. The second problem originates from the scene. spacing in these two directions was different. The second problem originates from the Currently, scene. curved poles cannot be extracted as the assumption is that pole-like road furniture contains straight Currently, curved poles cannot be extracted as the assumption is that pole-like road furniture poles. Beside this, pedestrian-oriented traffic lights (Figure 23c) which consist trafficconsist light heads contains straight poles. Beside this, pedestrian-oriented traffic lights (Figure 23c)ofwhich of traffic light withbe each other cannot be decomposed (Figure 23d). connected withheads eachconnected other cannot decomposed correctly (Figure correctly 23d).

(a)

(b)

(c)

(d)

Figure 23. 23. Incorrectly decomposed (a)two twotraffic trafficsigns signs were separated; Figure Incorrectly decomposedroad road furniture. furniture. (a) were notnot separated; (b) (b) oneone traffic light was decomposedtototwo twofragmented fragmented components; components; (c) and (d)(d) thethe point cloud of of traffic light was decomposed (c)the theimage image and point cloud traffic lights which werenot notseparated. separated. twotwo traffic lights which were

5. Conclusions 5. Conclusions In this paper, we proposed a new framework to detect pole-like road furniture and decompose In this paper, we proposed a new framework to detect pole-like road furniture and decompose them into different components based on their logical relations. This innovative framework is tested them into different components based on their logical relations. This innovative framework is tested in two test sites. After being processed by our new framework, the road furniture were detected and in two test sites. After being processed by our newfor framework, the road furniture detected interpreted by logical relations, which can be used precise semantics labeling. Thiswere proposed andframework interpreted by logical relations, which can be used for precise semantics labeling. This proposed can be potentially used for high defined 3D mapping. In this framework, we improved framework can be potentially used for dynamic high defined 3D mapping. this framework, weanalysis. improved road furniture detection by combining objects removal, poleIn slicing, and occlusion road furniture detection combining dynamic pole slicing, andwere occlusion The completeness and by correctness values of the objects pole-likeremoval, road furniture detection higher analysis. than The90% completeness and correctness values of the pole-like road furniture detection were higher in both the Enschede dataset and the Paris dataset. The main contribution was the decompositionthan 90%ofinthe both thefurniture Enschede dataset and the Compared Paris dataset. contribution the decomposition road and its evaluation. withThe ourmain previous work, thewas current framework is completely automatic the performance of decomposition has been improved by applying of the road furniture and itsand evaluation. Compared with our previous work, the current framework defined rules. is completely automatic and the performance of decomposition has been improved by applying next stage of our work will focus on the classification of decomposed road furniture. Even defined The rules. though logically separated were obtained, the meaning of of every component has notfurniture. been The next stage of our components work will focus on the classification decomposed road assigned yet. Therefore, in order to make use of the components for further applications such as Even though logically separated components were obtained, the meaning of every component has not mapping, these components will be semantically labeled based on their features.

been assigned yet. Therefore, in order to make use of the components for further applications such as mapping, these components will be semantically labeled based on their features.

Remote Sens. 2018, 10, 531

27 of 28

In our research, road furniture items have been decomposed into components by using mere geometric features. Color information has also been beneficial to the detection and decomposition. Many techniques on image semantics labeling such as the convolutional neural network can be applied to our research. 2D image data were also captured by two cameras mounted on a moving vehicle. The clear detection, decomposition, and classification of road furniture could benefit from the color information given by the 3D point cloud. Acknowledgments: The research is supported by Chinese Scholarship Council. The authors want to thank Marco Oesting for the English editing. Author Contributions: Fashuai Li designed the algorithms as described in this paper, and was responsible for the main organization and writing of the paper. All the work was done under the supervision of Sander Oude Elberink and George Vosselman. Conflicts of Interest: The authors declare no conflicts of interest.

References 1. 2. 3.

4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.

World Health Organization. In Global Status Report on Road Safety 2015; World Health Organization: Geneva, Switzerland, 2015. Adesiyun, A.; Avenoso, A.; Dionelis, K.; Cela, L.; Nicodème, C.; Goger, T.; Polidori, C. Effective and coordinated road infrastructures safety operations. Transp. Res. Proced. 2016, 14, 3304–3311. [CrossRef] Bauer, R.; Steiner, M.; Kühnelt-Leddhin, A.; Lyons, R.; Turner, S.; Walters, W.; Larsen, B.; Valkenberg, H.; Bejko, D.; Ellsaesser, G.; et al. Scope and patterns of under-reporting of vulnerable road users in official road accident statistics Monica Steiner. Eur. J. Public Health 2017, 27 (Suppl. 3). [CrossRef] U.S. Department of Transportation Federal Highway Administration. Available online: http://safety.fhwa. dot.gov/tools/data_tools/mirereport/ (accessed on 28 January 2018). Soilán, M.; Riveiro, B.; Sánchez-Rodríguez, A.; Arias, P. Safety assessment on pedestrian crossing environments using MLS data. Accid. Anal. Prev. 2018, 111, 328–337. [CrossRef] [PubMed] Ibañez-Guzman, J.; Laugier, C.; Yoder, J.D.; Thrun, S. Autonomous driving: Context and state-of-the-art. In Handbook of Intelligent Vehicles; Eskandarian, A., Ed.; Springer: London, UK, 2012; pp. 1271–1310. Jaakkola, A.; Hyyppä, J.; Hyyppä, H.; Kukko, A. Retrieval algorithms for road surface modelling using laser-based mobile mapping. Sensors 2008, 8, 5238–5249. [CrossRef] [PubMed] Oude Elberink, S.; Vosselman, G. 3D information extraction from laser point clouds covering complex road junctions. Photogramm. Rec. 2009, 24, 23–36. [CrossRef] Zhang, W. Lidar-based road and road-edge detection. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, 21–24 June 2010. Zhou, L.; Vosselman, G. Mapping curbstones in airborne and mobile laser scanning data. Int. J. Appl. Earth Obs. Geoinf. 2012, 18, 293–304. [CrossRef] Oude Elberink, S.; Khoshelhama, K.; Arastouniab, M.; Benitoc, D.D. Rail track detection and modelling in mobile laser scanner data. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2013, 2, 223–228. [CrossRef] Yang, B.; Fang, L. Automated extraction of 3-D railway tracks from mobile laser scanning point clouds. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2014, 7, 4750–4761. [CrossRef] Lehtomäki, M.; Jaakkola, A.; Hyyppä, J.; Kukko, A.; Kaartinen, H. Detection of Vertical Pole-Like Objects in a Road Environment Using Vehicle-Based Laser Scanning Data. Remote Sens. 2010, 2, 641–664. [CrossRef] Pu, S.; Rutzinger, M.; Vosselman, G.; Oude Elberink, S. Recognizing basic structures from mobile laser scanning data for road inventory studies. ISPRS J. Photogramm. Remote Sens. 2011, 66, S28–S39. [CrossRef] El-Halawanya, S.I.; Lichti, D.D. Detecting road poles from mobile terrestrial laser scanning data. GISci. Remote Sens. 2013, 50, 704–722. Li, D.; Oude Elberink, S. Optimizing detection of road furniture (pole-like objects) in mobile laser scanner data. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2013, 1, 163–168. [CrossRef] Cabo, C.; Ordoñez, C.; García-Cortés, S.; Martínez, J. An algorithm for automatic detection of pole-like street furniture objects from mobile laser scanner point clouds. ISPRS J. Photogramm. Remote Sens. 2014, 87, 47–56. [CrossRef]

Remote Sens. 2018, 10, 531

18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.

31.

32. 33. 34.

35. 36.

37. 38. 39.

28 of 28

Huang, J.; You, S. Pole-like object detection and classification from urban point clouds. In Proceedings of the IEEE International Conference on Robotics and Automation, Seattle, WA, USA, 26–30 May 2015. Fukano, K.; Masuda, H. Detection and classification of pole-like objects from mobile mapping data. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2015, 2, 57–64. [CrossRef] Yang, B.; Dong, Z.; Zhao, G.; Dai, W. Hierarchical extraction of urban objects from mobile laser scanning data. ISPRS J. Photogramm. Remote Sens. 2015, 99, 45–57. [CrossRef] Yu, Y.; Li, J.; Guan, H.; Wang, C.; Yu, J. Semiautomated extraction of street light poles from mobile lidar point-clouds. IEEE Trans. Geosci. Remote Sens. 2015, 53, 1374–1386. [CrossRef] Puttonen, E.; Jaakkola, A.; Litkey, P.; Hyyppä, J. Tree classification with fused mobile laser scanning and hyperspectral data. Sensors 2011, 11, 5158–5182. [CrossRef] [PubMed] Rutzinger, M.; Höfle, B.; Oude Elbernik, S.; Vosselman, G. Feasibility of facade extraction from mobile laser scanning data. Photogramm. Fernerkund. Geoinf. 2011, 3, 97–107. [CrossRef] Demantké, J.; Vallet, B.; Paparoditis, N. Streamed vertical rectangle detection in terrestrial laser scans for facade database production. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2012, 1, 99–104. [CrossRef] Yang, B.; Dong, Z. A shape-based segmentation method for mobile laser scanning point clouds. ISPRS J. Photogramm. Remote Sens. 2013, 81, 19–30. [CrossRef] Niemeyer, J.; Rottensteiner, F.; Soergel, U. Contextual classification of lidar data and building object detection in urban areas. ISPRS J. Photogramm. Remote Sens. 2014, 87, 152–165. [CrossRef] Riegl. 2017. Available online: http://www.riegl.com/uploads/tx_pxpriegldownloads/RIEGL_VMX-1HA_ at-a-glance_2017-05-04.pdf (accessed on 28 January 2018). Nurunnabi, A.; Belton, D.; West, G. Robust segmentation for large volumes of laser scanning three-dimensional point cloud data. IEEE Trans. Geosci. Remote Sens. 2016, 54, 4790–4805. [CrossRef] Li, F.; Oude Elberink, S.; Vosselman, G. Pole-like street furniture decomposition in mobile laser scanning data. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, 3, 193–200. [CrossRef] Golovinskiy, A.; Kim, V.G.; Funkhouser, T. Shape-based recognition of 3D point clouds in urban environments. In Proceedings of the 12th International Conference on Computer Vision (ICCV09), Kyoto, Japan, 27 September–4 October 2009. Munoz, D.; Vandapel, N.; Hebert, M. Directional associative markov network for 3D point cloud classification. In Proceedings of the Fourth International Symposium on 3D Data Processing, Visualization and Transmission, Atlanta, GA, USA, 18 June 2008. Yang, B.; Liu, Y.; Liang, F.; Dong, Z. Using mobile laser scanning data for features extraction of high accuracy driving maps. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, 41, 433–439. [CrossRef] Huang, J.; You, S. Point cloud labeling using 3d convolutional neural network. In Proceedings of the International Conference on Pattern Recognition, Cancun, Mexico, 4–8 December 2016. Soilán, M.; Riveiro, B.; Martínez-Sánchez, J.; Arias, P. Traffic sign detection in MLS acquired point clouds for geometric and image-based semantic inventory. ISPRS J. Photogramm. Remote Sens. 2016, 114, 92–101. [CrossRef] Hackel, T.; Wegner, J.D.; Schindler, K. Fast semantic segmentation of 3d point clouds with strongly varying density. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, 3, 177–184. [CrossRef] Yu, Y.; Li, J.; Wen, C.; Guan, H.; Luo, H.; Wang, C. Bag-of-visual-phrases and hierarchical deep models for traffic sign detection and recognition in mobile laser scanning data. ISPRS J. Photogramm. Remote Sens. 2016, 113, 106–123. [CrossRef] Oude Elberink, S.; Kemboi, B. User-assisted object detection by segment based similarity measures in mobile laser scanner data. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2014, 40, 239–246. [CrossRef] Vosselman, G.; Maas, H.G. (Eds.) Airborne and Terrestrial Laser Scanning; Whittles Publishing: Caithness, UK, 2010. Vallet, B.; Brédif, M.; Serna, A.; Marcotegui, B.; Paparoditis, N. TerraMobilita/iQmulus urban point cloud analysis benchmark. Comput. Graph. 2015, 49, 126–133. [CrossRef] © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).