A Data Quality-driven Framework for Asset Condition ...

A Data Quality-driven Framework for Asset Condition Assessment Using LiDAR and Image Data 1 Pedram Oskouie , Burcin Becerik-Gerber2, Lucio Soibelman3 1

PhD Student, Sonny Astani Department of Civil and Environmental Engineering, University of Southern California, Los Angeles, CA; PH (213) 572-9373; Fax (213) 744-1426; email: [email protected] 2 Assistant Professor, Sonny Astani Department of Civil and Environmental Engineering, University of Southern California, Los Angeles, CA; PH (213) 740-4383; Fax (213) 744-1426; email: [email protected] 3 Professor and Chair, Sonny Astani Department of Civil and Environmental Engineering, University of Southern California, Los Angeles, CA; PH (213) 740-0609; Fax (213) 744-1426; email: [email protected]

ABSTRACT Laser scanners provide high-precision geometrical information, however, due to various scan errors, the generated point clouds often do not meet the data quality criteria set by project stakeholders for accurate data processing. Although, there exists several studies on identifying scan errors in literature, there is limited research on defining a data quality-driven scan plan for accurate detection of geometrical features. The authors propose a novel framework to integrate image-processing methods with point cloud processing techniques for defining a data quality-driven approach for scan planning. The framework includes the following steps: 1) capturing images of a target using a commercially available unmanned aerial vehicle (UAV) and generating a 3-D point cloud, 2) recognizing the project’s geometrical information using the point cloud, 3) extracting the features of interest (FOI) using the point cloud, 4) generating multiple scanning scenarios based on the data processing requirements and the extracted features, 5) identifying the best scan plan through iterative simulation of different scanning scenarios, and 6) validating the scan plan using real life data. The framework was evaluated using preliminary results of a case study. The results showed that the data quality requirements of the case study were met using the proposed framework. INTRODUCTION AND BACKGROUND Visual remote sensing technologies, such as Light Detection and Ranging (LiDAR) systems and digital cameras, are prevalently used for generating as-built/asis 3-D models. Reconstructing accurate 3-D models by using such technologies have been extensively studied in recent years (Anil et al. 2011; Balali and Golparvar-Fard 2014; Balali and Golparvar-Fard 2015; Tang et al. 2010). Performances of 3-D laser scanners and digital cameras in creating realistic 3-D models have been also compared in many studies (Dai et al. 2012; Zhu and Brilakis 2009). According to Dai et al. 2012, the performance of laser scanners is more consistent and their accuracy is 10 times higher than image and video-based methods. Researchers have identified factors that influence scan-data accuracy. For example, Shen et al. (2013) found that scan resolution, scanner’s distance, color and intensity of the scanned objects are the factors that contribute most to scan errors. In order to have an accurate 3-D point cloud and to 1

minimize the errors for different purposes, such as Scan to BIM, it is essential to have an accurate scan plan, which takes all parameters that could affect a scan’s accuracy into account. Most of the scan planning studies focus on analyzing the visibility requirements and do not consider the data quality requirements (e.g. data density and tolerance). The use of a uniform scan plan can result in redundant level of detail (LOD) in parts of a point cloud and lack of the required details in other parts. A scan plan should also consider the project specific LOD requirements, which may vary in a project. The LOD requirements of a geometrical feature of a project could be initially defined based on the asset condition assessment (ACA) goals. For instance, an ACA goal for a bridge could be the inspection of concrete columns for crack detection. In order to monitor the columns using a laser scanner, point clouds should provide detailed data points on the columns so the data processor could analyze the severity of cracks or damages. If the rest of the bridge is scanned with settings similar to the columns, a scan plan based solely on the visibility requirements could potentially result in a time consuming and costly scan process. Recently, researchers have developed new scan planning approaches focusing on scan data quality. Pradhan and Moon (2013) introduced a simulation-based framework to identify scan locations for better capturing the critical components of a bridge such as piers and girders. Based on their findings, capturing geometrical measurements of certain parts of a structure is more critical than the others since different portions of structures have different levels of importance in terms of performance/response metrics. Song et al. (2014) proposed a novel approach for scan planning, which integrated 3-D data quality analysis and clustering methods to group the geometrical features based on their LOD requirements. Their study showed that the automatic scan planning algorithm results in a denser point cloud without the need to increase the data collection time. Also, their study is based on the assumption that a BIM model of the project is available, however, this is not always the case, especially when the as-is condition of an infrastructure is different than the archived asbuilt/designed models. Moreover, the line-of-sight and portability limitations of terrestrial laser scanners have to be considered in the scan plan. For instance, a scan plan has to provide a solution for capturing the features that are located in the blind spots of a laser scanner. Therefore, a hybrid scan plan including different LiDAR equipment such as long range scanners and aerial imaging sensors may be required to scan infrastructure systems with accessibility limitations. FRAMEWORK In order to define a high quality scan plan, there is a need for a holistic approach that takes into account project specific characteristics and data quality requirements. The framework proposed in this paper was designed to enable project stakeholders to define an integrated scan plan, which is centered on data quality requirements. (Figure 1). The input data of the proposed framework is comprised of every project’s specific information. The main information that drives the scan plan is realization of the condition assessment goals. The LOD requirements for different geometrical features of the project are defined by project stakeholders based on these goals. LOD requirements are usually not constant and may change throughout the project. The next input data to be derived is the list of all available LiDAR sensors and their parameters. 2

This information is directly tied to the project’s constraints, such as time and budget, weather conditions, accessibility, etc. For instance, in order to overcome the project’s accessibility constraints, an alternative solution would be using long range terrestrial laser scanners, aerial LiDAR sensors, or UAV imaging. The last input data for the framework is the 3-D model of the project (if available). In this paper, the authors propose using an image-based 3-D model for planning the scan when the updated BIM model is not available.

Figure 1. An overview of data quality-driven scan planning framework Image-based 3-D models can provide an acceptable representation of a project’s layout. However, compared to 3-D laser scanners, they might not offer an accurate data for detailed analysis required for an ACA. Nonetheless, comparing to other remote sensing technologies and with respect to their low cost, UAVs have less mobility constraints and therefore, they can complement the scanning of a project by covering the laser scanners’ blind spots. It is noteworthy to mention that using UAV for image collection has its own challenges. For instance, UAV’s flight trajectory and elevation have to be determined prior to flight as they can affect the quality of collected images. Once the image collection process is completed, a 3-D model is reconstructed using the structure from motion (SfM) technique. In order for the 3-D model to be used for scan planning, it has to be augmented with semantics about FOI’s geometrical information in the project. Hence, two principal information is extracted from the 3-D model: 1) the project’s layout that includes the boundary coordinates which is then integrated with the project’s accessibility constraints information, 2) FOI (points, planes, or 3-D objects in general) geometrical and RGB data. The project’s layout information assists the scanning crew to identify the feasible areas for scanner circulation and establishing scan positions. Using the extracted information, the features are classified based on their locations, orientations in space, and the LOD requirements. The classification criteria are selected based on their direct relationships with scanner’s position and data quality. The interactions of features classification criteria and sensor parameters (range, accuracy, and angular resolution) pinpoint the best scan position. In the next step, the features are prioritized based on their classes and a simulation-based decision making process determines the best scan position across different sides of the project. The output of the decision making process is a 3

scan scenario, which is then evaluated by a sensor simulation software (i.e., Blensor). The sensor simulation output enables evaluation of the proposed scan scenario by measuring the data quality for different features. If the data quality is not satisfying, the decision making process is repeated with the new information using the sensor simulation output. The other module of the proposed framework focuses on integrating LiDAR and UAV-based imaging data to generate a single coherent 3-D point cloud, in which all the blind spots of the target are covered. Due to the fact that point clouds from different sensors might have non-equal spatial resolution, precision, and accuracy, their registration process is challenging. In the case of having point clouds from multiple sensors, an iterative registration process using multiple common features/control points could improve the accuracy of final point cloud. Once the registration process is completed, if there are missing data in the final point cloud, there will be a need to augment the 3-D model by extrapolating the missing data using their surrounding points’ coordinates and RGB values. CASE STUDY In order to provide a preliminary evaluation of the proposed framework, the authors selected the Mudd Hall building’s courtyard located on University of Southern California campus. The Mudd Hall building has been named as one of the Historical Cultural Monuments by the Los Angeles City Council and the fact that it is a feature rich building makes it a suitable case study for the purpose of this research. Figure 2b shows some of the architectural features on the building’s courtyard. UAV Image Collection. 3-D scene reconstruction using image sequences has been a growing field of inquiry across multiple disciplines such as cultural heritage preservation (Stanco et al. 2011), archaeology (Kersten and Lindstaedt 2012), and construction (Golparvar-Fard et al. 2009; Ham and Golparvar-Fard 2013; RodriguezGonzalvez et al. 2014). Generating 3-D models from 2-D images follow the SfM technique which includes: 1) feature extraction and description using Scale-invariant Feature Transform (SIFT) algorithm, 2) pairwise matching of images using SIFT descriptors, 3) estimation of motion and structure using the matched images, 4) refining the estimates using Bundle Adjustment, and 5) creating surface meshes using imagebased triangulation. The authors used a DJI Phantom 2 Vision Plus drone to take images from the case study building. The drone comes with a 14 Megapixel built-in camera, which is mounted on a gimbal, making it stable during its flight. The camera has a large field of view (FOV = 125°), therefore the images are highly distorted. We selected the lowest available FOV option (85°), even then the collected images were slightly distorted. We then calibrated the camera to circumvent the distortion effect and to rectify the images. We installed 10 ground control points (GCP) and scanned them with a Leica TS06plus Total Station to be able to geo-reference the 3-D reconstructed model (Figure 2a). A total of 236 images were taken from the building’s courtyard using a drone. Image-based 3-D Reconstruction. We used commercially available and open source tools, such as VisualSfM (Wu 2011), Autodesk Recap 360, and Agisoft Photoscan Pro to generate a dense point cloud and a 3-D mesh of the courtyard. After visual inspection of the reconstructed models, we decided to use Agisoft’s output as it provided a denser point cloud comparing to VisualSfM and since Autodesk Recap only provides 3-D mesh. Note that the selection of the software tools might have effects on 4

the results; however, optimum selection of the tools will be part of the future directions of this research. We then geo-referenced the model by assigning surveyed coordinates to GCP on the point cloud. The 3-D reconstruction process was completed in 155 mins using a Microsoft© Windows workstation laptop with Intel® Core i7 processor, 32 GB RAM memory, and NVIDIA© K2000M graphic card.

a. Large Windows (FOI) b. Architectural features c. 3D reconstructed model Figure 2. Case Study-Mudd Hall Courtyard DATA PROCESSING AND RESULTS Geometrical Features Recognition. Following the steps in our proposed framework, we used the images and corresponding point cloud to detect, extract, and study the FOIs. For the purpose of this study, the west bound of the courtyard was examined, where four large windows were selected as the FOIs (Figure 2). The authors used Haar-like features to identify the FOIs from the images. Previous research has shown that improved Haar-like features yields less rate of False Positives (FP) and False Negatives (FN), as oppose to other prevalent image-based object detection methods, such as HOG and etc. (Zhang et al. 2013). Haar-like features were introduced by Viola and Jones (2001) who proposed convolving images with Haar wavelet sequences. Haar-like features are found by calculating the difference between sums of pixel intensities in surrounding rectangles of a location in an image. The Haar-like classifier slides a window over an image and looks for objects of interest that were previously used for training the classifier. Considering the fact that using sums of pixel intensities as the only feature would result in a weak learner, a cascade classifier, which includes a large number of Haar-like features is used in the Viola and Jones framework. We ran pilot tests to determine the optimum input parameters for training Haarlike cascade classifier. The parameters were determined based on the number of pictures and the quality of them: number of training stages = 4, sliding window size = 128*128, false positive rate = 0.2, and true positive rate = 0.99. We trained the cascade classifier using different numbers of positive images (images that only include objects of interest with different poses (position/orientation)) to evaluate the performance of the Haar-like object detector. Note that we selected a low FP rate due to the low number of positive images. Table 1a shows the results for different number of training sizes. 𝑇𝑃 𝑇𝑃 𝑇𝑃+𝑇𝑁 The precision, recall, and accuracy are computed based on 𝑇𝑃+𝐹𝑃 , 𝑇𝑃+𝐹𝑁 , 𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁 , respectively. Note that the maximum desired number of true positives in each image was four (there are only four large windows in the courtyard), which is a low number and therefore has resulted in large jumps in precision and recall values. Once the FOIs were detected in the images, the corresponding points should be localized in the point cloud. A reverse engineering method using the SfM principles could be employed to 5

address the localization problem. Figure 3 illustrates different steps to match 2-D pixels on the images to 3-D points on the point cloud. The images were previously pair-wised by matching similar SIFT features during the SfM process and a tree data structure was made for the pair-wised images. Therefore, a tree-based search can identify co-visible images that contain a particular FOI. Using the previously estimated camera poses of the co-visible images, fiducial planes (G1 and G2) of the two viewpoints along with the rotation matrix (R1,2) are computed and used to localize the detected object in the 3-D model. For this preliminary study, we manually localized the FOI in the point cloud to ensure the accuracy of the feature classification and the following steps. As part of the future work, we will examine the proposed automated localization approach. Table 1. Results and LOD definition a) Haar-Like Results b) Scan Results c) LOD Definition

Geometrical Features Classification. In this step, the features (windows including their trimming) were classified based on their location, orientation, and LOD requirements. The process of identifying FOI’s orientation begins with estimating the feature’s surface normals. Two methods could be used for this purpose. The first one is generating a mesh from the point cloud and estimating the triangular planes’ normal. The second method, which is more accurate due to studying points rather than approximated planes, is estimating the normal to a point by solving a least-square plane fitting problem. The overall steps for the latter method are: getting the k nearest neighbors of the point P on the point cloud, computing the surface normal, and finally determining the sign of the normal. Determining the sign of a surface normal in point clouds that are created from multiple views is a complicated problem, however, it can be simplified by determining the correct normal sign based on the neighbor surfaces’ normals. Due to space limits of this paper, the detailed mathematics of the problem will be discussed in our future work. The surface normals of the points on the FOIs were computed using the second method and the dominating normal was used to represent the orientation of the feature with respect to the point cloud.

Figure 3. Steps for Localization of Features of Interest in 3-D Point Cloud 6

Laser Scanner Position Determination. We propose defining the new notion of Pose-Quality (PQ) vector to represent 3-D objects. In order to decrease the computational complexity to determine the best scan position, we simplify the studying of 3-D objects by representing them as a PQ vector. The PQ vector’s size represents the LOD for the feature, the coordinates are defined based on the orientation of the feature, and are located on the center of a feature. Our current method computes the resultant of all the PQ vectors for every bounds of the scan target to determine the best scanning position for the laser scanner. The optimal number of scan positions for capturing the required detail for FOI is computed, using the overlap between identified scan positions from different bounds, through an iterative simulation process. The PQ vectors for all the features on the west bound of the courtyard (north side in Figure 4) were computed and the resultant was derived. The west bound of the courtyard has four large windows (the case study’s FOIs) and a door on the right corner. Standardized LOD requirements were acquired from US-GSA (2009) (Table 1.c). According to US GSA, historical documentation of buildings requires LOD 1 or 2. We assigned LOD 1 requirement to the windows. Also, in order to have different data quality requirements within the west bound, we assumed LOD 2 requirement for the door. Figure 4 shows the determined scan position using the proposed framework. Finally, we scanned the courtyard using the determined scan position with a RIEGL VZ-400 terrestrial laser scanner. We set the vertical and horizontal angular resolutions to 0.06 (0.01 m * 0.01 m * 100 m), which is known as a medium resolution setting. The length of the windows were manually measured using a Total Station and were used as the ground truth data. We then measured the same length using the scan data to evaluate the results (Table 1b). In addition, we took point cloud samples of (50 mm * 50 mm) from the four windows to verify the required data density is met. The results indicated that the scan data successfully satisfied the LOD 1 requirements for all the windows. As can be seen in Table 1c, the resolution and accuracy have decreased for the windows that are farther and have greater angle of incidence. The comparison of the proposed scan planning results with common industry practices will be part of the future work.

Figure 4. Determination of Scanner Position based on Features Orientations CONCLUSION This paper presented a data quality-driven scan planning framework and a pilot case study for preliminary evaluation of the framework. The authors did not study all the existing geometrical features of the case study and the laser scanner position was determined through a semi-automated approach based on limited parameters. In addition, the reconstructed point cloud had some missing data due to the distortion in the images. The future work will focus on: 1) improving the accuracy of 3-D 7

reconstructed model using the UAV sequence images as well as exploring alternative imaging methods such as stereo vision camera systems, 2) creating an ontology of all parameters that have to be considered for a data quality-driven scan planning, 3) studying the interactions between different scan planning parameters, 4) evaluating existing image-based and 3-D point cloud-based feature detection techniques for extracting FOI, and 5) proposing an optimization algorithm to define the scan plan based on the feature map of the project. Finally, the computational complexity of the proposed framework will be evaluated and necessary modifications will be made. ACKNOWLEDGEMENTS The authors would like to thank Mrs. Eloisa Dezen-Kempter and Mr. Meida Chen for their supports particularly in data collection for this research. REFERENCES Anil, E., Akinci, B., and Huber, D. (2011). "Representation requirements of as-is building information models generated from laser scanned point cloud data." ISARC. Balali, V., and Golparvar-Fard, M. (2014). "Video-Based Detection and Classification of US Traffic Signs and Mile Markers using Color Candidate Extraction and Feature-Based Recognition." Computing in Civil and Building Engineering (2014), 858-866. Balali, V., and Golparvar-Fard, M. (2015). "Segmentation and recognition of roadway assets from carmounted camera video streams using a scalable non-parametric image parsing method." Automation in Construction, 49, Part A(0), 27-39. Dai, F., Rashidi, A., Brilakis, I., and Vela, P. (2012). "Comparison of Image-Based and Time-ofFlight-Based Technologies for 3D Reconstruction of Infrastructure." Construction Research Congress 2012, 929-939. Golparvar-Fard, M., Peña-Mora, F., and Savarese, S. (2009). "Application of D4AR–A 4-Dimensional augmented reality model for automating construction progress monitoring data collection, processing and communication." ITcon, 14, 129-153. Ham, Y., and Golparvar-Fard, M. (2013). "EPAR: Energy Performance Augmented Reality models for identification of building energy performance deviations between actual measurements and simulation results." Energy and Buildings, 63(0), 15-28. Kersten, T., and Lindstaedt, M. (2012). "Image-Based Low-Cost Systems for Automatic 3D Recording and Modelling of Archaeological Finds and Objects." Progress in Cultural Heritage Preservation, Springer Berlin Heidelberg, 1-10. Pradhan, A., and Moon, F. (2013). "Formalized Approach for Accurate Geometry Capture through Laser Scanning." Computing in Civil Engineering (2013), 597-604. Rodriguez-Gonzalvez, P., Gonzalez-Aguilera, D., Lopez-Jimenez, G., and Picon-Cabrera, I. (2014). "Image-based modeling of built environment from an unmanned aerial system." Automation in Construction, 48(0), 44-52. Shen, Z., Tang, P., Kanaan, O., and Cho, Y. (2013). "As-Built Error Modeling for Effective 3D Laser Scanning on Construction Sites." Computing in Civil Eng., 533-540. Song, M., Shen, Z., and Tang, P. (2014). "Data Quality-oriented 3D Laser Scan Planning." Construction Research Congress 2014, 984-993. Tang, P., Huber, D., Akinci, B., Lipman, R., and Lytle, A. (2010). "Automatic reconstruction of asbuilt building information models from laser-scanned point clouds: A review of related techniques." Automation in Construction, 19(7), 829-843. US-GSA (2009). "BIM Guide for 3D Imaging." http://www.gsa.gov. Viola, P., and Jones, M. "Rapid object detection using a boosted cascade of simple features." Proc., Computer Vision and Pattern Recognition, 2001. CVPR 2001. Wu, C. (2011). "VisualSFM: A Visual Structure from Motion System." Zhu, Z., and Brilakis, I. (2009). "Comparison of optical sensor-based spatial data collection techniques for civil infrastructure modeling." Journal of Computing in Civil Engineering, 23(3), 170177.

8

9