Pavement damage is the main factor affecting road performance. Pavement cracking, a common type of road damage, is a key challenge in road maintenance. In order to achieve an accurate crack classification, segmentation, and geometric parameter calculation, this paper proposes a method based on a deep convolutional neural network fusion model for pavement crack identification, which combines the advantages of the multitarget single-shot multibox detector (SSD) convolutional neural network model and the U-Net model. First, the crack classification and detection model is applied to classify the cracks and obtain the detection confidence. Next, the crack segmentation network is applied to accurately segment the pavement cracks. By improving the feature extraction structure and optimizing the hyperparameters of the model, pavement crack classification and segmentation accuracy were improved. Finally, the length and width (for linear cracks) and the area (for alligator cracks) are calculated according to the segmentation results. Test results show that the recognition accuracy of the pavement crack identification method for transverse, longitudinal, and alligator cracks is 86.8%, 87.6%, and 85.5%, respectively. It is demonstrated that the proposed method can provide the category information for pavement cracks as well as the accurate positioning and geometric parameter information, which can be used directly for evaluating the pavement condition.
In 2016, Zhang et al. [18] proposed a crack detection method based on deep learning. They trained a deep CNN based on supervised learning, proving the feasibility of combining deep learning with pavement crack recognition. In 2017, Zhao et al. [19] proposed a pavement crack detection method based on a CNN using images of different scales and taken at different angles for training, achieving the detection of cracks of various shapes. However, owing to road surface interference and noise, the detection accuracy of this system peaked at 82.5%. In 2017, Markus et al. developed the open dataset GAPs for the training of deep neural network and evaluated the pavement damage detection technology for the first time, which is of great significance [20, 21]. In 2018, Nhat-Duc et al. [22] established an intelligent method for the automatic recognition of pavement crack morphology; this study constructs a machine learning model for pavement crack classification that included multiple support vector machines and an artificial swarm optimization algorithm. Using feature analysis, a set of features is extracted from the image projection integral, which can significantly improve the prediction performance. However, the algorithm is complex and programming it becomes very difficult. In 2020, Zhaoyun Sun et al. [23] proposed a method to detect pavement expansion cracks with the improved Faster R-CNN, which can achieve accurate expansion crack location detection through the optimization model. The aforementioned studies only detect and classify pavement cracks and their location but cannot quantify certain crack characteristics, such as crack width and area. On the other hand, there are also many studies on crack segmentation. In 2018, Zhang and Wang [24] proposed CrackNet, which is an efficient architecture based on CNN to predict the class of each image pixel, but its network structure is related to input image size, which prevents the generalization of the method. In the same year, Sen Wang et al. [25] proposed to use the full convolutional networks (FCNs) to detect cracks and built the Crack-FCN model taking into account the shortcomings of the FCN model in the crack segmentation experiment and obtained a complete crack image. However, the highest accuracy obtained by their method is only 67.95%; thus, segmentation performance needs to be improved. In 2019, Piao Weng et al. [26] proposed a pavement crack segmentation method based on the VGG-U-Net model. It solves the problem of fracture in the crack segmentation result in complex background, but its training time is slightly longer and its efficiency is low. In 2020, Zhun Fan et al. [27] proposed an encoder-decoder architecture based on hierarchical feature learning and dilated convolution (U-HDN) detects cracks in an end-to-end manner. The U-HDN method can extract and fuse different context sizes and different levels of feature mapping, so it has high performance. In the same year, Zhun Fan et al. [28] proposed an ensemble of convolutional neural network based on probability fusion for automatic detection and measurement of pavement cracks, and the predicted crack morphology is measured by skeleton extraction algorithm. In summary, these previous studies only use the segmentation method, which cannot achieve accurate crack classification and location determination.
crack Fusion Connect 2016 download
Given the abovementioned problems in pavement crack identification, this paper proposes a method based on a deep convolutional neural network fusion model for pavement crack identification, which is applicable in many crack detection cases (including detector vehicle and smartphone). By training on a learning image data having a variety of sources and sizes, the method can effectively identify cracks, and recognition accuracy can be guaranteed. At the same time, a detected crack can be segmented, and the segmented binary image can be used to calculate the geometric parameters of the crack. Therefore, the proposed model is of great significance for intelligent pavement detection and it can also achieve detection and segmentation simultaneously, thereby significantly improving model efficiency.
In this paper, a crack identification method based on a deep CNN fusion model is proposed. First, the image dataset is established, and the image noise in the dataset is filtered out to increase the contrast between road cracks and background. Next, the processed images are provided as input into an improved single shot multibox detector (SSD) crack detection model and an improved U-Net crack segmentation model for training. Then, the binary image of a crack obtained by the segmentation model is used to calculate the geometric parameters of the crack. By integrating the advantages of the two models, this pavement crack identification method can effectively overcome the single-model limitations of inaccurate positioning and imperfect information. The overall process flowchart is shown in Figure 1. The details of each step are discussed in Section 2.1.
Increasing the number of network layers can improve the accuracy of the network in identifying pavement cracks. Therefore, in this study, the feature extraction network structure Visual Geometry Group 16 (VGG16) in the SSD network model was replaced with a deep residual network to improve the pavement crack identification accuracy. The deep residual network [33, 34] solves this problem by fitting a residual map instead of the original map and by adding multiple connections between layers.
By replacing the feature extraction structure of the original SSD network with the deep residual network, the network accuracy and recall rate in predicting pavement cracks were substantially improved. This analysis of the experimental results shows that the proposed method achieves good results in the classification and detection of cracks. From the prediction effect, however, a classification by the pavement crack detection method based on the single SSD crack location model is incomplete, is not conducive to subsequent crack geometry parameter computation steps, and will produce larger calculation errors. Thus, as the practical application value is still lacking at this point, this study adopted the fusion segmentation model approach to address this problem.
The crack segmentation model uses the cascade mode of multiple residual elements, which can effectively extract the morphological characteristics of the pavement crack and improve the learning effectiveness of the neural network on the crack characteristics. A single crack segmentation method based on U-Net can provide the crack pixel location information, but it cannot classify the crack [35]. Therefore, in this study, a fusion of two models was adopted to identify pavement crack images and to obtain the crack category, location information, and geometric parameters, thereby facilitating accurate quantification and evaluation of pavement cracks.
As shown in Figure 17, when only the SSD detection network is used, the number of cracks can be accurately obtained, but the crack width cannot be quantified. Use of only the U-Net segmentation model will lead to misjudgment of the number of cracks; e.g., a fractured crack may be misidentified as multiple cracks. The fusion of the detection and segmentation networks can avoid this phenomenon and ensure that the crack is identified as a single crack. Thus, the advantage of the fusion model is that it can accurately identify the number of cracks and ensure that cracks are quantified correctly.
The proposed fusion model adopts the following order: (1) detection, (2) segmentation. Because the detection network will obtain the result of a detection box, which contains one crack, and then segment that crack. This method can not only ensure crack number accuracy but also quantify the crack width information, which can prevent the misjudgment of crack number caused by using only the segmentation model. If the order is changed, i.e., segmentation first and detection second, in the presence of fractures in the cracks, the segmentation result will be misjudged into multiple cracks. Moreover, the segmented image is a binary image, which is not suitable for detection.
In order to demonstrate the effectiveness of the model proposed in this paper, three sample images of each pavement crack type were randomly selected from the test set, and the model-identified parameter information for each crack is shown in Table 11. The confusion matrices for the training and testing phases are shown in Figure 25. 2ff7e9595c
Comments