童 磊

UAVs have a broad application prospect in the field of railway inspection due to their excellent mobility and flexibility. However, it still faces challenges, such as high human labor costs and low intelligence levels. Therefore, it is of great significance to develop a real-time intelligent rail recognition algorithm that can be deployed on the onboard computing device to guide the UAV’s camera to follow the target rail area and complete the inspection automatically. However, a significant issue is that rails from the perspective of UAVs may appear with changing pixel widths and various inclination angles. Concerning the issue, a general and adaptive rail representation method based on projection length discrimination (RRM-PLD) is proposed. It can always select the optimal representation direction, horizontal or vertical, to represent any kind of rails. With the RRM-PLD, a novel architecture (Real-Time Rail Recognition Network, TriRNet) is proposed. In TriRNet, a designed inter-rail attention (IRA) mechanism is presented to fuse local features of single rails and global features of other rails to accurately discriminate the geometric distribution of all rails in the image in a regressive way and thus improve the final recognition accuracy. Further, one-to-one mapping from anchor points to final feature maps is established. It greatly simplifies the model design process and improves the model's interpretability. Besides, detailed model training strategies are also presented. Extensive experiments have verified the effectiveness and superiority of the proposed formulation in terms of both network reasoning latency and recognition accuracy.

Accepted version (Oct. 20, 2023)

 
@ARTICLE{10310655,
  author={Tong, Lei and Wang, Zhipeng and Jia, Limin and Qin, Yong and Song, Donghai and Miao, Bidong and Tang, Tian and Geng, Yixuan},
  journal={IEEE Transactions on Intelligent Transportation Systems}, 
  title={TriRNet: Real-Time Rail Recognition Network for UAV-Based Railway Inspection}, 
  year={2023},
  volume={},
  number={},
  pages={1-17},
  doi={10.1109/TITS.2023.3328379}
}
         

Autonomous railway inspection with unmanned aerial vehicles (UAVs) has huge advantages over traditional inspection methods. As a prerequisite for UAV-based autonomous following of railway lines, it is quite essential to develop intelligent railway track detection algorithms. However, there are no existing algorithms currently that can efficiently adapt to the demand for the various forms and changing inclination angles of railway tracks in the UAV aerial images. To address the challenge, this paper proposes a novel anchor-adaptive railway track detection network (ARTNet), which constructs a dual-branch architecture based on projection length discrimination to realize full-angle railway track detection for the UAV aerial images taken from arbitrary viewing angles. Considering the potential capacity imbalance of the two branches that can be caused by the uneven distribution of railway tracks in the dataset, a balanced transpose co-training strategy is proposed to train the two branches coordinately. Moreover, an extra customized transposed consistency loss is designed to guide the training of the network without increasing any computational complexity. A set of experiments have been conducted to verify the feasibility and superiority of the ARTNet. It is demonstrated that our approach can effectively realize full-angle railway track detection and outperform other popular algorithms greatly in terms of both detection accuracy and reasoning efficiency. ARTNet can achieve a mean F1 of 76.12 and run at a speed of 50 more frames per second.

Accepted version (Mar. 18, 2023)

 
@article{https://doi.org/10.1111/mice.13004,
  author = {Tong, Lei and Jia, Limin and Geng, Yixuan and Liu, Keyan and Qin, Yong and Wang, Zhipeng},
  title = {Anchor-adaptive railway track detection from unmanned aerial vehicle images},
  journal = {Computer-Aided Civil and Infrastructure Engineering},
  volume = {38},
  number = {18},
  pages = {2666-2684},
  doi = {https://doi.org/10.1111/mice.13004},
}
         

UAV-based automatic railway inspection is expected to have the potential to reform the inspection of railways. In this area, real-time railway scene parsing is quite essential. However, the limited computation resources of the UAV onboard computer pose a huge challenge for the algorithm to juggle a precise prediction with strong timeliness. Concerning this issue, this paper proposes a novel algorithm named deep fully decoupled residual convolutional network, which consists of fully decoupled residual blocks (Non-bottleneck-FDs) to deal with the dilemma between the high demand of real-time and limited resources. The residual block is constructed based on a new convolution which divides the standard convolution into three sequential convolutions to decouple the conventional operational correlations fully. Furthermore, a customized auxiliary line loss (LL) function is proposed to constrain the segmentation of railway and non-railway simultaneously without increasing the computation complexity. The proposed LL can force the predicted railway areas to concentrate in long strip areas precisely and inhibit their appearances in other impossible local areas. Subsequently, an integrated loss backpropagation strategy of the LL and cross-entropy function is presented. A comprehensive set of experiments are conducted for verification. Experiments demonstrate the superior performance of our approach with a more than 2× reduction in parameters and computation cost. Moreover, our approach also has a faster inference speed than the most existing lightweight architectures while providing comparable or higher accuracy. It is proven that our approach can reconcile the precise prediction with strong timeliness for railway scene parsing within the limitation of onboard computers. Besides, the results also imply its highest performance in terms of local details and edges of railway areas.

Accepted version (Dec. 8, 2021)

 
@ARTICLE{9655444,
  author={Tong, Lei and Wang, Zhipeng and Jia, Limin and Qin, Yong and Wei, Yanbin and Yang, Huaizhi and Geng, Yixuan},
  journal={IEEE Transactions on Intelligent Transportation Systems}, 
  title={Fully Decoupled Residual ConvNet for Real-Time Railway Scene Parsing of UAV Aerial Images}, 
  year={2022},
  volume={23},
  number={9},
  pages={14806-14819},
  doi={10.1109/TITS.2021.3134318}
}
     

The normalized and rapid intrusion detection of the railway operation environ-ment is of great significance for real-time monitoring and early detection of for-eign objects that threaten the railway's safe operation. The existing manual patrol and inspection vehicles cannot meet the high demands for rapid detection of some occasional or accidental risks in a short period, such as the intrusion of potential or actual foreign objects. Combining railway segmentation and foreign object de-tection, this paper proposes a two-stage perception model to achieve hierarchical area-matched intrusion detection and advance the risk assessment of the railway operating environment under the wide field of view of unmanned aerial vehicles (UAVs). With the railway area accurately segmented, the hierarchical division of different risk areas and the foreign objects detection in those hierarchical areas is realized. Leveraging the maneuverability and flexibility of UAVs, the proposed two-stage perception model can effectively improve the efficiency and quality of rapid intrusion inspection of the railway line and surrounding environment.

Accepted version (Jun. 8, 2023)

 
@InProceedings{
  author={Tong, Lei and Wang, Zhipeng and Qin, Yong and Tang, Tian and Miao, Bidong, and Li, Zhaoyu},
  title={Leveraging UAVs for Rapid and Hierarchical Railway Intrusion Detection},
  booktitle={Proceedings of the 6th International Conference on Electrical and Information Technologies for Rail Transportation (EITRT) 2023},
  year={2024},
  publisher={Springer Singapore},
  address={Singapore},
  pages={},
  isbn={}
  doi={}
}
       

At present, the manual inspection along railway lines is still a major method to ensure railway operation safely, but the cost is high and work efficiency is low. Therefore, unmanned aerial vehicles (UAVs) patrol inspection is required. This paper presents the effective segmentation of scenes along railway lines (SRL) from remote sensing perspective of UAVs based on the full convolutional networks (FCN). Firstly, the datasets needed in this research are collected and produced from Langfang section of the Beijing–Shanghai high-speed railway. The datasets are expanded by using data augmentation to constrain the overfitting in the training process. Secondly, the segmentation model FCN-8s for SRL is developed and trained. The related setting and hardware environment in the training process are described in this paper. The experimental results show that a single image prediction needs 151.2 ms, to achieve 6.6 fps when input size is 384 × 384. Good accuracy is obtained on the test dataset, i.e., 55.8% MIoU and 70.2% MPA, which meets the expectations of FCN. At the same time, it is also found that the segmentation of railway area achieves the best result thus the railway area is extracted accordingly.

Accepted version (Jul. 10, 2019)

 
@InProceedings{10.1007/978-981-15-2866-8_47,
  author={Tong, Lei and Jia, Limin and Wang, Zhipeng and Wu, Yunpeng and Wang, Ning},
  title={Research on the Segmentation and Extraction of Scenes Along Railway Lines Based on Remote Sensing Images of UAVs},
  booktitle={Proceedings of the 4th International Conference on Electrical and Information Technologies for Rail Transportation (EITRT) 2019},
  year={2020},
  publisher={Springer Singapore},
  address={Singapore},
  pages={481--492},
  isbn={978-981-15-2866-8}
  doi={10.1007/978-981-15-2866-8_47}
}
       

Deep neural network (DNN) is extensively explored for LiDAR-based 3D object detection, a crucial perception task in the field of autonomous driving. However, the presence of redundant parameters and complex computations pose challenges for the practical deployment of DNNs. Despite knowledge distillation (KD) is an effective approach for accelerating models, extremely small number of efforts explore its potential on LiDARbased 3D detectors. Besides, existing studies neglect to elaborately investigate 3D voxel-wise features for compression. To this end, we propose instance-aware knowledge distillation (InstKD) for 3D detector compression. The proposed method conducts KD by fully excavating two types of knowledge related to 3D voxelwise features. Firstly, the 3D voxel-wise feature of teacher is transferred to teach the student. In order to prioritize the knowledge with strong guiding capacity, we introduce expanded bounding box (E-Bbox) to distinguish and balance the foreground and background regions. Besides, we generate contribution map (CM) by calculating the gap between the classification response of teacher and student models to further dynamically balance individual instance for distillation. Secondly, we also align the relation-based knowledge of 3D voxel-wise features between the distillation pairs. To avoid incalculable relation on a massive number of 3D voxel-wise features, we distill the relation among instances selected by E-Bboxes, where the intra-relation of homogeneous instances and inter-relation of heterogeneous instances are transferred in a dual-pathway manner. In the experiments, we compress different models on benchmarks with varying scales. The results demonstrate that our method achieves the lightweight 3D detector with slight performance drop. For example, on KITTI dataset, our 2× compressed SECOND (75.5% parameters and 74.5% FLOPs reduction) achieves 66.83% mAP, surpassing its teacher model. The key code is available at https://github.com/zhnxjtu/InstKD.

 
@ARTICLE{10531046,
  author={Zhang, Haonan and Liu, Longjun and Huang, Yuqi and Lei, Xinyu and Tong, Lei and Wen, Bihan},
  journal={IEEE Transactions on Intelligent Vehicles}, 
  title={InstKD: Towards Lightweight 3D Object Detection With Instance-Aware Knowledge Distillation}, 
  year={2024},
  volume={},
  number={},
  pages={1-13},
  keywords={Three-dimensional displays;Detectors;Solid modeling;Feature extraction;Task analysis;Object detection;Point cloud compression;3D object detection;model compression;knowledge distillation;point cloud;autonomous driving},
  doi={10.1109/TIV.2024.3401461}}
         

Point cloud semantic segmentation for railway infrastructures is an essential step towards establishing railway digital twins. Deep learning-based methods have shown great potential in this field compared to traditional methods that rely on hand-crafted features. However, deep learning-based methods for railway point clouds still face typical challenges that need to be addressed. In this regard, we propose a novel learning framework named SALAProNet, which consists of a set of effective and concise modular solutions. The first challenge addressed is the massive data scale of railway point clouds, which makes it difficult to directly process large-scale point clouds due to memory limitations. To solve this problem, we adapt efficient random sampling in the network and propose the Self-Attentive Aggregation (SAA) module based on an attention mechanism to greatly expand the receptive field, which covers the unsampled points and successfully retains information in a high-dimensional feature space. The second challenge is fine-grained segmentation, where we propose the Local Geometry Embedding (LGE) module to embed local geometry. With the help of context information provided by SAA, the network can perform fine-grained segmentation for railway infrastructures. The third challenge is the insufficient generalization ability of the network, where we propose a Prototype Guided Regularization (PGR) method to guide the network to segment the point cloud among railways with different construction standards. This method enhances the network’s interpretability and improves its generalization ability. We have validated our proposed framework through experiments on different datasets, and it outperforms state-of-the-art approaches.

 
@ARTICLE{10144473,
  author={Wang, Zhipeng and Geng, Yixuan and Jia, Limin and Qin, Yong and Chai, Yuanyuan and Tong, Lei and Liu, Keyan},
  journal={IEEE Transactions on Intelligent Transportation Systems}, 
  title={Self-Attentive Local Aggregation Learning With Prototype Guided Regularization for Point Cloud Semantic Segmentation of High-Speed Railways}, 
  year={2023},
  volume={24},
  number={10},
  pages={11157-11170},
  doi={10.1109/TITS.2023.3281352}
}
       

Point cloud semantic segmentation (PCSS) is crucial for digital twins of high-speed railways. By now, the concerned subjects are confined within the interior infrastructures of railways. However, the surrounding environments are also important for the safe operation. Concerning this issue, a full-range high-speed railway scanning scheme based on UAV borne LiDAR is utilized. However, the massive data volume and data distribution imbalance pose great challenges for PCSS. To address these issues, a novel PCSS framework called 3DGraphSeg is proposed in this paper. To cope with the massive data volume, a structural representation algorithm named Local Embedding Super-Point Graph (LE-SPG) is proposed to represent the vast point cloud into a concise graph while retain the data's inherent topology structure by local spatial embedding. Then, the Gated Integration Graph Convolutional Network (GIGCN) is proposed to contextual segment the graph. In GIGCN, to prevent the gradients from vanishing or exploding, the hidden states of Gated Recurrent Units (GRUs) in every layer are integrated using a new layer named Gated Hidden States Integration (GHSI). GHSI strengthens the back propagation by giving the loss function direct access to each layer and absorbs the features of different layers comprehensively, which enables the network to produce a smoother decision boundary and prevents the overfitting problem. Besides, to enhance its robustness to data imbalance, we propose a loss function: Adaptive Weighted Cross Entropy (AWCE). Finally, five experiments are designed for verification. The proposed framework has excelled in different datasets and outperforms state-of-the-art approaches on the SemanticRail dataset.

 
@ARTICLE{10049148,
  author={Geng, Yixuan and Wang, Zhipeng and Jia, Limin and Qin, Yong and Chai, Yuanyuan and Liu, Keyan and Tong, Lei},
  journal={IEEE Transactions on Industrial Informatics}, 
  title={3DGraphSeg: A Unified Graph Representation- Based Point Cloud Segmentation Framework for Full-Range High-Speed Railway Environments}, 
  year={2023},
  volume={19},
  number={12},
  pages={11430-11443},
  doi={10.1109/TII.2023.3246492}
}
       

The height and stagger of the contact wire directly affect the energy supply of high-speed trains. To ensure the operation safety, there is an urgent demand for high-speed railways to measure the static parameters of contact wires all over the line with high precision and efficiency. However, this issue is barely discussed. Concerning the issue, this paper proposes a UAV-LiDAR-based measuring framework for the static height and stagger of high-speed railway contact wire. By mounting LiDAR on the UAV, the framework can efficiently collect data from the lines in service without occupying the train operating diagrams. It is extremely significant for the high-speed and high-density railways. Then, we present self-adaptive extraction algorithms to extract critical infrastructures (rails, contact wires, masts and other suspensions) based on their specific geometric characteristics as well as the continuity and consistency of the spatial distributions along the line. Finally, the height and stagger are calculated by formulas automatically. To verify the framework in practice, we tested it on Beijing-Shanghai highspeed railway, which is the busiest high-speed railway in China. It is shown that the measurement error is within 9mm and the framework has potential to reform the inspection of high-speed railways.

 
@ARTICLE{9439910,
  author={Geng, Yixuan and Pan, Fengjun and Jia, Limin and Wang, Zhipeng and Qin, Yong and Tong, Lei and Li, Shiqi},
  journal={IEEE Transactions on Intelligent Transportation Systems}, 
  title={UAV-LiDAR-Based Measuring Framework for Height and Stagger of High-Speed Railway Contact Wire}, 
  year={2022},
  volume={23},
  number={7},
  pages={7587-7600},
  doi={10.1109/TITS.2021.3071445}
}
       

Unmanned aerial vehicle (UAV) is expected to have the potential to complete the collection of defect information in track areas with lower labor costs and higher efficiency. In this field, autonomous railway track detection and real-time tracking to guide UAVs are quite essential. However, the limited computation resources of the UAV onboard computer make it difficult to maintain high accuracy in realtime detection and tracking using the deep learning model with a complex structure. Concerning this issue, for the daily detection scene of the track, this paper proposes a novel autonomous railway track detection and real-time tracking algorithm named Largest Connected-ERFNet, which combines ERFNet and the largest connected component labeling to ensure the accuracy of the track area detection and tracking. A comprehensive set of experiments on UAV onboard computer are conducted for verification. Experiments demonstrate the superior performance of the algorithm proposed in this paper. Under the condition of limited training data and computation resources, the detection precision of the algorithm reaches 89.2%, the detection speed reaches 5.5 fps, and the smoothness reaches 99.4%. It is proven that the proposed method can meet the practical needs of using UAVs for railway track inspection.

 
@article{JIANG20237591,
  title = {Largest Connected-ERFNet for Autonomous Railway Track Detection and Real-time Tracking},
  journal = {IFAC-PapersOnLine},
  volume = {56},
  number = {2},
  pages = {7591-7596},
  year = {2023},
  note = {22nd IFAC World Congress},
  issn = {2405-8963},
  doi = {https://doi.org/10.1016/j.ifacol.2023.10.671},
  url = {https://www.sciencedirect.com/science/article/pii/S2405896323010455},
  author = {Yaopeng Jiang and Zhipeng Wang and Limin Jia and Yong Qin and Lei Tong and Dongzhu Jiang},
  keywords = {fault information collection, railway track inspection, UAV, Semantic segmentation, largest connected component labeling, remote sensing images},
  }
     

The rail fastener is an important infrastructure of railway line system to ensure the safety of railway operation. There is an urgent need for a set of automatic defect inspection scheme for rail fasteners which have high efficiency and accuracy. To this end, this paper proposes a modified lightweight YOLOv5 model considering the application scenario of UAV. We reconstruct the backbone on the basis of ShuffleNetV2 and RepVGG, and switch the detector head to decoupled type from YOLOX. Data augmentation is adopted to address the problem of deficient defect samples. The results show that Yoloxs-lite_s model with ShuffleNetV2 backbone and YOLOX-s head is the most ideal model.

 
@INPROCEEDINGS{9941763,
  author={Dai, Zhen and Wang, Zhipeng and Jia, Limin and Qin, Yong and Tong, Lei and Cui, Jing},
  booktitle={2022 Global Reliability and Prognostics and Health Management (PHM-Yantai)}, 
  title={An Improved Lightweight YOLOv5 Network for Defect Detection of Rail Fasteners}, 
  year={2022},
  volume={},
  number={},
  pages={1-6},
  doi={10.1109/PHM-Yantai55411.2022.9941763}
}
     

With the increase in railway operating mileage, the pressure on railway inspections has also increased significantly. The realization of automated inspections along the railway can not only improve work efficiency, but also save manpower and material resources. The catenary is an important component of the train power system, and it is necessary to study the automatic inspection technology of the catenary. This paper proposes to use a tracking algorithm based on correlation filtering to track the catenary. Considering to meet real-time requirements, the ECO-HC algorithm based on manual features is adopted. The experimental results show that the ECO-HC algorithm basically realizes the tracking of the catenary and the algorithm speed reaches 57 fps, which meets the real-time requirements; the comparison between the ECO-HC algorithm and the classic tracking algorithm based on correlation filtering proves the superiority of the ECO-HC algorithm.

 
@InProceedings{
  author={Liu, Keyan and Jia, Limin and Qin, Yong and Wang, Zhipeng and Tong, Lei and Geng, Yixuan},
  title={Target Tracking for High-Speed Railway Catenary Based on Correlation Filtering Algorithm},
  booktitle={Proceedings of the 5th International Conference on Electrical and Information Technologies for Rail Transportation (EITRT) 2021},
  year={2022},
  publisher={Springer Singapore},
  address={Singapore},
  volume={867},
  pages={243-250},
  isbn={978-981-16-9909-2}
  doi={10.1007/978-981-16-9909-2_27}
}