Bridging the gap between point clouds for GeoAI: Role of supervised, reinforced and unsupervised learning Frank Xue Dept. of Real Estate and Construction, University of Hong Kong 10 December 2023, Hong Kong, China Introduction to HKU iLab – the urban big data lab  https://ilab.hku.hk/  44 active members  >25 alumni Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 2 My background  Xue, Fan (Frank)  Edu. background  BEng in Automation 2004 2007  MSc in Computer Science  PhD(PolyU) in System Engineering 2013  PDF*/RAP/AP in Construction IT  Research interests  Professional  MACM, MHKGISA, MIEEE, SMCGS, MASC,  Vice-Chair ACM-HK, Com. CGS-BIM, Com. ASC-Smart Construction  15M grants, 150 papers, 30 awards  Urban sensing and computing  As-built BIM and Digital Twin  Automation/IT in construction  Operations research, ML  Blockchain applications in construction Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 3 Section 1 INTRODUCTION TO POINT CLOUD AND GEOAI Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 4 1 GeoAI  The world we live  Needs in-depth knowledge  For sustainable development  GeoAI  First coined at ACM SIGSPATIAL 2017 o “AI for Geographic Knowledge Discovery” Workshop (6th GeoAI this year) o A.k.a. “Geospatial AI”  Enriches a computer technology ‘AI’ with ‘Geo’ knowledge, to me o E.g., database  GIS, with geographical concepts and spatial laws  Contributes to the SDGs, hopefully back to computer science, too o Like GIS  graph database, due to intensive geometric (e.g., intersection), topological (paths), metric operations (distance buffer) Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 5 1 3D point cloud  3D Point  A location in space, 0D (no width, length, or thickness)  Structured format: {x, y, z}, [R, G, B, Nx, Ny, Nz, Cls, Int., …]  Cloud 0D 1D 2D 3D  An unstructured collection [of water droplets or ice crystals]  Dense when looking at a distance, sparse closely  Urban point cloud A point cloud of HKU Campus (Source: Author, 2019) Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 A close look of cloud at Mount Hua (Source: Author) 6 1 Sources and typical applications  SAR  Use cases  mm-accuracy  Coverage  LiDAR  mm/cm/dm Ground settlement, UNESCO heritage sites (Wu et al. 2020a; Tapete & Cigna 2017))  No distortion  Intensity  Photogrammetry 3D details (Xue et al. 2019b), indoor CFD simulation (Source: Xue 2023a)  cm-accuracy  Colorful  Cheaper Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 Kowloon Wall City 3D model HKU @MineCraft (Source: Xue et al., 2023b) 7 (Source: patrick-@sketchfab.com) 1 Gap between the points for GeoAI  Semantic gap [Point-level]  No geo knowledge representation, e.g., class, materials, etc.  “semantic labeling”  Relational gap [Point patch-level]  “unstructured [collection of] data”  “object detection” as patch Semantic labels (Li et al. 2023)  Patch to geo object  Hierarchical gap [object level]  Taxonomy and system of geo objects  Other issues  Spatial occlusion, clutters, uneven density, etc. Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 Heritage building 3D scan: 廣同會館 30 million points (Source: OkayGIS, 2021) 8 Section 2 ML’S ROLE Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 9 2.1 Supervised Learning  Supervised learning  E.g., most deep learning  Labels each point  Fills the semantic gap  Needs annotation data (💸💸💸💸💸💸)  HRHD-HK dataset  Based on PlanD (2019) 3D model  World only HRHD dataset, with sea & mount  7 semantic labels, 9.375 km2  273 million color points, in HKGS 1980  Download https://doi.org/10.25442/hku.23701866.v1 (Li et al. 2023) Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 10 2.1 Further GeoAI  Enhancing the AI with Geospatial knowledge (a working paper)  + HK boundary, rough footprint, DTM, road network  Improved deep learning on F0 Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 11 2.2 Reinforced learning  Reinforced learning  Evolves iteratively from reward functions  Match points to similar 3D objects  Partially fills the relational gap  Time = 6.44s (Manual > 300s)  RMSE = 3.87 cm  (Xue et al. 2019) Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 12 2.3 Unsupervised learning ISPRS P&RS 2020 Featured Article  Unsupervised learning  Groups similar by attributes  A.k.a. clustering of objects  Fills the hierarchical gap  E.g., CIM (Xue et al. 2020)  Symmetry detection o By reinforced learning  Cross-sections o Longitudinal / transverse  Hierarchical clustering of patches using cross-sections  Clustering of 3D models onto Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 patches for CIM Section 3 SUMMARY Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 14 3 A recap  GeoAI = Geo × AI  rather than Geo + AI, from my perspective  Point cloud  Indispensable data source, yet with many gaps  Machine learning  Possible future directions  Time-dynamic spatiotemporal  Geo-interpretable AI  Geo-self-supervised AI  Geo-probability in AI  Supervised: fills semantic gap o Geo data helps AI in turn  Reinforced: partially fills relational gap o Existing 3D models help AI  Unsupervised: fills hierarchical gap o Design sections and models helps AI Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 15 Acknowledgement  Presented work supported by  HK RGC GRF/ECS (17201717, 17200218, 27200520)  Guangdong S&T (2020B1212030009, 2023A1515010757)  HK ITC (ITP/004/23LP)  And an energetic team  Students’ works involved or cited  Maosu Li  Yijie Wu  Siyuan Meng Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 16 References  Li, M., Wu, Y., Yeh, A. G. O., & Xue, F. (2023). HRHD-HK: A benchmark dataset of high-rise and high-density urban scenes for 3D semantic segmentation of photogrammetric point cloud. Proceedings of 2023 IEEE International Conference on Image Processing Challenges and Workshops. 3714-3718. https://doi.org/10.1109/ICIPC59416.2023.10328383  Tapete, D., & Cigna, F. (2017). InSAR data for geohazard assessment in UNESCO World Heritage sites: State-of-the-art and perspectives in the Copernicus era. International journal of applied earth observation and geoinformation, 63, 24-32.  Wu, S., Yang, Z., Ding, X., Zhang, B., Zhang, L., & Lu, Z. (2020). Two decades of settlement of Hong Kong International Airport measured with multi-temporal InSAR. Remote Sensing of Environment, 248, 111976.  Xue, F., Lu, W., Chen, K., & Zetkulic, A. (2019). From semantic segmentation to semantic registration: derivative-free optimization–based approach for automatic generation of semantically rich as-built building information models from 3D point clouds. Journal of Computing in Civil Engineering, 33(4), 04019024.  Xue, F., Lu, W., Chen, Z., & Webster, C. J. (2020). From LiDAR point cloud towards digital twin city: Clustering city objects based on Gestalt principles. ISPRS Journal of Photogrammetry and Remote Sensing, 167, 418-431. 2020 Featured Article.  Xue, F., Lu, W., Tan, T., & Chen, K. (2019b). Semantic enrichment of city information models with LiDAR-based rooftop albedo. In Sustainable Buildings and Structures: Building a Sustainable Tomorrow (pp. 207-212). CRC Press. Best Paper Award   Xue, F., Zhang, W., Xu, G., Zhou, Q., & Wu, Y. (2023a). Surface or skeleton? Automatic hierarchical clustering of 3D point clouds of bronze frog drums for heritage digital twins. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 293-299. Xue, F., Chen, Z., Wang, J., & Chan, I. (2023b). Invigorating AEC education using Minecraft: A case of LiDAR surveying and virtual learning. In EC3 Conference 2023 (Vol. 4, pp. 0-0). European Council on Computing in Construction. Xue: PCD for GeoAI. HKPolyU, Hong Kong. 2023 17 Thank you! Q&A