Digital Heritage Conservation With Spatial
Information Technologies
Anthony GO Yeh, Fan Xue
Faculty of Architecture, University of Hong Kong
國家文物局“空間信息技術”基地 - 港珠澳工作站（港大）

25 Jul 2024

0 The HZM Station (HKU)
 國家文物局“空間信息技術”重點科研基地（清華大學）
 通過空間信息技術在文化遺产保護中的應用研究，全面强化

文物科技創新，推進文物領域交叉學科建設，加强建築遺产
保護研究力量，为中國特色的建築遺产保護奠定理論基礎。

 港珠澳工作站（香港大學）
 該工作站的研究以大灣區嶺南及港澳特色文化遺產為研究對

象，由葉嘉安院士作為總召集人，綜合發展面向數字遺產文
物保護的空間信息技術。https://smartheritage.hku.hk/

 Team from all the departments:
o Director: Prof Anthony Yeh (DUPAD),
o AD: Frank (REC),

o Members: Kasing (REC), Katherine (REC), Linda (DLA), Bin

(DLA)

Xue: Digital heritage. Jul. 2024.

2

0 My background
 Xue, Fan (Frank)
 Edu. background
2004
2007

 Professional

 BEng in Automation

 MACM, MHKGISA, MIEEE, SMCGS,

MASC, MISDE

 MSc in Computer Science

 Vice-Chair ACM-HK, Com. CGS-BIM,

 PhD(*PolyU) in System Engineering

2013  PDF*/RAP/AP/AssocP in Construction IT

 Research interests
 Urban sensing and computing
 As-built BIM and Digital Twin
 Automation/IT in construction

Com. ASC-SC

 Engineering panel of RGC APSF

 16M grants, >100 papers, 30 awards


ESI Top 1% Researcher



World Top 2% Scientist

 Operations research, ML
 Blockchain applications in construction

Xue: Digital heritage. Jul. 2024.

3

Section 1

INTRO TO DIGITAL HERITAGE
CONSERVATION

Xue: Digital heritage. Jul. 2024.

4

1 Case of existing: Notre-Dame of Paris
 UNESCO world

heritage site

 A fire in 4/2019
o Last time 1789

 Reopens in 12/2024

(Src: WaPo staff)
Xue: Digital heritage. Jul. 2024.

Notre-Dame: building a digital twin | CNRS (https://youtu.be/p-2J0H5i6-4?si=kBdSDweN0adD9Hc7&t=141)
5

1 Case of hidden: Finding a Viking ship, Norway
 Gjellestad ship in
GPR scans in 2018
 By NIKU

(Norwegian
Institute for Cultural
Heritage Research)

 Excavated in 2021

(Source: NIKU Norway, https://www.youtube.com/watch?v=RXZNk3R8YKU )
Xue: Digital heritage. Jul. 2024.

(Source: Science Norway, https://www.sciencenorway.no/archaeology-viking-agevikings/archaeologists-sound-the-alarm-urgent-action-needed-to-preserve-the-gjellestad-ship-remains/2382363

6

)

1 Case of gone: 张家湾“京杭大运河第一码头”
 1920-50s historical
city model restored
 Drone + old photos

(Source: Guangzhou OkayGIS 2022, our partner)
Xue: Digital heritage. Jul. 2024.

7

Section 2

SOME OF OUR SPATIAL
TECHNOLOGY APPLICATIONS

Xue: Digital heritage. Jul. 2024.

8

2.1 HK Tram Trail (& HKU 100)
 Hong Kong Tramways
 A.k.a. Ding Ding
 120 years in 2024
 Lively and vibrant cultural

heritage

 GPS story App by Yeh (2011)

Xue: Digital heritage. Jul. 2024.

2011

2.1 HK Tram Trail (& HKU 100)

1965

2011

Xue: Digital heritage. Jul. 2024.

Kennedy Town New Praya
堅尼地城新海旁

10

2.2 360 Virtual tour (with story)

Xue: Digital heritage. Jul. 2024.

https://dbdc.catholic.org.hk/RDC/home/tc/index.html

https://dbdc.catholic.org.hk/RDC/001_CathedralofImmaculateConception/church360_chi
n/church360/index.html

https://dbdc.catholic.org.hk/RDC/001_CathedralofImmaculateConception/church360_chi
n/church360/index.html

2.2 Pokfulam Conduit (story, mockup)
 Pokfulam Conduit (Built 1876-77), now a hiking trail
 VR-ready web GIS http://147.8.124.72:8080/vrtour/canal/
 Unity 3D https://play.unity.com/mg/other/webgl-builds-217561



Xue: Digital heritage. Jul. 2024.

A project collaborated with Dr Katherine
Deng and Dr S.W. Poon

2.4 长城 镇北台
 镇北台
 “万里长城第一台”
 陝西省榆林市
 全國重點文物保護單位 – No. 5-442
 By drone + Luma AI. DEMO (NeRF + webGS)
o More accurate than traditional photogrammetry

Src: Author (2024). https://lumalabs.ai/capture/55b74e2c-59b5-4fc2-9c0b-73317315612b
Xue: Digital heritage. Jul. 2024.

15

Section 3

AUTOMATING 3D MODEL
RECONSTRUCTION

Xue: Digital heritage. Jul. 2024.

16

3.1 A general workflow of Scan-to-BIM/CIM
 In 4 steps

 Error metrics
differ quite a lot

1.1 Point-level

 Between steps

1.2 Primitive-level
1.3 3D BIM details
2. BIM/CIM Applications
Input 3D scan

(Src: CV4AEC, Authors, Xue
et al. 2019)
Xue: Digital heritage. Jul. 2024.

1.1 Segmentation

Time cost: ~5%

Point-level (0D)

1.2 General
architectural
elements (AEs)
wall

floor

column

beam

Time cost: ~20%

primitive-level (2.5D)

1.3 Detailed BIM
objects
door

window

lighting

HVAC

furniture
Time cost: ~75%

object-level (3D)

2. BIM/CIM
applications
BIM/
OpenBIM
CIM

system-level (nD)
17

3.1 Point-level classification (Wu et al, 2023; Li et al. 2023)
 E.g., supervised Deep learning
 Adds a ‘label’ to each point

DL

o “Wall”, “columns”, “tree”…

Trains

 Point-level semantics

 A high-rise high-density dataset
 150 tiles of HKI and KLN
o From LandsD/PlanD’s city model

 Sampled and annotated for city objects

SBASE/FLKPP
A gold and a silver, 3rd Scan2BIM
Challenge, CVPR2023

An HRHD urban dataset (Li et al. 2023)
Supports

 To be open-sourced soon

Xue: Digital heritage. Jul. 2024.

18

3.2 2.5D general AEs
 Either manual or region
growth
 Collusion adds new errors
KPConv

PointContrast

Manual modelling with point segmented. Src: Author (2024)
Xue: Digital heritage. Jul. 2024.

PointNeXt

Tested with Scan-to-BIM Challenge, CVPR2023/24
Point-level mIoU: > 0.82
Object-level mIoU ~ 0.45
Src: Wu et al (2023a)
19

3.3 Similarity for registering 3D objects (Xue et al. 2016; 2019b)
 Time = 6.44s (Manual = 300s), RMSE = 3.87 cm

Xue: Digital heritage. Jul. 2024.

 (Class = Furniture)

20

3.3 Applications to office and heritage DT
Fitting BIM objects for location, rotation, and
relational semantics (Xue 2019)

Xue: Digital heritage. Jul. 2024.

Fitting 3D columns for a timber architecture
(廣州梅庵, Source: OkayGIS)

21

3.4 Clustering using 3D skeletons
 Bronze drums (Lu et al., 2020)
 Spiritual, sacrifice, and musical instruments
 Across South China and Southeast Asia since 700s BCE
 Traditional storage: buried in soil (now >2,400 conserved)
 8 types 八大類型（ 萬家壩、 石寨山、冷水沖、北流等）

 Decorative bronze frogs have more diversity 細節更多樣

Discovery of a 2,000-yr drum in
Guangxi, on 25 May 2023
(Photo src: news.cn)

 A variety (design, size, materials) evolved over time
 “bronze drums often unearthed in Guangxi by the tillers … with a

perfect circle with bent body … five sitting frogs, each with a
baby on its back.” (Zhou 1187) 周去非 (宋)《岭外代答》

A1

 “surrounding frogs indicate [the chief’s] title; the more frogs, the

more honorable title.” (Zhu 1948) 朱昌奎《宾阳县志·铜鼓考》

Xue: Digital heritage. Jul. 2024.

B1
22

3.4 Clustering using 3D skeletons
 Cases: 铜鼓蛙饰
 3 types, 14 frogs

 Objectives
 Grouping similar

(Auto detect defects)

 3D shape skeleton

 3 steps
 Dis-similarity
 Auto Clustering

(e.g., Covid-family tree)
 Comparison
Xue: Digital heritage. Jul. 2024.

(Xue et al. 2023a)

23

3.4 Clustering using 3D skeletons
 Both surface points &
shape skeleton

 A “family” tree of all frogs

 Similar patterns
 maxske/maxsur≈2

 Clustering (Obj. #1)
 Four groups in both
 Reflecting the style
 In line with the instruments

and shape groups

 Outlier (damaged)

highlighted automatically

Xue: Digital heritage. Jul. 2024.

24

3.4 Clustering using 3D skeletons
 Comparative results (Obj. #2)
 Same for 12 out of 14 ✔
 Assuming 3D surface grouping

was true, the metrics of skeleton
were: Precision = 0.850, Recall =
0.883, F1 = 0.866

 2 inconsistencies due to: limited

presentation of traditional
(inscribed ball) skeleton in CGAL

Group
using
surface
point
clouds

I. Three-legged
with baby
II. Four-legged
with baby
III. Three-legged
IV. Four-legged
V. Unusual

Xue: Digital heritage. Jul. 2024.

Group using shape skeleton
I
II
III IV V
3
2

1
3

3

1
1

25

3.5 Texture segmentation
 Suitable cases: Textures on a surface

 Cases: 广同会馆、广西铜鼓

 As carving/casting

初步計算過程：(1) NURBS 擬合 + (2) 邊緣檢測 + (3) DBSCAN 臨近聚類 (Meng et al. 2023)
Xue: Digital heritage. Jul. 2024.

26

3.6 4D point cloud for streaming actions
现
实
施
工
过
程
数 字孪生
4DPC
Xue: Digital heritage. Jul. 2024.

 Fine motions
 For some
intangible
heritage ?
 Demo

http://147.8.124.72:8080/vr
tour/SassoonRoad/dt.html

Section 4

SUMMARY
小結

Xue: Digital heritage. Jul. 2024.

28

4 A recap
 Digital model
 Very useful for cultural heritage conservation
 No matter existing, hidden, or gone

 Spatial information technology
 Vital data source for digital heritage

 Auto modeling of 3D points
 Point-level, object-level

 Limitations
 A huge gap between point-level and object-level detection
 Low automation level

 New opportunities like 3DGS/4DPC
Xue: Digital heritage. Jul. 2024.

29

Acknowledgement
 Presented work supported by
 HK RGC GRF/ECS (17201717, 17200218, 27200520)
 Guangdong S&T (2020B1212030009, 2023A1515010757)
 HK ITC (ITP/004/23LP)

 Courtesy by Prof Yeh and OkayGIS
 Students’ works involved or cited
 Maosu Li
 Yijie Wu
 Siyuan Meng
 Dong Liang
 Sou-Han Chen
Xue: Digital heritage. Jul. 2024.

30

Thank you!
Q&A