Design for eXcellence (DfX) with Digital Twins: From Reality Data to Semantic Models to Optimized Design Fan Xue Dept. of Real Estate and Construction, University of Hong Kong at Faculty of Architecture and the Built Environment, TU Delft 30 June 2023, Delft, Netherlands Section 1 INTRODUCTION TO OUR LAB Xue: DfX with DT. TUDelft, 30 Jun 2023. 2 1 Hong Kong The only city hosting 5 of world’s top 100 universities (QS2024) - HKU - CUHK - HKUST - HKPolyU - HKCityU Xue: DfX with DT. TUDelft, 30 Jun 2023. 3 1 University of Hong Kong  Since 1912  10 Faculties  China’s only English language comprehensive research-based university (Source: CPAO Multimedia, HKU) Xue: DfX with DT. TUDelft, 30 Jun 2023. 4 1 Faculty of Architecture Xue: DfX with DT. TUDelft, 30 Jun 2023. (Source: FoA, HKU) 5 1 iLab – the urban big data lab  https://ilab.hku.hk/  30 + 11 members  Two interest groups  23 alumni Xue: DfX with DT. TUDelft, 30 Jun 2023. 6 1 My background and research interest  Xue, Fan (Frank)  Professional  Edu. background  MACM, MHKGISA, MIEEE,  BEng in Automation 2004  MSc in Computer Science 2007  PhD in System Engineering  SMCGS, MASC, MHKABAEIMA  Vice-Chair ACM-HK, Com. CGS-BIM, Com. ASC-Smart Construction 2013  PDF/RAP/AP in Construction IT  Research interests  Urban sensing and computing  As-built BIM and Digital Twin  Automation/IT in construction  Operations research, ML  Blockchain applications in construction Xue: DfX with DT. TUDelft, 30 Jun 2023. 7 1 My recent work #1: Scan-to-BIM automation  Winner of Scanto-BIM Challenge,  2. 3D deep learning segmentation + clustering (3+ years) CVPR2023  3. Our fully auto “dream”: Automate >60% workload (1 year)  1. Normal segment + mesher + ifcopenshell (10+ years) 1. Mesher + ifcopenshell 2. 3D DL + clustering 3. Our fully auto dream Xue: DfX with DT. TUDelft, 30 Jun 2023. Not covered Not covered Not covered NC 8 1 My recent work #2: Clustering for heritage DT  Traditional deviation: 3D surface  Cases: Bronze frog drums  Q1: Clustering (like building a Covid family tree)?  Result: F1 = 0.87.  Q2: Can shape skeleton ?  Not bad, 0.13 rooms to improve Xue: DfX with DT. TUDelft, 30 Jun 2023. (Xue et al. 2023) 9 Section 2 DFX WITH DT Xue: DfX with DT. TUDelft, 30 Jun 2023. 10 2 DfX, generative design  Design for MA  Manufacturing and Assembly  Design for X / eXcellence also in  structure, quality, cost, logistics, sustainability, resilience, …  Objectives to optimize, better with reality data  Generative design (Krish 2011)  A design exploration process o Given an idea o Populated by an algorithm (iterative sometimes) o Judged (optimized) by human designers based on the outputs  Designer (decision-maker) as a human  A human-centric approach for DfX Xue: DfX with DT. TUDelft, 30 Jun 2023. Generative design process (Krish 2011) 11 2 DfX with DT  A digital twin (UKNIC 2017)  “A virtual representation of a physical object or system  across its lifecycle, using real-time data  to enable understanding, learning, and reasoning.”  How can DT enable in Krish’s (2013) diagram ?  DTs of building materials  DTs of common styles  DTs of environment  “AI designer” (DT of low-level decision-making) Xue: DfX with DT. TUDelft, 30 Jun 2023. 12 2.1 Case 1: DfX with DTs of materials  ZCB Bamboo Pavilion  Dr. Kristof Crolla kcrolla@hku.hk CLIENT BRIEF - Host public events - Showcase bamboo potential - Promote sustainability Q: maximum usage for sustainability?  Associate Prof.  Investigator of our on-going DfX project Xue: DfX with DT. TUDelft, 30 Jun 2023. 13 2.1 Method: Optimizing poles to bamboo “DTs” Xue: DfX with DT. TUDelft, 30 Jun 2023. 14 2.1 Completion and awards Xue: DfX with DT. TUDelft, 30 Jun 2023. Web: Email: bsl.hku.hk kcrolla@hku.hk 15 2.2 Case 2: DfX with DTs of common styles  City mesh/point cloud models too huge  Q1: Compact building reconstruction?  Q2: Generating blocks for mimicking local styles?  Miss Yijie Wu yijiewu@connect. hku.hk  Yr-2 PhD candidate 50KB 220MB Xue: DfX with DT. TUDelft, 30 Jun 2023.  Team members New Hong Similar style??? Similar style??? Kong-ish blocks? 16 2.2 Method: BSS for compact 3D modelling(Wu et al. 2023)  MIB skeleton in geometry  Max Inscribed Ball centers  Counter-intuitive for plans  Building Section Skeleton (BSS)  Extends MIB for polygon plans  Data-driven building style descriptor  High-level abstraction  Q1 answered Xue: DfX with DT. TUDelft, 30 Jun 2023. 17 2.2 Generation of buildings and blocks  Data-driven groups  buildings  virtual blocks at TST Today on Google Earth BSS Group A C B D Xue: DfX with DT. TUDelft, 30 Jun 2023. 18 2.3 Case 3: DfX with DTs of climate and 3D env.  Q: MiC floorplan design for passive energy and natural lighting (conflicting ‘Xs’ in HK)  Miss Qianyun Zhou qianyunz@hku.hk  Q-Add: Modular-integrated Construction (MIC) brings discrete design variables  Case project: HKU High West student hostel (Block H1) o 19-story, 31 modules, 3 (6) types, for 470 students o Constraints: Same GFA, same module sizes, etc. HKU High HKU WCH Xue: DfX with DT. TUDelft, 30Hostel Jun 2023. West Student Hostel  Yr-1 PhD in Sept. 19 2.3 Method: GA + GH env. simulation (Zhou & Xue 2023) Xue: DfX with DT. TUDelft, 30 Jun 2023. 20 2.3 Results  After 15 hrs (GA+simu), 5 improved plans  in a preferred area, improving both Xs  EUI improved up to 0.42%, spatial daylight autonomy (sDA) improved up to 9.7% Xue: DfX with DT. TUDelft, 30 Jun 2023. 21 2.3 Analysis of generated Pareto optima  More iter. = more results 2h 16h 3 days  windowsouth is more sensitive  5 out of 11 design variables in the production discouraged by the Pareto optima Xue: DfX with DT. TUDelft, 30 Jun 2023. 22 2.4 Case 4: AI-DfX??  Deep learning AI, especially GAN (Generative Adversarial Network)  Trained on many input samples o Against classes, e.g., cat, dog, or latent  Can “judge” outputs – to a certain extent  Taking the “supervisor” role from human  The outputs are  Generated by an algorithm G and  Judged by the other D  So-called “adversarial” Xue: DfX with DT. TUDelft, 30 Jun 2023. 23 2.4 Intuitive examples  Geometric prompts/inputs  Apartment interiors (interpolation)  Input 1: Real samples  Input 2: Boundary + windows  By o ArchiGAN o Chaillou (2019), MArch (Harvard)  Also for exteriors  By o StyleGAN-ada o Rodrigues (2021). Xue: DfX with DT. TUDelft, 30 Jun 2023. 24 2.4 Research question  Q: “Can GAN generate floorplans for the music?” o Even better if readers can enjoy similar feelings for the verses  In essence: Music-to-plans  The GAN way  Step 1: Music => latent class  Step 2: Latent class + real plans => new plans  Step 3: Judge and select  Any successful story?  Step 1 + 2: LucidSonicDreams https://youtu.be/iEFqcMrszH0  Step 2: StyleGAN https://twitter.com/erikswahn/status/1123951017148788738 o https://mobile.twitter.com/erikswahn/status/1129472697514242048?cxt=HHwWgMC17eTb2KwfAAAA Xue: DfX with DT. TUDelft, 30 Jun 2023. 25 2.4 Input: HKU Anthem  A brief history  March 11, 1912: first performed at the Opening Ceremony of HKU, in front of the newly completed Main Building.  Until 1930s: used at formal University occasions  After 1940s war: forgotten  May 18, 2011: Revived to celebrate HKU’s centenary The 1912 Anthem* o Recorded with 150 musicians in the City Hall  Lyrics by Sir Cecil Clementi (20th Governor)  4 verses, mentioning o “modern from western,” “science [and] art hidden,” o “train youth’s vigor” and “light of wisdom” (Selected subjectively, based on the translation) Xue: DfX with DT. TUDelft, 30 Jun 2023. (*: Source: https://lib.hku.hk/muslib/HKUanthem.html ) The lyrics* 26 2.4 The ready-to-use music-to-video pipeline  Software: Lucid Sonic Dream, StyleGAN2  https://github.com/mikaelalafriz/lucid-sonic-dreams  https://github.com/NVlabs/stylegan2  Platform: Google Colab  Free GPU for 2 hrs every day  Audio source: HKUL  Real plan sketches  Collected by Mayur Mistry  o Antique-like styles  Python codes (30 lines)  1. Upload the Anthem to Colab virtual machine  2. Load Lucid Sonic Dream with the song  3. Load pre-trained StyleGAN2 model (300MB) for floor plans  4. Run  5. Download video  Final pipeline  Anthem  Lucid Sonic Dream  StyleGAN2  video of plans  selected plans Xue: DfX with DT. TUDelft, 30 Jun 2023. 27 2.4 Results of generated plans (video)  A 5:54 video  With morphing plans o Similar to the morphing arts on Page 7  “Interpolations” of the training plans  I collected some Xue: DfX with DT. TUDelft, 30 Jun 2023. (Full video link on the last page) 28 2.4 Featured on the cover page of Dean’s RoundUp  Upper:  Spectrogram of the Anthem  By “foobar2k”  Lower:  Four selected subjectively  Next…  May the plans trigger similar feelings to those from Anthem? Xue: DfX with DT. TUDelft, 30 Jun 2023. 29 2.4 30-line codes and class adjustment  A tutorial of LucidSonicDreams: https://colab.research.google .com/drive/1Y5i50xSFIuN3V4 Md8TB30_GOAtts7RQD#scr ollTo=Z7DkKcO9cfM_  For assigning class mapping, use the parameter below Xue: DfX with DT. TUDelft, 30 Jun 2023. 30 Section 3 DISCUSSION Xue: DfX with DT. TUDelft, 30 Jun 2023. 31 3.1 Summary Data Semantics in granularity “DT” Handy tools Simulation-based optimization 2.1 Individual bamboo Bamboo lengths GH Yes 2.2 Building / block Buildings’ section skeleton CloudCompare to edit 3D points Data-driven + integer programming 2.3 MiC module Env. (climate, 3D env.) GH/ wallacei, energy Yes (GA) (?Music?) (?Verse, tone, volume?) 2.4 Xue: DfX with DT. TUDelft, 30 Jun 2023. plus, ladybug, honeybee, radiance AIGC/ Lucid Sonic Dream, StyleGAN2 (?My manual selection?) 32 3.2 A recap  DfX indicates optimization – finding the best(s)  DT can help DfX and generative design in different aspects o Abstraction, generic model, constraints, and decision-making  Generative design is a human-centric approach for DfX  A DT contains nothing more than you need  Value-driven, Level-of-Detail, Level Of Information Needed  Many handy tools are on GH  Some are open-sourced elsewhere; some need Python coding  Disclaimer  I am a software architect, not a real architect.  My apology for potential misunderstanding or far-fetched arguments  Designer is still a human for AIGC  “AI designer” as an “employee” for low-level, tedious decisions Xue: DfX with DT. TUDelft, 30 Jun 2023. 33 Acknowledgement & job vacancies  Supported by our on-going DfX-related projects  Hong Kong RGC (C7080-22GF, 5.3M) – Generative DfX in high-rise modular building: An expert-augmented cascade graph learning and optimisation approach  Hong Kong RGC (T22-504/21-R, 34.6M) – Healthy and resilient city with pervasive LoCHs (localised outdoor thermal-comfort hubs)  Hong Kong ITF (ITP/004/23LP, 7.5M) – “SBASE” project  Job vacancies in my group  Postdoctoral Fellow: 1~5  Research Assistant: 5  PhD posts: 1~2 Xue: DfX with DT. TUDelft, 30 Jun 2023. 34 Acknowledgement (cont.) Generative DfX in high-rise modular building Localised outdoor thermal-comfort hubs Xue: DfX with DT. TUDelft, 30 Jun 2023. 35 References  Chaillou, S. (2019). ArchiGAN: a generative stack for apartment building design. NVIDIA Corporation.  Krish, S. (2011). A practical generative design method. Computer-Aided Design, 43(1), 88-100.  Rodrigues. (2021). https://www.youtube.com/watch?v=AbWfHdSNo_M  UKNIC. (2017). Data for the Public Good. National Infrastructure Commission, UK, London. https://www.nic.org.uk/publications/data-public-good/  Wu, Y., Xue, F., Li, M., & Chen, S.-H. (2023). A novel Building Section Skeleton for compact 3D reconstruction from point clouds: A study of high-density urban scenes. ISPRS Journal of Photogrammetry and Remote Sensing, under review.  Xue, F., Zhang, W., Xu, G., Zhou, Q., & Wu, Y. (2023). Surface or skeleton? Automatic hierarchical clustering of 3D point clouds of bronze frog drums for heritage digital twins. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences. X-M-1-2023, 293-299. https://doi.org/10.5194/isprs-annals-X-M-1-2023-293-2023  Zhou, Q., & Xue, F. (2023). Pushing the Boundaries of Modular-Integrated Construction: A Symmetric Skeleton Grammar-Based Multi-objective Optimization of Passive Design for Energy Savings and Daylight Autonomy. Energy and Buildings, under review.  Full video (78M in 1080p) generated from the Anthem:  https://www.dropbox.com/s/n02e5z83f17h73w/Floor%20plans%20generated%20using%20HKU%20Anthem%20as%20the %20input.mp4?dl=0 Xue: DfX with DT. TUDelft, 30 Jun 2023. 36 Enjoy your incoming weekend! Q&A